The Puppet Labs Issue Tracker has Moved:

Feature #15455

add support for running acceptance tests via rpms

Added by Chris Price almost 2 years ago. Updated over 1 year ago.

Status:AcceptedStart date:07/10/2012
Priority:NormalDue date:
Assignee:Justin Stoller% Done:


Category:-Spent time:-
Target version:-
Patch: Branch:

We've Moved!

Ticket tracking is now hosted in JIRA:

This ticket may be automatically exported to the CPR project on JIRA using the button below:


We currently have a path for running acceptance tests using our deb packages. This is awesome.

However, we don’t have a similar path for running on redhat/EL systems using our rpm packages—this seems like a worthwhile addition.

This is a very general problem that affects many other projects besides puppetdb, so I’d really love to see us solve this in a general way that would be re-usable by other teams… and I don’t think it would be much extra work to do it that way.

My proposal would be to do the following:

Add something to the acceptance framework that would allow you to tell it to install something from a package. I think this could probably be handled with a very minor variation of the “—yagr” flag that we added recently; the idea would be that you’d specify multiple instances of this flag and they’d point to a git URL. The framework would then clone the specified git repo, and run a rake task that we’ve come up with a naming convention for. (e.g. ‘rake package’). Once the package had been built, the framework could scp it to the necessary hosts and install it.

The one minor issue I can foresee here is that not all machines will be capable of building packages. (This is why Nick’s current jenkins job ssh’s into the “deb-builder” machine in order to build the deb.) Thus, we’d probably need to standardize on what kinds of packages can be built on which hosts, and have that knowledge baked into the acceptance framework. That doesn’t seem too hard, though, and it still seems like it wouldn’t be much extra work at all to handle that generically in the acceptance framework rather than a custom solution built specifically for puppetdb.

Lastly, we’ll need some way to indicate what type of package to build (deb, rpm, ??). This could be an explicit flag that you pass to the framework (in which case you’re probably restricted to running each acceptance test run in a homogenous environment, but that’s probably not a huge deal), or we could see if there’s some reasonable way for the framework to be smart enough to pick the right package type(s) for the current list of hosts in the config file. The explicit flag is probably an easier (and reasonable, incremental) first step.

Related issues

Related to Puppet Acceptance - Feature #15282: add generic support for installing code via packages inst... Rejected 06/28/2012


#1 Updated by Chris Price almost 2 years ago

Added a few watchers:

Andy, because I know this is something that he’s been wanting for a long time… Justin because he probably has good advice for how this should or shouldn’t be implemented in the framework, and Haus in case he has some insight into which machines it would be acceptable to loop into the CI ecosystem to use for building packages.

#2 Updated by Deepak Giridharagopal almost 2 years ago

  • Status changed from Unreviewed to Accepted

#3 Updated by Justin Stoller almost 2 years ago

So I’m not sure how I feel about putting the logic for building packages too near the harness. If you have an immediate concern for rpms, you could probably also have that job ssh to rpm-builder?

I had a lot more written but I think is probably pretty self explanatory, this is a list (in rough order) of how I had seen this ultimate goal panning out, and what I believe has been done already in this regard.

  • Normalize and improve Rake tasks around package namespace for projects. (Moses and Haus have already done a ton of work to improve package rake tasks, but I believe there’s still a lot to do)

  • Puppetize Rpm-Builder (Sles-Builder?), Deb-Builder and Win-Builder. (I think there’s a basic template for the builders, Cooper has “volunteered” to help me puppetize Win-Builder)

  • Deploy Builder Jenkins in DC (including creating jobs for each project). (There’s actually several Jenkins' we need to build this was in my backlog after the IDW but a few other things came up including the last throws of PE 2.5.2, I believe it is on Dom’s plate for next week?)

  • Create Testing Repositories (I believe Ops has puppetized our public facing apt and yum repos, I believe Moses has deployed RC repos, don’t know how much work it would be to get testing repos put up and updated on demand)

  • Allow Harness to install system packages from local or repo packages (I started refactoring yagr, puppet, facter, hiera, hiera-puppet, module, and plugin flags to simply use yagr’s logic as a backend, and well as deprecate them all for a --install flag, I think the flag should take a pkg:// uri in addition to a git:// uri to differentiate between system packages and git repositories, I won’t have that pull request ready till next week)

  • Tie jobs into pipeline, specs –> build packages –> promote to testing repo –> acceptance using packages –> promote to beta/rc repo? (starts of this have been done in a haphazard way when done at all, we’ll be bringing up a new public jenkins as well as a new builder jenkins and when we do we will need to redo our jobs to be consistent and more maintainable)

  • Scale builders to allow developers to request packages built for various systems from integration branches. (I feel that getting a good pipeline running for official releases first is the way to go, then scaling that infra to allow devs to use it as much as possible, but I’m open to suggestions if your team would like to canary something)

I think when you have time/an iteration to work on this, finding something on this list and pinging someone in Delivery to see if you can move that item forward would be the most productive thing to do….

Someone please chime in if I’m way off base on any of these.

#4 Updated by Chris Price almost 2 years ago

Thanks, Justin, I didn’t realize that so much thought had been put into this already. Your plan sounds great…

Here are the only caveats / questions that I can think of at the moment:

  1. It would be really nice to be able to do heterogenous runs through acceptance (i.e., a config file that runs a mix of redhat/debian/win nodes). Off of the top of my head, I can’t think of an easy way to make that happen using a “pkg://” url; it seems like you’d need to have multiple flags (—rpm-pkg, —deb-pkg, —win-pkg). If the acceptance job was also in charge of building the packages, you could use the single git url and some well-known rake task names to solve this. Maybe we could use a naming convention to solve it via a pkg:// url as well…
  2. I do see the value of separating the build phase from the acceptance phase, as long as there is some way to have the upstream build job trigger a parameterized downstream build and pass the exact package filename / url. Without that parameterization it seems like there could be a race condition where the build job ran again while an acceptance job was in the queue, and by the time the acceptance job ran it was using a newer package than the one that originally should have triggered it?
  3. We had some trouble at my last job with the script in the jenkins jobs getting too complex and then copied and pasted all over the place. I like the idea of separating the build/package logic from the main acceptance harness but it would still be nice if it could be in VCS somewhere, and shared amongst all the build/package jobs on jenkins. Maybe a separate harness for packaging? I dunno, you’ve always been really awesome at making sure that the script in the jenkins jobs doesn’t get too out of hand so I’m sure you’ve already thought of this and that whatever solution you’ve got in mind will be great.

#5 Updated by Chris Price almost 2 years ago

Oh, one more thought:

  • It seems likely that some projects will need to install more than one package, so hopefully the —install flag would end up being something that could be specified multiple times?

p.s., thanks for the reply. I don’t know exactly how urgent the rpm stuff is for us, was just trying to plant the seed a bit ahead of time. We will definitely check in with you guys in person before taking any action on anything.

My main line of thinking was that if we could move forward with rpm support for puppetdb tests in any way that would be more general than just adding another jenkins job with a bunch of custom shell/ssh stuff, that would be awesome. But it would be easy to do it the ssh-to-rpm-builder way if that lines up better with your plans.

#6 Updated by Justin Stoller over 1 year ago

I think nlew has spent a fair bit of time working on this lately (just added him as a watcher). I would love to hear how his work may or may not fit into this outline/discussion.

#7 Updated by Justin Stoller over 1 year ago

Added the rest of Delivery to the ticket as this, to the best of my knowledge is a good place to where most of this knowledge can be captured.

#8 Updated by Chris Price over 1 year ago

  • Project changed from PuppetDB to Puppet Community Package Repository
  • Assignee set to Justin Stoller

Hey Justin—we’re done with this on the PuppetDB side for now, so I wanted to move it to a more appropriate project. Not sure if this is the right one or not—mind putting it wherever you think it belongs?

Also available in: Atom PDF