The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Feature #11900

Dynamic environment interpolation in puppet master configuration

Added by Mike McLane over 4 years ago. Updated almost 3 years ago.

Status:Needs More InformationStart date:01/11/2012
Priority:NormalDue date:05/01/2012
Assignee:-% Done:

0%

Category:server
Target version:-
Affected Puppet version:development Branch:
Keywords:dynamic environment, puppet master configuration, interopolated configuration, environment parameter

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

In providing a single Puppet master to multiple teams, where each team has their own test/release cadence, we maintain that each team would like to include their own dynamic module path based on the environment passed from the agent. We have a re-usability case where an enterprise level of common modules is provided at the lowest path inclusion priority, the local team has a set of common modules at the next highest level of include priority, and then the dynamic environment has the highest level of inclusion priority. This allows for development and testing cases to be carried out through dynamic pulls of environment-labeled branches in git.

Some form of regex matching of the environment to the configuration section, might help in this regard.

In such a setup it would be nice to configure a puppet master as:

[teama*]
 manifest   =  $confdir/environments/manifests/teamb/$environment/site.pp
 modulepath =  $confdir/environments/modules/teamb/$environment/:$confdir/common/modules/teamb/:$confdir/common/modules/enterprise/
 
[teamb*]
 manifest   =  $confdir/environments/manifests/teama/$environment/site.pp
 modulepath =  $confdir/environments/modules/teama/$environment/:$confdir/common/modules/teama/:$confdir/common/modules/enterprise/

…or, alternatively.. such a case where the environment passed in is parsed within the configuration, allowing config-file variables to be set as such:

[production]
# passed in $environment = "team-production-product"
  $group = $environment.split("-")[0]
# $releasecycle = $environment.split("-")[1]
# $product = $environment.split("-")[2]
 manifest   =  $confdir/environments/manifests/$group/$environment/site.pp
 modulepath =  $confdir/environments/modules/$group/$environment/:$confdir/common/modules/$group/:$confdir/common/modules/enterprise/

Within the base environments path, I would initialize the directory as a git repository and keep a post-receive wrapper script to dynamically fetch environments based on the $environment path.


Related issues

Related to Puppet - Bug #13858: Custom types in environments require loading into master'... Closed 04/08/2012
Related to Puppet - Bug #4409: puppetmasterd does not find custom types for environment Closed 07/30/2010

History

#1 Updated by Mike McLane over 4 years ago

My example sucks. The regex example option might make more sense as:

[teamA*]
# $environment could be = teamA-production or teamA-testing
 manifest   =  $confdir/environments/manifests/teamA/$environment/site.pp
 modulepath =  $confdir/environments/modules/teamA/$environment/:$confdir/common/modules/teamA/:$confdir/common/modules/enterprise/
 
[teamB*]
# $environment could be = teamB-production or teamB-testing or teamB-foofeaturetest1
 manifest   =  $confdir/environments/manifests/teamB/$environment/site.pp
 modulepath =  $confdir/environments/modules/teamB/$environment/:$confdir/common/modules/teamB/:$confdir/common/modules/enterprise/

#2 Updated by eric sorenson over 4 years ago

+1 for this, i have exactly the same problem.

Is this after the node classifier? Could an ENC variable be made visible here?

#3 Updated by Kelsey Hightower over 4 years ago

Mike,

Thanks for reporting this issue. I have a couple of questions:

  • What version of Puppet are your running?
  • Are you using and ENC?

You could use custom facts on each node to set $group, $releasecycle, and $product. Those values would then be accessible within the Puppet manifests. Writing Custom Facts

#4 Updated by Kelsey Hightower over 4 years ago

  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Mike McLane

#5 Updated by eric sorenson over 4 years ago

Kelsey Hightower wrote:

You could use custom facts on each node to set $group, $releasecycle, and $product. Those values would then be accessible within the Puppet manifests. Writing Custom Facts

Kelsey — correct me if I’m wrong, but I believe ENC evaluation happens after environment determination, and therefore ENC values are not available in puppet.conf sections to determine, for example, which module path to consult for a particular node.

In my environment (and I’m not Mike obviously, but we talked this through on IRC and our use-case is quite similar) we do have an ENC but would like to have the ability to run code (even as simple as Mike’s split example on the environment string) to do something more complex than inline interpolation of the client-provided $environment into the module path.

#6 Updated by Kelsey Hightower over 4 years ago

Eric Sorenson wrote:

Kelsey — correct me if I’m wrong, but I believe ENC evaluation happens after environment determination, and therefore ENC values are not available in puppet.conf sections to determine, for example, which module path to consult for a particular node.

ENC does have the ability to set the environment for a node see: External Nodes, it just does not work 100% #3910

In my environment (and I’m not Mike obviously, but we talked this through on IRC and our use-case is quite similar) we do have an ENC but would like to have the ability to run code (even as simple as Mike’s split example on the environment string) to do something more complex than inline interpolation of the client-provided $environment into the module path.

You could use the built-in split function on the $environment var inside of your manifests. Puppet Split Function

#7 Updated by Mike McLane over 4 years ago

  • Assignee changed from Mike McLane to K Hightower

This is under Puppet 2.7.9 under CentOS 5.x and CentOS 6.x.


I am trying to keep to a design pattern where a type of module/class designated as a “service module” do not include configuration values and we use a module designated as “site modules” to keep specific configuration classes with all of our special business bits in them. This allows teams throughout the organization to create, use, and eventually contribute back to the enterprise class sharing without exposing their data bits and allows each team to continue with whatever change management cadence they current follow (all of the teams have disconnected schedules).


Dropping each and every combination/permutation at the service module level isn’t ideal as it really bloats those classes out — and it makes some assumptions about the release cycle of each team. We have data/configuration bits that may be private and shouldn’t be shared between teams. We might be able to do this at a the site module level, but then we collide with release cycles.


In our multi-team environment it is important that each team can ingest enterprise-provided classes but also have the ability to over-ride those classes with updates either at the team-wide path or at their team-product specific path.


Simple use case example to help explain might be: we have a /etc/motd class at the enterprise level inclusion path (classname ‘motd’ version ‘1.0.0’), {$confdir/common/modules/enterprise/motd/}.


  • TeamA updates classname ‘motd’ to version ‘1.1.0’, and includes it at their team path, {$confdir/common/modules/teamA/motd/}.
  • TeamB updates classname ‘motd’ to version ‘1.0.1’ (bugfix) and includes it at their team path, {$confdir/common/modules/teamB/motd/}.
  • TeamB-developer1 creates their own module for development/testing ‘motd’ and updates to version ‘1.2.0’ and includes this at their dynamic environment (team-product-developername) path, {$confdir/environments/modules/teamB/$environment/motd/ => $confdir/environments/modules/teamB/developerxyz-testing/motd/}.

In a case where we set up custom facts that still wouldn’t satisfy class file inclusion paths.. which have at least 3 levels (General to Specific):


  • Enterprise
  • Team
  • Team’s Product

Unless each environment (team/product) can manipulate the module inclusion path from within the module, I’m just not sure how that would work. We want to be able to use multiple environments and we believe the module multiple include path usage would make that work.


We also would prefer not to having to restart the puppet masters every time a new environment needs to be added. Dynamic environments (as I understand) would allow each team to break their own environment, without breaking the puppet master. There’s some risk in doing a puppet master restart in a case where someone has bad or broken manifests would prevent the puppet master from being able to start.


It sounds like we may need to go with a hybrid approach where we have a set of very static environments and inclusion paths, but then also have a single dynamic environment option that non-defined environments default to.


I’d like to use ENC (via the dashboard), but the community version of dashboard lacks fairly significant role based access authentication features.

#8 Updated by Kelsey Hightower over 4 years ago

Nigel,

We have a pretty good use case going here, care to weigh in?

#9 Updated by Nigel Kersten over 4 years ago

I’m just checking that everyone here realizes that we have the ability to dynamically determine environments now.

[master]
modulepath=/foo/$environment/modules
manifest=/foo/$environment/manifests/site.pp

If you do this, and don’t actually define [environment_name] blocks, whatever the agent specifies as the environment will flow through, and the relevant modulepath/manifest will be consulted, so long as the directory actually exists.

Does that solve the core problem here?

#10 Updated by Mike McLane over 4 years ago

Thank you for looking into this Nigel.

Yes, I believe those involved with this thread understand how dynamic environments work currently.

What I’m trying to do is set different module load paths based on the environment.

This might be satisfied if there was a way to set multiple dynamic environments on a single master, modify module load paths at run-time, or do variable interpolation from within the puppet master config file.

#11 Updated by Nigel Kersten over 4 years ago

Mike McLane wrote:

Thank you for looking into this Nigel.

Yes, I believe those involved with this thread understand how dynamic environments work currently.

What I’m trying to do is set different module load paths based on the environment.

This might be satisfied if there was a way to set multiple dynamic environments on a single master,

That’s why I’m a little confused :) as you can do this now.

You just need to create the relevant directory hierarchy underneath /foo/ in my example above. Each environment will have a different module load path.

modify module load paths at run-time, or do variable interpolation from within the puppet master config file.

Yep, I’ve hit cases in the past where I wished we could set the module load path at run time…

Let me get a little more concrete:

In our multi-team environment it is important that each team can ingest enterprise-provided classes but also have the ability to over-ride those classes with updates either at the team-wide path or at their team-product specific path

So wouldn’t this work?

[master]
modulepath = /var/lib/puppet/environments/$environment/modules:/var/lib/puppet/core-enterprise/modules
manifest = /var/lib/puppet/environments/$environment/manifests/site.pp

I assume you’re using version control, and you can set up a sync task between some VCS location and /var/lib/puppet/environments.

The teams can dynamically define environments by checking code in that maps to /var/lib/puppet/environments/foobarbaz and their agents can use ‘foobarbaz’ as the environment.

We also would prefer not to having to restart the puppet masters every time a new environment needs to be added.

I took a different approach in the past with a very similar set of needs, and had a .yaml file in version control that described each environment and the module path. This meant developers could add environments with their own combination of directories for the modulepath.

This synced regularly to the master, and we used an .erb template to evaluate the yaml file, and create [environmentname] blocks in puppet.conf.

I did this because we needed to be able to not only define environments to segment teams, but for the classic dev -> test -> prod cycle for each team-specified environment as well.

If you touch puppet.conf, the puppetmaster process automatically picks up the changes and you don’t need to restart it.

#12 Updated by Mike McLane over 4 years ago

Oh! I was unaware that the master detects changes in the conf and reloads the config files. Does that reload just merge in new config sections, or replace existing configuration? I am curious as to whether a reload might interrupt a concurrent connection into a puppet master, while that config-refresh is happening. Is the pupper master still susceptible to bad manifests bombing the entire server? Seems like that might be an acceptable risk (everyone should be lint testing their check-ins anyway, right?).

So if that’s the case — yeah. We could add in a process to allow all stakeholders to more or less auto-magically add their own environments/paths to the puppet master, through revision control. We set up some automation to fetch new conf bits, sync the modules/manifests to the puppet master visible paths, apply the new environment into puppet.conf, and we’re a-rockin.

I believe this may satisfy what we’re looking for. I’ll start modeling some proof of concepts on it.

..This could very well make for a solid presentation at a future PuppetConf.

#13 Updated by eric sorenson over 4 years ago

I understand all you’ve written nigel and I still would like the ability to break the environment into two keys; specifically, one to specify the organisational unit and one to pick the branch of code inside that unit that ought to be run. Hmm, unless the modulepath=/root/$environment/modules honors the ‘/’ character inside $environment, that’d be perfect…

#14 Updated by Nigel Kersten over 4 years ago

Mike McLane wrote:

Oh! I was unaware that the master detects changes in the conf and reloads the config files. Does that reload just merge in new config sections, or replace existing configuration? I am curious as to whether a reload might interrupt a concurrent connection into a puppet master, while that config-refresh is happening. Is the pupper master still susceptible to bad manifests bombing the entire server? Seems like that might be an acceptable risk (everyone should be lint testing their check-ins anyway, right?).

It will reparse and replace the existing config.

It will do that at the start of the next run, so you should be fine, but yes, I’d absolutely have a version control pre-commit hook that stops people checking in bad manifests or .erb templates.

So if that’s the case — yeah. We could add in a process to allow all stakeholders to more or less auto-magically add their own environments/paths to the puppet master, through revision control. We set up some automation to fetch new conf bits, sync the modules/manifests to the puppet master visible paths, apply the new environment into puppet.conf, and we’re a-rockin.

I believe this may satisfy what we’re looking for. I’ll start modeling some proof of concepts on it.

..This could very well make for a solid presentation at a future PuppetConf.

and we’d love to have you do it! Or even a blog post?

I’m nigel@puppetlabs.com and please feel free to ping me off-ticket if you want to have other chats about this approach.

#15 Updated by Nigel Kersten over 4 years ago

eric sorenson wrote:

I understand all you’ve written nigel and I still would like the ability to break the environment into two keys; specifically, one to specify the organisational unit and one to pick the branch of code inside that unit that ought to be run. Hmm, unless the modulepath=/root/$environment/modules honors the ‘/’ character inside $environment, that’d be perfect…

I genuinely don’t know as I haven’t tried it…. maybe? :)

Is it possible for you to generate directories based upon the possible values of the two keys? Do you know what they all would be?

Ultimately it would be quite awesome if we could simply set manifest and modulepath in an ENC… but you’d have to accept that you’d get:

  • fact values from existing facts
  • ENC determines modulepath for pluginsyncing facts
  • fact values from pluginsynced facts.

which could be rather frustrating, not being able to make modulepath decisions based upon facts in modules…

#16 Updated by eric sorenson over 4 years ago

Nigel Kersten wrote:

I genuinely don’t know as I haven’t tried it…. maybe? :)

I’ll give it a try once I’m back on first-world internets.

Is it possible for you to generate directories based upon the possible values of the two keys? Do you know what they all would be?

Could be, but that’s another step to lookup isn’t it? Where is the hook to do that step?

Ultimately it would be quite awesome if we could simply set manifest and modulepath in an ENC… but you’d have to accept that you’d get:

  • fact values from existing facts
  • ENC determines modulepath for pluginsyncing facts
  • fact values from pluginsynced facts.

which could be rather frustrating, not being able to make modulepath decisions based upon facts in modules…

That would actually all be wonderful, because I am a source-of-truth absolutist: I think if you’re using an ENC, you’re doing so because you have your business logic in an external source-of-truth and therefore you can (&& should) make all your decisions with that same mechanism. As I understand it now, ENC determination happens after / as a result of puppet.conf evaluation and therefore the situation’s backwards from this bulleted list.

#17 Updated by Nigel Kersten over 4 years ago

eric sorenson wrote:

Is it possible for you to generate directories based upon the possible values of the two keys? Do you know what they all would be?

Could be, but that’s another step to lookup isn’t it? Where is the hook to do that step?

cron? VCS hook? It depends where your source of truth is about these keys.

Ultimately it would be quite awesome if we could simply set manifest and modulepath in an ENC… but you’d have to accept that you’d get:

  • fact values from existing facts
  • ENC determines modulepath for pluginsyncing facts
  • fact values from pluginsynced facts.

which could be rather frustrating, not being able to make modulepath decisions based upon facts in modules…

That would actually all be wonderful, because I am a source-of-truth absolutist: I think if you’re using an ENC, you’re doing so because you have your business logic in an external source-of-truth and therefore you can (&& should) make all your decisions with that same mechanism. As I understand it now, ENC determination happens after / as a result of puppet.conf evaluation and therefore the situation’s backwards from this bulleted list.

But you’d be perfectly happy not being able to use any custom facts to determine the modulepath for a fresh host?

I don’t think an external source of truth is the only reason to use an ENC.

You may find it more flexible as a source of truth than node {} declarations in your manifests.

For others reading this thread, it also may not be obvious that you can set your environment agent-side with a custom fact that returns environment and not set it in the agent puppet.conf if you want.

#18 Updated by K Hightower over 4 years ago

  • Assignee deleted (K Hightower)

#19 Updated by eric sorenson about 4 years ago

An update:

it turns out that $environment can be a slash-separated directory structure, and the puppet.conf variable interpolation just expands it in-place.

So given a directory structure like:

environments/group1/test/{modules,manifests}
environments/group1/release/{modules,manifest}
environments/group2/test/{modules,manifests}
environments/group2/release/{modules,manifests}
environments/common/test/modules
environments/common/release/modules

a puppet.conf like:

manifestdir=/etc/puppet/environments/$environment/manifests/site.pp
modulepath=/etc/puppet/environments/$environment/modules:/etc/puppet/environments/common/release

and a client run with puppet agent --environment group1/test

The right thing happens and the modules and manifests from /etc/puppet/environments/group1/test are loaded. So that’s exactly what I needed. Just please don’t start enforcing an alphanum-only character set on $environment at some point down the line :)

#20 Updated by Nigel Kersten about 4 years ago

  • Assignee set to Anonymous

Hah. Awesome.

I’m going to assign this to Daniel to comment on how we can make sure ‘/’ still works in environment names for you, and in case we’ve already broken that in Telly. :)

#21 Updated by eric sorenson about 4 years ago

I spoke a little too soon. Modules are found OK but pluginsync is broken; the first path element is lopped off so an environment like mygroup/trunk gets errors like this:

puppet agent -tv --environment=mygroup/trunk
info: Retrieving plugin
err: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': 
Error 403 on SERVER: For client.mydomain(10.12.11.13) access to /trunk/file_metadatas/plugins [find] authenticated at line 60

It’d be cool if this actually worked, but it’s not a deal-breaker. Current plan is to use a directory of symlinks as the landing place for the environment string modulepath, so the individual repositories that the different groups use can be structured however they want and we’ll just link into the right place.

#22 Updated by Jerome Loyet over 3 years ago

eric sorenson wrote:

An update:

it turns out that $environment can be a slash-separated directory structure, and the puppet.conf variable interpolation just expands it in-place.

So given a directory structure like:

environments/group1/test/{modules,manifests}
environments/group1/release/{modules,manifest}
environments/group2/test/{modules,manifests}
environments/group2/release/{modules,manifests}
environments/common/test/modules
environments/common/release/modules

a puppet.conf like:

manifestdir=/etc/puppet/environments/$environment/manifests/site.pp
modulepath=/etc/puppet/environments/$environment/modules:/etc/puppet/environments/common/release

and a client run with puppet agent --environment group1/test

The right thing happens and the modules and manifests from /etc/puppet/environments/group1/test are loaded. So that’s exactly what I needed. Just please don’t start enforcing an alphanum-only character set on $environment at some point down the line :)

well it should work on a standalone configuration but it’s not working when the agent is connected to a master: – file auth.conf (http://docs.puppetlabs.com/guides/rest_auth_conf.html) doesn’t like it – slashed in environment will be confused with the URL dash separator – in lib/network/http/api/v1.rb (line 28), environments are restricted to be /\w+$/

We have the same kind of needs in our compagny. Dealing with a lots of environments and sub-environments, from a bunch of teams, with different acccess rights, … We are using dynamically determine environments in puppet.conf to have a generic configuration but we have a lots of environments and it becomes to be hard to have all environments in one directory, we need to “hash” $environment somehow to split environments between sub directories (split, splice, …).

Here’s a refected pull request to see the spirit: https://github.com/puppetlabs/puppet/pull/1028

#23 Updated by Anonymous almost 3 years ago

  • Assignee deleted (Anonymous)

Also available in: Atom PDF