The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Feature #11784

Hiera should support alternate environments

Added by Hunter Haugen over 4 years ago. Updated over 2 years ago.

Status:Needs More InformationStart date:01/05/2012
Priority:NormalDue date:
Assignee:eric sorenson% Done:

0%

Category:hiera
Target version:2.0.0
Keywords:hiera Affected Hiera Version:
Branch:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/HI-46


Description

Currently hiera supports one hiera.yaml file hardcoded to be in the same location as puppet.conf (which is the config puppet directive.

Having separate hiera.yaml’s per puppet environment would go along with having separate site.pp’s, modules, etc. per environment.


Related issues

Related to Hiera - Bug #21669: Hiera.yaml will not interpolate variables if datadir is s... Closed

History

#1 Updated by Kelsey Hightower about 4 years ago

  • Category set to hiera
  • Status changed from Unreviewed to Needs Decision
  • Assignee set to Kelsey Hightower

Hunner,

I must say, you keep the coming up with great feature requests! Now that hiera is being better integrated with Puppet it should be pretty easy to make this work. I’ll work up a patch as part of the work being done for Project Lowrider.

#2 Updated by Nigel Kersten about 4 years ago

Actually, I was originally convinced of this, and RI talked me out of it with a compelling argument.

Data is different to manifests, and the environment is best used as a level in the hierarchy, not to define completely different hierarchies.

Think about setting up a machine to run in “test” rather than “prod”. It’s not reasonable to force people to copy all their data to the test environment, as the data is largely the same data. There will be some differences though, so having an ability to specify environment-specific data is necessary by virtue of another lookup layer.

This allows you to set environment-specific data, but doesn’t require you copy all your data around.

If you really want this, you can always do it with interpolation in the file locations with an environment prefix.

#3 Updated by Nigel Kersten about 4 years ago

  • Status changed from Needs Decision to Rejected

I’m going to reject this fwiw. Feel free to respond with counter-arguments.

#4 Updated by Hunter Haugen almost 4 years ago

  • Status changed from Rejected to Re-opened

The problem is when you want to create a different hierarchy for an environment without changing production’s hierarchy. That’s the one thing there is no current solution for, and why this ticket was originally made.

Example: Say that you have a simple hierarchy for production, and decide to refactor the datastore layout and also restructure the hierarchy to be more extensible. You can restructure the data in your datadir and use %{environment} in the datadir path, but you can’t actually modify hierarchy in hiera.yaml on a per-environment basis.

So either you need to be able to have one hiera.yaml with multiple hierarchies in it per-environment, or multiple hiera.yaml files, one per environment.

#5 Updated by Nigel Kersten almost 4 years ago

  • Assignee changed from Kelsey Hightower to R.I. Pienaar

RI, thoughts on Hunter’s argument here?

#6 Updated by R.I. Pienaar almost 4 years ago

It seems to me if you’re going to redesign the hierarchy – which would affect the modules etc – then you should be managing a different hiera.yaml for the environment rather than overload hiera.yaml to also be environment aware.

So with puppet 3 and its hiera integration I’d say the puppet.conf option that says where to find the hiera config file should be environment compatible. This would be more in line with the overall design of puppet and its config model see for example the ‘manifest’ option.

#7 Updated by Nigel Kersten almost 4 years ago

  • Status changed from Re-opened to Accepted
  • Assignee deleted (R.I. Pienaar)
  • Keywords set to hiera

I like that idea a lot more actually.

#8 Updated by Kelsey Hightower almost 4 years ago

Since we are in RC with Puppet should we add the configuration option to Puppet to support this?

#9 Updated by Hunter Haugen almost 4 years ago

Adding a configuration directive for the location of hiera.yaml was my original thoughts about this, but when it came to logging the ticket amid the rush of getting hiera into PE 2.5 and Puppet being feature-frozen, I didn’t actually mention that.

#10 Updated by Kelsey Hightower almost 4 years ago

  • Target version set to 2.0.0

#11 Updated by Patrick Carlisle about 3 years ago

  • Status changed from Accepted to Needs More Information

Since we have a hiera_config setting in puppet, and you can use $environment in setting it, is there anything else needed?

#12 Updated by Jeff Goldschrafe about 3 years ago

Patrick Carlisle wrote:

Since we have a hiera_config setting in puppet, and you can use $environment in setting it, is there anything else needed?

Actually, there is one tiny little detail that’s a tiny bit significant for us, and trivial to implement. If there are relative paths specified for :datadir in hiera.yaml, they should be interpreted relative to the hiera.yaml file (i.e. relative to the environment’s root) rather than the current working directory of Puppet.

We use what I think is a pretty typical setup for developing our Puppet modules — our environment root is a Git repository, and people wanting to do serious module development are encouraged to check out this repository, make their changes locally, and smoke-test them using puppet apply before the code comes anywhere near a Puppetmaster. The problem that still exists with specifying hiera_config = /etc/puppet/$environment/hiera.yaml is that our admins can’t run against the same hiera.yaml in development as in production, because hiera.yaml requires an absolute path to :datadir that doesn’t (and shouldn’t) exist on the systems they’re developing on.

#13 Updated by Anonymous about 3 years ago

  • Assignee set to eric sorenson

Patrick Carlisle wrote:

Since we have a hiera_config setting in puppet, and you can use $environment in setting it, is there anything else needed?

The current way that puppet loads the hiera config is, surprise!, as a singleton. This means that the hiera_config setting is not environment aware.

#14 Updated by Simon Mügge about 3 years ago

Hi!

The proposed “workaround” of “hiera_config = /etc/puppet/$environment/hiera.yaml” would fit my needs perfectly, but I cant get this to work for some reason.

puppetmaster 2.7.18 (squeeze backports packages), latest gems for hiera and hiera-puppet. /etc/puppet/someenv/hiera.yaml: “—– :backends: yaml :yaml: :datadir: ‘/etc/puppet/%{::environment}/hieradata’ :hierarchy: – defaults”

Error reads as “Could not find data item foo in any Hiera data file[..]” which in this case seems to indicate that the file is not being used. When I put the file into the default location (/etc/puppet/hiera.yaml) all works as expected.

Which leads me to “puppet does not honor hiera_config”, at least not in this constellation (I tried the official squeeze packages for hiera and hiera-puppet as well, same result).

Any hints as to how to solve this, other than upgrading? I’d be willing to implement dirty solutions.. ;)

#15 Updated by Corey Osman about 3 years ago

Additionally Hiera should use ENC parameters to determine the datadir location.

Example:

Given ENC info:

--- 
  environment: development
  classes: 
    - "roles::appserver"
  parameters: 
    puppetmaster: localhost
    foreman_env: development

and hiera.yaml

---
:backends:
  - json
:json:
  :datadir: /etc/puppet/modules/%{::environment}/hieradata
:hierarchy:
  - nodes/%{fqdn}
  - common
:logger:
  - puppet

Hiera should use the ENC environment parameter to determine datadir.

#16 Updated by eric sorenson about 3 years ago

Andrew Parker wrote:

The current way that puppet loads the hiera config is, surprise!, as a singleton. This means that the hiera_config setting is not environment aware.

A lot of people seem to miss this, so I’ll quote it for emphasis.

Additionally some folks are trying to do dynamic (i.e. different lookup on each catalog compile) interpolation inside hiera.yaml itself. This also doesn’t work. It’s just the hierarchy which is looked up.

I understand the use case y'all are trying to solve and would like to have this in scope for hiera 2.

#17 Updated by eric sorenson almost 3 years ago

eric sorenson wrote:

Additionally some folks are trying to do dynamic (i.e. different lookup on each catalog compile) interpolation inside hiera.yaml itself. This also doesn’t work. It’s just the hierarchy which is looked up.

Sorry for the misinformation, putting %{environment} in your datadir variable does work and people are using it in production. Make sure your quoting is correct; see this thread for several working examples: https://groups.google.com/forum/#!msg/puppet-users/cCfimbolUio/SSzmG7GJfhIJ

#20 Updated by Max Williams over 2 years ago

The posts here seem to contradict each other a bit. As it stands now in Puppet 3.2, can we use the environment variable in the hiera_config directive of puppet.conf, like this?

hiera_config = /etc/puppet/$environment/hiera.yaml

I can see it does interpolate the variable by running “puppet master —configprint hiera_config” but I seem to be unable to get it to actually work when doing a puppet run.

#21 Updated by Jeffrey Chan over 2 years ago

Max Williams wrote:

The posts here seem to contradict each other a bit. As it stands now in Puppet 3.2, can we use the environment variable in the hiera_config directive of puppet.conf, like this?

hiera_config = /etc/puppet/$environment/hiera.yaml

I can see it does interpolate the variable by running “puppet master —configprint hiera_config” but I seem to be unable to get it to actually work when doing a puppet run.

I’m having the same problem as Max Williams, running Puppet 3.2. I can see that the hiera_config will only interpolate to one environment value and does not change dynamically.

Are there any plans to make this work in later versions, i.e. not load the hiera_config as a singleton, but treat it more like the way we can load modules per environment?

Also available in: Atom PDF