The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Feature #16107

hiera should support merging different keys from nested data structures in different areas of the hierarchy

Added by Tim Mooney over 3 years ago. Updated about 3 years ago.

Status:ClosedStart date:08/23/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:hiera
Target version:1.2.0
Keywords:hash Affected Hiera Version:0.3.0
Branch:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

This ticket was spawned from discussion in the puppet-users group, the thread with the subject “Getting all variable occurrences from Hiera” from 2012-Aug.

Both YAML and puppet support hashes, arrays, and nested data structures composed of the same. The issue is that although hiera supports looking up things based on your established hierarchy, it will not merge keys from nested data structures where parts of the data structure may appears in different areas of the hierarchy.

Consider a hiera.yaml like:

---
:backends: - yaml

:hierarchy: - fqdn/%{clientcert}
            - location/%{location}
            - common

:yaml:
            :datadir: /etc/puppet/hiera-data

Now consider the following two YAML files:

common.yaml:

---
sandbox_tvm:
  master:
    version: 1.2.3
    name: foo.bar.edu
  client:
    version: 1.2.1

fqdn/foo.bar.edu.yaml:

sandbox_tvm:
  master:
    version: 1.2.4

Hopefully the intent is clear: I want to establish defaults for several settings under the sandbox_tvm “namespace” in my common.yaml, and on the host with $::fqdn of foo.bar.edu I would like all of those defaults to be applied BUT I want to override just the $sandbox_tvm[‘master’][‘version’] setting.

Now, consider the following class:

$cat modules/sandbox/manifests/tvm.pp
#
# A private puppet sandbox for TVM.
#
class sandbox::tvm {

  notify { 'sandbox::tvm::1':
    message => "sandbox::tvm initialized.\n"
   }

  # experiment with true hierarchical types.
  $sandbox = hiera('sandbox_tvm', 'undefined')

  if ($sandbox == 'undefined') {
    notify { 'sandbox::tvm::2':
      message => "No \$sandbox defined\n"
    }
  } else {
    notify { 'sandbox::tvm:3':
      message => "Sandbox is defined\n"
    }
  }

  if ($sandbox['master'] == '') {
    notify { 'sandbox::tvm:4':
      message => "No \$sandbox['master'] defined\n",
    }
  } else {
    if ($sandbox['master']['version'] == '') {
      notify { 'sandbox::tvm:5':
        message => "No \$sandbox['master']['version'] defined\n",
      }
    } else {
      notify { 'sandbox::tvm:6':
        message => "\$sandbox['master']['version'] is #{$sandbox['master']['version']}\n",
      }
    }

    if ($sandbox['master']['name'] == '') {
      notify { 'sandbox::tvm:7':
        message => "No \$sandbox['master']['name'] defined\n",
      }
    } else {
      notify { 'sandbox::tvm:8':
        message => "\$sandbox['master']['name'] is #{$sandbox['master']['name']}\n"
      }
    }
  }

  if ($sandbox['client'] == '') {
    notify { 'sandbox::tvm:9':
      message => "No \$sandbox['client'] defined\n",
    }
  } else {
    if ($sandbox['client']['version'] == '') {
      notify { 'sandbox::tvm:10':
        message => "No \$sandbox['client']['version'] defined\n",
      }
    } else {
      notify { 'sandbox::tvm:11':
        message => "\$sandbox['client']['version'] is #{$sandbox['client']['version']}\n",
      }
    }
  }
}

# vim:sm:ts=2:expandtab

Finally, apply that class to foo.bar.edu:

$sudo puppet agent --test
info: Retrieving plugin
info: Loading facts in /var/lib/puppet/lib/facter/ipmi_product.rb
info: Loading facts in /var/lib/puppet/lib/facter/net_location.rb
info: Loading facts in /var/lib/puppet/lib/facter/root_home.rb
info: Loading facts in /var/lib/puppet/lib/facter/biosversion.rb
info: Loading facts in /var/lib/puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/lib/puppet/lib/facter/pacemaker.rb
info: Loading facts in /var/lib/puppet/lib/facter/net_info.rb
info: Caching catalog for foo.bar.edu
info: Applying configuration version '24278'
notice: sandbox::tvm initialized.

notice: /Stage[main]/Sandbox::Tvm/Notify[sandbox::tvm::1]/message: defined 'message' as 'sandbox::tvm initialized.
'
notice: $sandbox['master']['version'] is #{masterversion1.2.4['master']['version']}

notice: /Stage[main]/Sandbox::Tvm/Notify[sandbox::tvm:6]/message: defined 'message' as '$sandbox['master']['version'] is #{masterversion1.2.4['master']['version']}
'
notice: No $sandbox['master']['name'] defined

notice: /Stage[main]/Sandbox::Tvm/Notify[sandbox::tvm:7]/message: defined 'message' as 'No $sandbox['master']['name'] defined
'
notice: Sandbox is defined

notice: /Stage[main]/Sandbox::Tvm/Notify[sandbox::tvm:3]/message: defined 'message' as 'Sandbox is defined
'
notice: No $sandbox['client'] defined

notice: /Stage[main]/Sandbox::Tvm/Notify[sandbox::tvm:9]/message: defined 'message' as 'No $sandbox['client'] defined
'
notice: Finished catalog run in 97.52 seconds

The notify()’s are a bit of a mess, but hopefully it’s clear what’s happening:

fqdn/foo.bar.edu.yaml only specifies part of the nested data structure for sandbox_tvm and hiera isn’t doing any kind of a merge of the keys from various parts of the hierarchy, so it only sees what is specified in fqdn/foo.bar.edu.yaml.

I could see several places where some kind of a hiera_merge (or, as Eric suggested, maybe not even have it be a special call, and just have hiera() act this way by default) would be really useful. For example:

common.yaml:

ntp:
  type: 'client'
  servers:
    - '10.1.2.3'
    - '10.1.2.4'
  allow_query:
    - '10.1.0.0/255.255.0.0'
    - '10.2.0.0/255.255.0.0'
    - '10.3.0.0/255.255.0.0'
    - '2001:4930::/ffff:ffff::'

location/datacenter1.yaml:

ntp:
  servers:
    - '10.9.1.1'
    - '10.9.1.2'

fqdn/timeserver.bar.edu.yaml:

ntp:
  type: server
  servers:
    - '199.7.177.206'
    - '69.122.22.85'
  peers:
    - '10.1.2.3'

In other words:

default to being an ntp client with servers ‘10.1.2.3’ and ‘10.1.2.4’ and allowed queries from a certain range. For all systems with a location=datacenter1, keep the defaults for type and allow_queries, but instead use a different pair of servers, ‘10.9.1.1’ and ‘10.9.1.2’. Finally, for fqdn=timeserver.bar.edu, override the type to be server, override the server list, and add the key “peers”. Keep the default allow_query.

There are many reasons to consider using nested data structures for configuration information, but one of the things that got me looking at this was the possibility of eventually publishing some of our modules to the puppet module forge. If we can nest data structures, we can essentially “namespace” our config settings and avoid (not completely, but more likely) the possibility of conflicting with some other puppet module that happened to choose the exact same for some setting. Now one could always use hiera like this instead:

mysite_ntp_type: client
mysite_ntp_servers:
  - 10.1.2.3
  - 10.1.2.4
mysite_ntp_allow_query:
  - blah
  - more_blah

but considering hiera supports serializing very complicated structures, it seems artificially limiting to have to resort to things like this for puppet.


Related issues

Related to Hiera - Feature #18793: Hiera deep merge on hash doesn't have a "whiteout" or "re... Needs More Information

History

#1 Updated by eric sorenson over 3 years ago

  • Category set to hiera-puppet
  • Status changed from Unreviewed to Investigating
  • Assignee set to eric sorenson
  • Keywords set to hash

#2 Updated by Anonymous over 3 years ago

  • Category changed from hiera-puppet to hiera
  • Status changed from Investigating to Merged - Pending Release
  • Assignee deleted (eric sorenson)
  • Target version set to 2.0.0

Merged into master

As 7ce67c2. This currently won’t be released until Hiera 2.0 is released. It is backwards compatible with 1.x, however, so we could merge this into the 1.x branch if need be.

-Jeff

#3 Updated by Anonymous about 3 years ago

  • Target version changed from 2.0.0 to 1.2.0

Merged into 1.x as 6822a1f.

This should be released in 1.2.

Thanks again for the contribution!

-Jeff

#4 Updated by Anonymous about 3 years ago

  • Description updated (diff)

Updated the formatting of the description to place the code examples in pre blocks

#5 Updated by Matthaus Owens about 3 years ago

  • Status changed from Merged - Pending Release to Closed

Released in Hiera 1.2.0-rc1

Also available in: Atom PDF