The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #14759

state.yaml balooning when using dictionary as resource $name

Added by Rich Rauenzahn almost 2 years ago. Updated almost 2 years ago.

Status:Needs More InformationStart date:05/30/2012
Priority:NormalDue date:
Assignee:Rich Rauenzahn% Done:

0%

Category:-
Target version:-
Affected Puppet version: Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket may be automatically exported to the PUP project on JIRA using the button below:


Description

We had some puppet processes ballooning and I spent some time trying to figure out why. Turns out our state.yaml is something like 500MB after the run.

Examining the state.yaml shows a particular resource ballooning that we use for iterating over a moderately large list of dictionaries.

We use a custom function to generate a list of dictionaries, and then we pass the dictionary to the resource, the resource picks apart the dictionary, filters based on some keys, etc., and creates other resources.

I verified the custom function is behaving appropriately — it creates something around 180 items. The state.yaml balloons it up to about 400,000 – kind of a cartesian product I think.

oversimplified example:

$list = generate_list()

define process_list {

if $list[‘foo’] == ‘xxxx’ {

   file {
       ...
   }

} }

What I found in the state.yaml was each element (a dict) of $list stringified and listed as Process_List[] multiple times, but the stringification listed the dictionary in different orders, causing the same entry to be duplicated, but represented as a different strinfication of the dict.

My guess is that internally Puppet is stringifying the dictionary in multiple locations and this causes duplicate resources to be defined when creating the state to be saved in the state.yaml (I don’t think it is doing that for determining resources to be created; otherwise we’d be seeing duplication errors).

Anyway, I know it is kind of an odd case we’re using here, but the ballooning is so severe (~4GB) I wanted to report it.

History

#1 Updated by Rich Rauenzahn almost 2 years ago

$ puppet —version 2.7.1

#2 Updated by Rich Rauenzahn almost 2 years ago

Anyone looking for a workaround, instead of passing a list of dicts to iterate over a list of data in order to create resources like this…

define a_resource { $foo = $name[‘key1’] $bar = $name[‘key2’] }

a_resource { $my_list_of_dicts: }

Write a custom function that transforms your dict into something like:

$enumed = [ $list_of_keys, $dict ]

Where $list_of_keys is something like [ 1,2,3,4 ] and $dict is { 1: { … }, 2: {…} }

Then change a_resource to

define a_resource($key=$name, $dict) { $foo = $dict[$key][‘key1’] $bar = $dict[$key][‘key2’] }

a_resource { $enumed[0]: dict => $enumed[1] }

Which is really the better way to be doing this anyway rather than relying on puppet allowing $name to be a dict.

#3 Updated by Kelsey Hightower almost 2 years ago

  • Description updated (diff)
  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Rich Rauenzahn

Rich,

Thanks for reporting this. You’ve got me really curious now. I’ve noticed that you’re running Puppet 2.7.1; are you in a position to test this on never versions of Puppet? I would like to know if this is an issue on later versions of Puppet.

#4 Updated by Rich Rauenzahn almost 2 years ago

Let me see if I can make an isolated test case. Can I make a state.yaml be created from running a puppet /tmp/foo.pp?

Also, I should mention this list of hashes of ours is often created by reading in some json in a custom function, parsing it with pson, and returning the object to puppet.

#5 Updated by Rich Rauenzahn almost 2 years ago

So, I had to use the PSON parser to get this to reproduce:

I added this to one of our nodes temporarily:

$json = '[{"lookforme9": 9, "lookforme8": 8, "lookforme3": 3, "lookforme2": 2, "lookforme1": 1, "lookforme0": 0, "lookforme7": 7, "lookforme6": 6, "lookf
orme5": 5, "lookforme4": 4}, {"lookforme9": 19, "lookforme8": 18, "lookforme3": 13, "lookforme2": 12, "lookforme1": 11, "lookforme0": 10, "lookforme7": 17, "
lookforme6": 16, "lookforme5": 15, "lookforme4": 14}, {"lookforme9": 29, "lookforme8": 28, "lookforme3": 23, "lookforme2": 22, "lookforme1": 21, "lookforme0"
: 20, "lookforme7": 27, "lookforme6": 26, "lookforme5": 25, "lookforme4": 24}, {"lookforme9": 39, "lookforme8": 38, "lookforme3": 33, "lookforme2": 32, "look
forme1": 31, "lookforme0": 30, "lookforme7": 37, "lookforme6": 36, "lookforme5": 35, "lookforme4": 34}, {"lookforme9": 49, "lookforme8": 48, "lookforme3": 43
, "lookforme2": 42, "lookforme1": 41, "lookforme0": 40, "lookforme7": 47, "lookforme6": 46, "lookforme5": 45, "lookforme4": 44}, {"lookforme9": 59, "lookform
e8": 58, "lookforme3": 53, "lookforme2": 52, "lookforme1": 51, "lookforme0": 50, "lookforme7": 57, "lookforme6": 56, "lookforme5": 55, "lookforme4": 54}, {"l
ookforme9": 69, "lookforme8": 68, "lookforme3": 63, "lookforme2": 62, "lookforme1": 61, "lookforme0": 60, "lookforme7": 67, "lookforme6": 66, "lookforme5": 6
5, "lookforme4": 64}, {"lookforme9": 79, "lookforme8": 78, "lookforme3": 73, "lookforme2": 72, "lookforme1": 71, "lookforme0": 70, "lookforme7": 77, "lookfor
me6": 76, "lookforme5": 75, "lookforme4": 74}, {"lookforme9": 89, "lookforme8": 88, "lookforme3": 83, "lookforme2": 82, "lookforme1": 81, "lookforme0": 80, "
lookforme7": 87, "lookforme6": 86, "lookforme5": 85, "lookforme4": 84}, {"lookforme9": 99, "lookforme8": 98, "lookforme3": 93, "lookforme2": 92, "lookforme1"
: 91, "lookforme0": 90, "lookforme7": 97, "lookforme6": 96, "lookforme5": 95, "lookforme4": 94}]'
$items = our_json_parse($json)
notice($items)
define iterate_list {
   $x = $name['lookforme0']
   file { "/tmp/tmp.$x":
      ensure => file,
      content => "hello world",
   }
}

iterate_list { $items:
}

our_json_parse is basically this, although we add checks for exceptions, etc.

(untested…)

module Puppet::Parser::Functions
   newfunction(:our_json_parse, :type => :rvalue, :doc => "
Parse json string and return result.
") do |args|
   begin
      PSON.parse(json)
   end
end

Here is the state file, which has duplicates:

$ sudo grep lookforme /var/lib/puppet/state/state.yaml
"Ournode::Iterate_list[lookforme595lookforme292lookforme797lookforme898lookforme191lookforme999lookforme494lookforme393lookforme696lookforme090]": 
"Ournode::Iterate_list[lookforme747lookforme848lookforme141lookforme949lookforme444lookforme343lookforme646lookforme040lookforme545lookforme242]": 
"Ournode::Iterate_list[lookforme424lookforme525lookforme626lookforme727lookforme020lookforme121lookforme222lookforme323lookforme828lookforme929]": 
"Ournode::Iterate_list[lookforme444lookforme545lookforme646lookforme747lookforme040lookforme141lookforme242lookforme343lookforme848lookforme949]": 
"Ournode::Iterate_list[lookforme313lookforme616lookforme010lookforme515lookforme212lookforme717lookforme818lookforme111lookforme919lookforme414]": 
"Ournode::Iterate_list[lookforme494lookforme595lookforme696lookforme797lookforme090lookforme191lookforme292lookforme393lookforme898lookforme999]": 
"Ournode::Iterate_list[lookforme434lookforme535lookforme636lookforme737lookforme030lookforme131lookforme232lookforme333lookforme838lookforme939]": 
"Ournode::Iterate_list[lookforme535lookforme232lookforme737lookforme838lookforme131lookforme939lookforme434lookforme333lookforme636lookforme030]": 
"Ournode::Iterate_list[lookforme151lookforme959lookforme454lookforme353lookforme656lookforme050lookforme555lookforme252lookforme757lookforme858]": 
"Ournode::Iterate_list[lookforme969lookforme464lookforme363lookforme666lookforme060lookforme565lookforme262lookforme767lookforme868lookforme161]": 
"Ournode::Iterate_list[lookforme414lookforme515lookforme616lookforme717lookforme010lookforme111lookforme212lookforme313lookforme818lookforme919]": 
"Ournode::Iterate_list[lookforme464lookforme565lookforme666lookforme767lookforme060lookforme161lookforme262lookforme363lookforme868lookforme969]": 
"Ournode::Iterate_list[lookforme676lookforme070lookforme575lookforme272lookforme777lookforme878lookforme171lookforme979lookforme474lookforme373]": 
"Ournode::Iterate_list[lookforme020lookforme525lookforme222lookforme727lookforme828lookforme121lookforme929lookforme424lookforme323lookforme626]": 
"Ournode::Iterate_list[lookforme080lookforme585lookforme282lookforme787lookforme888lookforme181lookforme989lookforme484lookforme383lookforme686]": 
"Ournode::Iterate_list[lookforme44lookforme55lookforme66lookforme77lookforme00lookforme11lookforme22lookforme33lookforme88lookforme99]": 
"Ournode::Iterate_list[lookforme454lookforme555lookforme656lookforme757lookforme050lookforme151lookforme252lookforme353lookforme858lookforme959]": 
"Ournode::Iterate_list[lookforme99lookforme44lookforme33lookforme66lookforme00lookforme55lookforme22lookforme77lookforme88lookforme11]": 
"Ournode::Iterate_list[lookforme484lookforme585lookforme686lookforme787lookforme080lookforme181lookforme282lookforme383lookforme888lookforme989]": 
"Ournode::Iterate_list[lookforme474lookforme575lookforme676lookforme777lookforme070lookforme171lookforme272lookforme373lookforme878lookforme979]": 
$ sudo grep lookforme /var/lib/puppet/state/state.yaml | wc -l
20

The json was generated with:

#!/usr/bin/python                                                               
import json                                                                     
keys = []                                                                       
    for k in range(0,10):                                                           
        keys.append("lookforme%d" % k)                                              
items = []                                                                      
v = 0                                                                           
for count in range(0,10):                                                       
    d = {}                                                                      
    for key in keys:                                                            
        d[key] = v                                                              
        v += 1                                                                  
    items.append(d)                                                             

print json.dumps(items)    

Also available in: Atom PDF