Puppet

Earlier today I discovered that all of my nodes were being drained of their /etc/hosts entries and investigative work with dpittman@ helped me track it down to a storeconfig issue.

Based on this chunk of the postgresql log:

<pre>
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='host') AND puppet_tags.name = 'ec2private' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='host') AND puppet_tags.name = 'regular' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='mysql_database') AND puppet_tags.name = 'mysql_qa-automation1.unity.perimeterusa.com' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='mysql_user') AND puppet_tags.name = 'mysql_qa-automation1.unity.perimeterusa.com' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='mysql_grant') AND puppet_tags.name = 'mysql_qa-automation1.unity.perimeterusa.com' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='host') AND puppet_tags.name = 'ec2private' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='host') AND puppet_tags.name = 'regular' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='mysql_database') AND puppet_tags.name = 'mysql_qa-automation1.unity.perimeterusa.com' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='mysql_user') AND puppet_tags.name = 'mysql_qa-automation1.unity.perimeterusa.com' AND (host_id != 40))
LOG: statement: SELECT "hosts".* FROM "hosts" WHERE "hosts"."name" = 'qa-automation1.unity.perimeterusa.com' LIMIT 1
LOG: statement: SELECT "resources".* FROM "resources" INNER JOIN "resource_tags" ON "resource_tags"."resource_id" = "resources"."id" INNER JOIN "puppet_tags" ON "puppet_tags"."id" = "resource_tags"."puppet_tag_id" WHERE ((exported='t' AND restype='mysql_grant') AND puppet_tags.name = 'mysql_qa-automation1.unity.perimeterusa.com' AND (host_id != 40))
</pre> LOG: statement: SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
FROM pg_attribute a LEFT JOIN pg_attrdef d
ON a.attrelid = d.adrelid AND a.attnum = d.adnum
WHERE a.attrelid = &#x27;&quot;template_kinds&quot;&#x27;::regclass
AND a.attnum &gt; 0 AND NOT a.attisdropped
ORDER BY a.attnum

and the following queries:

select exported, count(exported) from resources group by exported

<pre>
exported | count
----------+-------
f | 3520
t | 31
(2 rows)
</pre>

select distinct restype from resources

<pre>


restype
------------------------
User
Resources
Postgres::Config
Postgres::Createuser
Users::Adduser
Host
Postgres::Initdb
Sqlexec
Exec
Stage
Group
Concat::Fragment
Augeas
Mysql_user
Service
Class
Yumrepo
File
Security::Sudoers::Add
Package
Firewall
Postgres::Enable
Postgres::Createdb
Mysql_grant
Apache2::Site
Cron
Ssh_authorized_key
Concat
Mysql_database
(29 rows)
</pre>

He was able to narrow it down to an issue with:

12:51 <dpittman> Awesome. Found why it isn't matching: restype='host' vs restype='Host'

The temporary workaround is:

You could patch your code in `lib/puppet/indirector/resource/active_record.rb` line 65 to be `arguments = [true, type.to_s.capitalize]`