The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #15608

Updated by Carl Caum over 1 year ago

I'm running Puppet on Ruby 1.9.2p290. Our environment is a mix of Centos 5.0 all the way up to Centos 6.2 systems. Our Puppetmaster is Centos 6.2. All of these servers are working fine.

Somewhat recently Centos 6.3 came out, and we've started trying to roll out new servers on Centos 6.3. But our puppet agents on those systems don't seem to work properly.

The puppet agent is unable to make file requests to the master. Everything else is working fine. File resources that use templates, for example, have no problems. But file resources that actually use source can't seem to retrieve their data anymore. I'm receiving a large number of "getaddrinfo: Name or service not known" errors.

I tried to run a manually retrieval on the file with curl, and got back a successful result:

<pre>
root@lvopsapptest01:/var/lib/puppet$ curl --cert /var/lib/puppet/ssl/certs/lvopsapptest01.lightningsource.com.pem --key /var/lib/puppet/ssl/private_keys/lvopsapptest01.lightningsource.com.pem --cacert /var/lib/puppet/ssl/certs/ca.pem -H 'Accept: pson' https://puppet.lightningsource.com:8140/production/file_metadata/modules/rsyslog/logrotate_rsyslog; echo

{"document_type":"FileMetadata","data":{"path":"/etc/puppet/modules/app/rsyslog/files/logrotate_rsyslog","relative_path":null,"links":"manage","owner":0,"group":0,"mode":420,"checksum":{"type":"md5","value":"{md5}d6a163ab4efcc4ffa07c06eaae197ae8"},"type":"file","destination":null},"metadata":{"api_version":1}}
</pre> &lt;pre&gt;

The result was equally successful for file_content.

I'm fairly certain that these errors aren't consistent. On every server exhibiting the problem, I'm able to verify that all of the files currently reporting that they can't be downloaded *have* been downloaded successfully at some point. The errors also seem to change from run to run. So I believe this may be some sort of a race condition.

Here is a run demonstrating some of the issues. Ive attached back-to-back runs because, after clearing the Puppet cached data, subsequent runs seem to demonstrate different issues:

<pre>
root@lvopsapptest01:/var/lib/puppet$ rm -rf client* state/ lib/ facts/
root@lvopsapptest01:/var/lib/puppet$ puppetd --test
info: Retrieving plugin
notice: /File[/var/lib/puppet/lib/puppet]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/provider]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/reports]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/provider/package]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/parser]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/parser/functions]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/parser/functions/join_array_with_spaces.rb]/ensure: defined content as '{md5}9222f23743d8284500f64f9c0a082535'
notice: /File[/var/lib/puppet/lib/puppet/provider/git_deploy]/ensure: created
notice: /File[/var/lib/puppet/lib/puppet/provider/git_deploy/git_deploy.rb]/ensure: defined content as '{md5}d168768afc5fe4404499ab8c3ff8c9ab'
notice: /File[/var/lib/puppet/lib/puppet/type]/ensure: created
err: /File[/var/lib/puppet/lib/puppet/type/git_deploy.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/puppet/type/config_entry.rb]/ensure: defined content as '{md5}3c2ca896f1b84e9ac8a3eefed1c62e80'
err: /File[/var/lib/puppet/lib/puppet/provider/package/pkgutil.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter]/ensure: created
notice: /File[/var/lib/puppet/lib/facter/facter]/ensure: created
notice: /File[/var/lib/puppet/lib/facter/facter/root_bubba.rb]/ensure: defined content as '{md5}1388f235c6831dbe17379fc2b2186042'
err: /File[/var/lib/puppet/lib/facter/env_vars.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/icinga_enabled.rb]/ensure: defined content as '{md5}b64c8f6e39f11fbb327d961dd95ddd75'
err: /File[/var/lib/puppet/lib/facter/timemachine.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/warranty.rb]/ensure: defined content as '{md5}a47ac3e4f8194b18eb2eda05a0c99a43'
err: /File[/var/lib/puppet/lib/facter/icinga_domain.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/sitecode_lc.rb]/ensure: defined content as '{md5}4ac8ab92c80d73eefa32a42a7e62d90f'
err: /File[/var/lib/puppet/lib/facter/root_bubba.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/sitecode.rb]/ensure: defined content as '{md5}1b5f5e3ec133dd2de0636051b2c35b2b'
err: /File[/var/lib/puppet/lib/facter/icinga_critical.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/mcollectiveversion.rb]/ensure: defined content as '{md5}743f6f2376db9314804ccf0d7e86a5b8'
notice: /File[/var/lib/puppet/lib/puppet/provider/config_entry]/ensure: created
err: /File[/var/lib/puppet/lib/puppet/provider/config_entry/config_entry.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/raidcontroller.rb]/ensure: defined content as '{md5}d93218d35efbb31160ae460ca69cdb8d'
err: /File[/var/lib/puppet/lib/puppet/type/svn_deploy.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/icinga_corporate.rb]/ensure: defined content as '{md5}7f741ab0142ca36d3bf7f62b8d0f12a1'
err: /File[/var/lib/puppet/lib/facter/ha_cluster_dc.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/raidtype.rb]/ensure: defined content as '{md5}f832382be3960c5e4a8ceddd1e9fde2f'
notice: /File[/var/lib/puppet/lib/puppet/provider/svn_deploy]/ensure: created
err: /File[/var/lib/puppet/lib/puppet/provider/svn_deploy/svn_deploy.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/puppet/reports/pagerduty.rb]/ensure: defined content as '{md5}6a40fe3250579bd6f115c73e28f6dd12'
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/parser/functions/join_array_with_spaces.rb
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/provider/git_deploy/git_deploy.rb
err: Could not load downloaded file /var/lib/puppet/lib/puppet/provider/git_deploy/git_deploy.rb: undefined method `provide' for nil:NilClass
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/type/config_entry.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/facter/root_bubba.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/icinga_enabled.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/warranty.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/sitecode_lc.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/sitecode.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/mcollectiveversion.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/raidcontroller.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/icinga_corporate.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/raidtype.rb
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/reports/pagerduty.rb
info: You need the `rest-client` gem to use the PagerDuty report
err: Could not load downloaded file /var/lib/puppet/lib/puppet/reports/pagerduty.rb: PagerDuty report config file /etc/puppet/pagerduty.yaml not readable
err: Could not retrieve catalog from remote server: getaddrinfo: Name or service not known
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
root@lvopsapptest01:/var/lib/puppet$ puppetd --test
info: Retrieving plugin
notice: /File[/var/lib/puppet/lib/puppet/type/svn_deploy.rb.puppettmp_815]/ensure: removed
notice: /File[/var/lib/puppet/lib/puppet/type/git_deploy.rb]/ensure: defined content as '{md5}b1fa7f41f60603260c6e3ee373d03002'
notice: /File[/var/lib/puppet/lib/puppet/type/git_deploy.rb.puppettmp_9257]/ensure: removed
notice: /File[/var/lib/puppet/lib/puppet/provider/package/pkgutil.rb]/ensure: defined content as '{md5}8c2330be94f858ce6bbe346a9262ce82'
notice: /File[/var/lib/puppet/lib/facter/icinga_domain.rb.puppettmp_2899]/ensure: removed
notice: /File[/var/lib/puppet/lib/facter/env_vars.rb.puppettmp_9788]/ensure: removed
notice: /File[/var/lib/puppet/lib/facter/timemachine.rb.puppettmp_3151]/ensure: removed
err: /File[/var/lib/puppet/lib/facter/env_vars.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/icinga_critical.rb.puppettmp_2406]/ensure: removed
notice: /File[/var/lib/puppet/lib/facter/timemachine.rb]/ensure: defined content as '{md5}0df50ff9d9368e6496e938630e8dfe8d'
notice: /File[/var/lib/puppet/lib/facter/ha_cluster_dc.rb.puppettmp_687]/ensure: removed
err: /File[/var/lib/puppet/lib/facter/icinga_domain.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/root_bubba.rb]/ensure: defined content as '{md5}1388f235c6831dbe17379fc2b2186042'
notice: /File[/var/lib/puppet/lib/facter/root_bubba.rb.puppettmp_3309]/ensure: removed
err: /File[/var/lib/puppet/lib/facter/icinga_critical.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/puppet/provider/config_entry/config_entry.rb.puppettmp_501]/ensure: removed
notice: /File[/var/lib/puppet/lib/puppet/provider/config_entry/config_entry.rb]/ensure: defined content as '{md5}c7731d5a12924961ac56ab7d14fa2ca5'
notice: /File[/var/lib/puppet/lib/puppet/provider/package/pkgutil.rb.puppettmp_7368]/ensure: removed
err: /File[/var/lib/puppet/lib/puppet/type/svn_deploy.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/facter/ha_cluster_dc.rb]/ensure: defined content as '{md5}e9da00b5e820d82e226086d98cf3706b'
err: /File[/var/lib/puppet/lib/puppet/provider/svn_deploy/svn_deploy.rb]/ensure: change from absent to file failed: Could not set 'file on ensure: getaddrinfo: Name or service not known
notice: /File[/var/lib/puppet/lib/puppet/provider/svn_deploy/svn_deploy.rb.puppettmp_9990]/ensure: removed
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/type/git_deploy.rb
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/provider/package/pkgutil.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/timemachine.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/root_bubba.rb
info: Loading downloaded plugin /var/lib/puppet/lib/puppet/provider/config_entry/config_entry.rb
info: Loading downloaded plugin /var/lib/puppet/lib/facter/ha_cluster_dc.rb
info: Caching catalog for lvopsapptest01.lightningsource.com
info: Applying configuration version '1342720777'
err: /Stage[main]/Rsyslog/File[/etc/sysconfig/rsyslog]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/rsyslog/sysconfig_rsyslog: getaddrinfo: Name or service not known at /etc/puppet/modules/app/rsyslog/manifests/init.pp:42
err: /Stage[main]/Sudosh/File[/etc/sudosh.conf]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/sudosh/sudosh.conf: getaddrinfo: Name or service not known at /etc/puppet/modules/app/sudosh/manifests/init.pp:7
err: /Stage[main]/Base::Os::Linux/File[/etc/sysconfig/ntpd]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/base/sysconfig-ntpd: getaddrinfo: Name or service not known at /etc/puppet/modules/local/base/manifests/os/linux.pp:168
err: /Stage[main]/Authentication::Winbind/File[/etc/pam.d/system-auth-winbind]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/authentication/system-auth-winbind: getaddrinfo: Name or service not known at /etc/puppet/modules/local/authentication/manifests/winbind.pp:43
notice: /Stage[main]/Authentication::Winbind/File[/etc/pam.d/system-auth]: Dependency File[/etc/pam.d/system-auth-winbind] has failures: true
warning: /Stage[main]/Authentication::Winbind/File[/etc/pam.d/system-auth]: Skipping because of failed dependencies
err: /Stage[main]/Base::Os::Unix/File[/var/lib/puppet/ssl/certs/ca.pem]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/base/ca.pem: getaddrinfo: Name or service not known at /etc/puppet/modules/local/base/manifests/os/unix.pp:29
notice: /Stage[main]/Base::Os::Unix/File[/etc/pki/tls/certs/b88fe4d1.0]: Dependency File[/var/lib/puppet/ssl/certs/ca.pem] has failures: true
warning: /Stage[main]/Base::Os::Unix/File[/etc/pki/tls/certs/b88fe4d1.0]: Skipping because of failed dependencies
err: /Stage[main]/Oracle::Client/File[/usr/lib/oracle/11.1.0.1/client64//network/admin/SQLNET.ORA]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/oracle/SQLNET.ORA: getaddrinfo: Name or service not known at /etc/puppet/modules/app/oracle/manifests/client.pp:50
err: /Stage[main]/Oracle::Client/File[/usr/lib/oracle/11.1.0.1/client//network/admin/SQLNET.ORA]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/oracle/SQLNET.ORA: getaddrinfo: Name or service not known at /etc/puppet/modules/app/oracle/manifests/client.pp:50
notice: /Stage[main]/Puppet/Puppet::Client[client]/File[/var/lib/puppet/client_yaml]/group: group changed 'root' to 'nagios'
notice: /File[/var/lib/puppet/client_yaml/catalog]/group: group changed 'root' to 'nagios'
notice: /File[/var/lib/puppet/client_yaml/catalog/lvopsapptest01.lightningsource.com.yaml]/group: group changed 'root' to 'nagios'
notice: /Stage[main]/Rsyslog/Service[rsyslog]: Dependency File[/etc/sysconfig/rsyslog] has failures: true
warning: /Stage[main]/Rsyslog/Service[rsyslog]: Skipping because of failed dependencies
err: /Stage[main]/Cups/File[/etc/cups/cupsd.conf]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/cups/allowall-cupsd.conf: getaddrinfo: Name or service not known at /etc/puppet/modules/app/cups/manifests/init.pp:23
notice: /Stage[main]/Cups/Service[cups]: Dependency File[/etc/cups/cupsd.conf] has failures: true
warning: /Stage[main]/Cups/Service[cups]: Skipping because of failed dependencies
err: /Stage[main]/Mcollective::Server/File[/usr/libexec/mcollective/mcollective//agent/]: Failed to generate additional resources using 'eval_generate: getaddrinfo: Name or service not known
err: /Stage[main]/Mcollective::Server/Mcollective::Agent_exec[mc-service]/File[/usr/bin/mc-service]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/mcollective/agent-execs/mc-service: getaddrinfo: Name or service not known at /etc/puppet/modules/app/mcollective/manifests/agent_exec.pp:15
err: /Stage[main]/Mcollective::Server/Mcollective::Agent_exec[mc-pgrep]/File[/usr/bin/mc-pgrep]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/mcollective/agent-execs/mc-pgrep: getaddrinfo: Name or service not known at /etc/puppet/modules/app/mcollective/manifests/agent_exec.pp:15
err: /Stage[main]/Mcollective::Server/Mcollective::Agent_exec[mc-package]/File[/usr/bin/mc-package]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/mcollective/agent-execs/mc-package: getaddrinfo: Name or service not known at /etc/puppet/modules/app/mcollective/manifests/agent_exec.pp:15
err: /Stage[main]/Mcollective::Server/Mcollective::Agent_exec[mc-puppetd]/File[/usr/bin/mc-puppetd]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/mcollective/agent-execs/mc-puppetd: getaddrinfo: Name or service not known at /etc/puppet/modules/app/mcollective/manifests/agent_exec.pp:15
err: /Stage[main]/Mcollective::Server/Mcollective::Agent_exec[mc-nrpe]/File[/usr/bin/mc-nrpe]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/mcollective/agent-execs/mc-nrpe: getaddrinfo: Name or service not known at /etc/puppet/modules/app/mcollective/manifests/agent_exec.pp:15
err: /Stage[main]/Apache/File[/etc/httpd//conf.d/php.conf]: Could not evaluate: getaddrinfo: Name or service not known Could not retrieve file metadata for puppet:///modules/php/php.conf: getaddrinfo: Name or service not known at /etc/puppet/modules/app/apache/manifests/init.pp:39
notice: /Stage[main]/Apache/Service[httpd]: Dependency File[/etc/httpd//conf.d/php.conf] has failures: true
warning: /Stage[main]/Apache/Service[httpd]: Skipping because of failed dependencies
info: Creating state file /var/lib/puppet/state/state.yaml
notice: Finished catalog run in 30.90 seconds
root@lvopsapptest01:/var/lib/puppet$
</pre>

Back