Puppet

i’ve been trying to mitigate them mostly on 2.2.7, but then read #2386 and downgraded to 2.2.2 – at least (1) + (2) + (3) + (4) also appeared here…

Any comments Christian?

how it goes really bad – 5 consecutive runs from the same client. those were served by 1 or maximum 2 of passenger threads (passenger 2.2.2).

[root@tweetie puppet]# while true ; do puppetd --test  ; echo --- ; sleep 10 ;  done
notice: Ignoring --listen on onetime run
info: Retrieving plugin
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Caching catalog for tweetie.looney.acme.com
info: Applying configuration version '1260309419'
notice: Finished catalog run in 13.24 seconds
---
notice: Ignoring --listen on onetime run
info: Retrieving plugin
err: /File[/var/puppet/lib]: Failed to retrieve current state of resource: Could not retrieve information from source(s) puppet://puppet/plugins
notice: /File[/var/puppet/lib/facter]: Dependency file[/var/puppet/lib] has 1 failures
warning: /File[/var/puppet/lib/facter]: Skipping because of failed dependencies
notice: /File[/var/puppet/lib/facter/mounts.rb]: Dependency file[/var/puppet/lib] has 1 failures
warning: /File[/var/puppet/lib/facter/mounts.rb]: Skipping because of failed dependencies
notice: /File[/var/puppet/lib/facter/passenger_version.rb]: Dependency file[/var/puppet/lib] has 1 failures
warning: /File[/var/puppet/lib/facter/passenger_version.rb]: Skipping because of failed dependencies
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Loading facts in passenger_version
info: Loading facts in mounts
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not find class user at /etc/puppet/site/manifests/nodes.pp:2263 on node tweetie.looney.acme.com
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
---
notice: Ignoring --listen on onetime run
info: Retrieving plugin
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Loading facts in passenger_version
info: Loading facts in mounts
err: Could not retrieve catalog from remote server: Error 400 on SERVER: database is locked
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run
---
notice: Ignoring --listen on onetime run
info: Retrieving plugin
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Caching catalog for tweetie.looney.acme.com
info: Applying configuration version '1260309477'
notice: Finished catalog run in 13.13 seconds
---
notice: Ignoring --listen on onetime run
info: Retrieving plugin
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Loading facts in passenger_version
info: Loading facts in mounts
info: Caching catalog for tweetie.looney.acme.com
info: Applying configuration version '1260309458'
err: //nagios::check::puppet/File[check_puppet]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/nagios/check_puppet Could not retrieve fil2
err: //ntp/File[/etc/ntpd.conf]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/ntp/tweetie/conf Could not retrieve file metadata for puppet:6
notice: //ntp/Service[ntpd]: Dependency file[/etc/ntpd.conf] has 1 failures
warning: //ntp/Service[ntpd]: Skipping because of failed dependencies
err: //puppet::puppet/File[/etc/init.d/puppet]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/puppet/puppet.init Could not retrieve file 8
err: //snmp/File[/etc/snmp/snmpd.conf]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/snmp/tweetie/snmpd.conf Could not retrieve file metada5
err: //denyhosts/File[/var/lib/denyhosts/allowed-hosts]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/denyhosts/allowed-hosts Could not 5
notice: //denyhosts/Service[denyhosts]: Dependency file[/var/lib/denyhosts/allowed-hosts] has 1 failures
warning: //denyhosts/Service[denyhosts]: Skipping because of failed dependencies
err: //sudo/File[sudoers]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/sudo/tweetie/sudoers Could not retrieve file metadata for puppet://7
err: //yum::epel/File[/var/tmp/epel.asc]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/yum/epel.asc Could not retrieve file metadata for0
notice: //yum::epel/Yumrepocommit:epel]: Dependency file[/var/tmp/epel.asc has 1 failures
warning: //yum::epel/Yumrepo[epel]: Skipping because of failed dependencies
notice: //puppet::puppet/Package[augeas]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //puppet::puppet/Package[augeas]: Skipping because of failed dependencies
notice: //ruby::rails/Package[rubygem-mongrel_cluster]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Package[rubygem-mongrel_cluster]: Skipping because of failed dependencies
notice: //ruby::gem/Package[gcc]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem/Package[gcc]: Skipping because of failed dependencies
notice: //ruby::gem::prereq-rmagick/Package[freetype-devel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::prereq-rmagick/Package[freetype-devel]: Skipping because of failed dependencies
notice: //ruby::gem::prereq-mechanize/Package[libxml2-devel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::prereq-mechanize/Package[libxml2-devel]: Skipping because of failed dependencies
notice: //ruby::gem::prereq-mysql/Package[mysql-devel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::prereq-mysql/Package[mysql-devel]: Skipping because of failed dependencies
notice: //ruby::gem::prereq-mechanize/Package[libxslt-devel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::prereq-mechanize/Package[libxslt-devel]: Skipping because of failed dependencies
notice: //ruby::gem::prereq-rmagick/Package[msttcorefonts]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::prereq-rmagick/Package[msttcorefonts]: Skipping because of failed dependencies
notice: //ruby::gem::prereq-rmagick/Package[ImageMagick-devel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::prereq-rmagick/Package[ImageMagick-devel]: Skipping because of failed dependencies
notice: //puppet::puppet/Package[ruby-shadow]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //puppet::puppet/Package[ruby-shadow]: Skipping because of failed dependencies
notice: //puppet::puppet/Package[puppet]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //puppet::puppet/Package[puppet]: Skipping because of failed dependencies
notice: //puppet::puppet/Cron[puppet]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //puppet::puppet/Cron[puppet]: Skipping because of failed dependencies
notice: //ruby::gem/Package[ruby-devel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem/Package[ruby-devel]: Skipping because of failed dependencies
notice: //ruby::rails/Package[rubygem-mongrel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Package[rubygem-mongrel]: Skipping because of failed dependencies
notice: //ruby::rails/Package[rubygem-fastthread]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Package[rubygem-fastthread]: Skipping because of failed dependencies
notice: //ruby::rails/Package[rubygem-rails]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Package[rubygem-rails]: Skipping because of failed dependencies
notice: //ruby::gem/Package[rubygems]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem/Package[rubygems]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[daemons]/Exec[install-daemons-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[daemons]/Exec[install-daemons-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[hpricot]/Exec[install-hpricot-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[hpricot]/Exec[install-hpricot-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[rspec]/Exec[install-rspec-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[rspec]/Exec[install-rspec-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[will_paginate]/Exec[install-will_paginate-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[will_paginate]/Exec[install-will_paginate-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[gem_plugin]/Exec[install-gem_plugin-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[gem_plugin]/Exec[install-gem_plugin-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[rake]/Exec[install-rake-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[rake]/Exec[install-rake-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[rufus-scheduler]/Exec[install-rufus-scheduler-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[rufus-scheduler]/Exec[install-rufus-scheduler-gem]: Skipping because of failed dependencies
notice: //ruby::gem::rmagick/Ruby::Gem::Install[rmagick]/Exec[install-rmagick-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::rmagick/Ruby::Gem::Install[rmagick]/Exec[install-rmagick-gem]: Skipping because of failed dependencies
notice: //ruby::gem::mysql/Ruby::Gem::Install[mysql]/Exec[install-mysql-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::mysql/Ruby::Gem::Install[mysql]/Exec[install-mysql-gem]: Skipping because of failed dependencies
notice: //ruby::rails/Ruby::Gem::Install[cgi_multipart_eof_fix]/Exec[install-cgi_multipart_eof_fix-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Ruby::Gem::Install[cgi_multipart_eof_fix]/Exec[install-cgi_multipart_eof_fix-gem]: Skipping because of failed dependencies
notice: //ruby::gem::mechanize/Ruby::Gem::Install[mechanize]/Exec[install-mechanize-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::mechanize/Ruby::Gem::Install[mechanize]/Exec[install-mechanize-gem]: Skipping because of failed dependencies
err: //nagios::check::puppet/File[monitor_puppet]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/puppet/monitor.py Could not retrieve fil2
err: //puppet::puppet/File[/etc/sysconfig/puppet]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/puppet/sysconfig Could not retrieve file7
err: //ruby::passenger/File[/etc/init.d/passenger]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/ruby/passenger.init Could not retrieve 0
notice: //yum/Exec[yum_clean]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //yum/Exec[yum_clean]: Skipping because of failed dependencies
err: //ruby::rails/File[/etc/logrotate.d/mongrel]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/ruby/logrotate Could not retrieve file m1
notice: //ruby::rails/Service[mongrel]: Dependency file[/etc/logrotate.d/mongrel] has 1 failures
notice: //ruby::rails/Service[mongrel]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::rails/Service[mongrel]: Skipping because of failed dependencies
err: //denyhosts/File[/usr/share/denyhosts/data/allowed-hosts]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/denyhosts/allowed-hosts Cou7
err: //denyhosts/File[/usr/share/denyhosts/data/restricted-usernames]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/denyhosts/restricted2
err: //snmp/File[jvm.mib]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/snmp/JVM-MANAGEMENT-MIB.mib Could not retrieve file metadata for5
notice: //snmp/Service[snmpd]: Dependency file[/usr/share/snmp/mibs/JVM-MANAGEMENT-MIB.txt] has 1 failures
notice: //snmp/Service[snmpd]: Dependency file[/etc/snmp/snmpd.conf] has 1 failures
warning: //snmp/Service[snmpd]: Skipping because of failed dependencies
err: //yum::endpoint/File[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/yum/endpoint.asc Could no5
notice: //yum::endpoint/Exec[endpoint_import]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //yum::endpoint/Exec[endpoint_import]: Skipping because of failed dependencies
notice: //yum::endpoint/Yumrepo[endpoint]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //yum::endpoint/Yumrepo[endpoint]: Skipping because of failed dependencies
notice: //ruby::enterprise/Package[ruby-enterprise-rubygems]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //ruby::enterprise/Package[ruby-enterprise-rubygems]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[rufus-scheduler]/Exec[install-rufus-scheduler-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[rufus-scheduler]/Exec[install-rufus-scheduler-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[rufus-scheduler]/Exec[install-rufus-scheduler-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[rake]/Exec[install-rake-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[rake]/Exec[install-rake-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[rake]/Exec[install-rake-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[hpricot]/Exec[install-hpricot-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[hpricot]/Exec[install-hpricot-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[hpricot]/Exec[install-hpricot-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[daemons]/Exec[install-daemons-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[daemons]/Exec[install-daemons-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[daemons]/Exec[install-daemons-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[gem_plugin]/Exec[install-gem_plugin-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[gem_plugin]/Exec[install-gem_plugin-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[gem_plugin]/Exec[install-gem_plugin-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-installcommit:rspec]/Exec[install-rspec-enterprise-gem: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-installcommit:rspec]/Exec[install-rspec-enterprise-gem: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-installcommit:rspec]/Exec[install-rspec-enterprise-gem: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[cgi_multipart_eof_fix]/Exec[install-cgi_multipart_eof_fix-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[cgi_multipart_eof_fix]/Exec[install-cgi_multipart_eof_fix-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[cgi_multipart_eof_fix]/Exec[install-cgi_multipart_eof_fix-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-installcommit:rails]/Exec[install-rails-enterprise-gem: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-installcommit:rails]/Exec[install-rails-enterprise-gem: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-installcommit:rails]/Exec[install-rails-enterprise-gem: Skipping because of failed dependencies
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[will_paginate]/Exec[install-will_paginate-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Ruby::Gem::Enterprise-install[will_paginate]/Exec[install-will_paginate-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::passenger/Ruby::Gem::Enterprise-install[will_paginate]/Exec[install-will_paginate-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::enterprise/Package[ruby-enterprise]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //ruby::enterprise/Package[ruby-enterprise]: Skipping because of failed dependencies
notice: //ruby::gem::enterprise-rmagick/Ruby::Gem::Enterprise-install[rmagick]/Exec[install-rmagick-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::gem::enterprise-rmagick/Ruby::Gem::Enterprise-install[rmagick]/Exec[install-rmagick-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::enterprise-rmagick/Ruby::Gem::Enterprise-install[rmagick]/Exec[install-rmagick-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::gem::enterprise-mechanize/Ruby::Gem::Enterprise-install[mechanize]/Exec[install-mechanize-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::gem::enterprise-mechanize/Ruby::Gem::Enterprise-install[mechanize]/Exec[install-mechanize-enterprise-gem]: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::enterprise-mechanize/Ruby::Gem::Enterprise-install[mechanize]/Exec[install-mechanize-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::gem::enterprise-mysql/Ruby::Gem::Enterprise-installcommit:mysql]/Exec[install-mysql-enterprise-gem: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::gem::enterprise-mysql/Ruby::Gem::Enterprise-installcommit:mysql]/Exec[install-mysql-enterprise-gem: Dependency file[/var/tmp/epel.asc] has 1 failures
warning: //ruby::gem::enterprise-mysql/Ruby::Gem::Enterprise-installcommit:mysql]/Exec[install-mysql-enterprise-gem: Skipping because of failed dependencies
err: //puppet::puppet/File[/etc/puppet/puppetd.conf]: Failed to retrieve current state of resource: Error 400 on SERVER: Not authorized to call find on /file_metadata/puppet/tweetie/puppetd.conf Could not re8
notice: //puppet::puppet/Service[puppet]: Dependency file[/var/tmp/epel.asc] has 1 failures
notice: //puppet::puppet/Service[puppet]: Dependency file[/etc/puppet/puppetd.conf] has 1 failures
notice: //puppet::puppet/Service[puppet]: Dependency file[/etc/init.d/puppet] has 1 failures
notice: //puppet::puppet/Service[puppet]: Dependency file[/etc/sysconfig/puppet] has 1 failures
warning: //puppet::puppet/Service[puppet]: Skipping because of failed dependencies
notice: //ruby::passenger/Exec[install-passenger-enterprise-gem]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //ruby::passenger/Exec[install-passenger-enterprise-gem]: Skipping because of failed dependencies
notice: //ruby::passenger/Exec[mod_passenger_build]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //ruby::passenger/Exec[mod_passenger_build]: Skipping because of failed dependencies
notice: //ruby::passenger/File[/etc/httpd/conf.d/mod_passenger.conf]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
warning: //ruby::passenger/File[/etc/httpd/conf.d/mod_passenger.conf]: Skipping because of failed dependencies
notice: //ruby::passenger/Service[passenger]: Dependency file[/etc/pki/rpm-gpg/RPM-GPG-KEY-endpoint] has 1 failures
notice: //ruby::passenger/Service[passenger]: Dependency file[/etc/logrotate.d/mongrel] has 1 failures
notice: //ruby::passenger/Service[passenger]: Dependency file[/var/tmp/epel.asc] has 1 failures
notice: //ruby::passenger/Service[passenger]: Dependency file[/etc/init.d/passenger] has 1 failures
warning: //ruby::passenger/Service[passenger]: Skipping because of failed dependencies
notice: Finished catalog run in 5.19 seconds

All the auth problems should go away with http://github.com/reductivelabs/puppet/commit/c702f76b271515e9c42dcb923d379fbfac4c83cd applied.

Issue 1 looks like an AR problem which I don’t know lots about Issue 2 looks like sqlite and many connections Issue 3 – see above Issue 4 looks like a load order problem Issue 5 may also be because of failing auth Issue 6 – yeah. Filing a feature request may be useful.

(1) – should i report it to rails? (2) – shouldn’t this exception be handled (ie. wait and retry until timeout)? (3) – is it going to be released (0.25.2) or should we update a wiki with instructions on syncing and patching from github? (4) – yeah, i’m not sure about it. in 0.24.8 it used to work when you had some define() in module and calling it straight (but 0.24.8 refused to work with passenger at all). in 0.25.1 i need to call “include module” and then “module::some::define{}”. is it bug or feature?

for (6) i’ve added #2906

(1) + (2) i’ve switched to mysql. it seems to be only working backend for stored configs. (3) sorry, commit:c702f76b271515e9c42dcb923d379fbfac4c83cd is already applied in 0.25.1 – no change here.

any more ideas?

Sorry, the correct commit is this one: http://github.com/reductivelabs/puppet/commit/5ed2e2619fc366a9ea1e1b9866da055ba0cd57aa

(5) well… no. this is seconds after i applied reversed patch and restarted:

Dec 11 00:00:03 punch puppetmasterd[7867]: Fileserver module 'pki' not mounted
Dec 11 00:00:03 punch puppetmasterd[7867]: Fileserver module 'syslog' not mounted
Dec 11 00:00:04 punch puppetmasterd[7867]: Fileserver module 'denyhosts' not mounted
Dec 11 00:00:04 punch puppetmasterd[7867]: Fileserver module 'denyhosts' not mounted
Dec 11 00:00:04 punch puppetmasterd[7867]: Fileserver module 'denyhosts' not mounted
Dec 11 00:00:04 punch puppetmasterd[7867]: Fileserver module 'denyhosts' not mounted
Dec 11 00:00:04 punch puppetmasterd[7867]: Fileserver module 'pki' not mounted

(both passenger 2.2.2 and 2.2.7)

But is the auth problem gone now?

Some questions about the fileserver stuff:

Also, can you specify when exactly these fileserver messages are coming up, and if this happens for all, for only one, for specific clients and modules? Are the affected modules distributed across your environments? If yes, how? Why do you have environment=puppet in your config? Do you have an environment called “puppet” (not seeing it in the config)?

yes, auth problems are gone, however “not mounted” errors are ocurring frequently (at least on 50% of runs). it happens on all clients (i have mixed 0.24.8 and 0.25.1 clients), randomly on all of modules in manifests (i think). i think it never happens on 1st connection to passenger after restarting it. example – from client’s POV:

[root@test ~]# puppetd --version
0.24.8
[root@test ~]# puppetd --test
info: Loading fact mounts
notice: Ignoring --listen on onetime run
info: Retrieving plugins
notice: /File[/var/lib/puppet/lib]/mode: mode changed '755' to '775'
notice: /File[/var/lib/puppet/lib/facter]/mode: mode changed '755' to '775'
info: Loading fact mounts
info: Caching catalog at /var/lib/puppet/localconfig.yaml
notice: Starting catalog run
notice: //puppet::puppet/File[/var/lib/puppet/lib]/mode: mode changed '775' to '755'
notice: //puppet::puppet/File[/var/lib/puppet/lib/facter]/mode: mode changed '775' to '755'
err: //ha::haproxy/File[/etc/haproxy/err503-x1.http]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/fwlb/err503-x1.http: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:133
err: //ha::haproxy/File[/etc/haproxy/err503-x2.http]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/fwlb/err503-x2.http: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:213
err: //ha::haproxy/File[/etc/haproxy/err503-x3.http]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/fwlb/err503-x3.http: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:196
err: //ha::haproxy/File[/etc/haproxy/err503-x4.http]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/fwlb/err503-x4.http: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:187
err: //ha::haproxy/File[/etc/haproxy/err503-x5.http]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/fwlb/err503-x5.http: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:124
err: //ha::haproxy/File[/etc/haproxy/err503-x6.http]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/fwlb/err503-x6.http: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:204
notice: //ha::haproxy/Service[haproxy]: Dependency file[/etc/haproxy/err503-x1.http] has 1 failures
notice: //ha::haproxy/Service[haproxy]: Dependency file[/etc/haproxy/err503-x2.http] has 1 failures
notice: //ha::haproxy/Service[haproxy]: Dependency file[/etc/haproxy/err503-x3.http] has 1 failures
notice: //ha::haproxy/Service[haproxy]: Dependency file[/etc/haproxy/err503-x4.http] has 1 failures
notice: //ha::haproxy/Service[haproxy]: Dependency file[/etc/haproxy/err503-x5.http] has 1 failures
notice: //ha::haproxy/Service[haproxy]: Dependency file[/etc/haproxy/err503-x6.http] has 1 failures
warning: //ha::haproxy/Service[haproxy]: Skipping because of failed dependencies
err: //puppet::puppet/File[/etc/sysconfig/puppet]: Failed to retrieve current state of resource: Fileserver module 'puppet' not mounted Could not describe /puppet/sysconfig: Fileserver module 'puppet' not mounted at /etc/puppet/site/modules/puppet/manifests/init.pp:57
err: //nagios::check::pacemaker/File[check_xml]: Failed to retrieve current state of resource: Fileserver module 'nagios' not mounted Could not describe /nagios/check.xsl: Fileserver module 'nagios' not mounted at /etc/puppet/site/modules/nagios/manifests/init.pp:750
err: //nagios::check::pacemaker/File[config_xml]: Failed to retrieve current state of resource: Fileserver module 'nagios' not mounted Could not describe /nagios/test/config.xml: Fileserver module 'nagios' not mounted at /etc/puppet/site/modules/nagios/manifests/init.pp:760
notice: //nagios::check::pacemaker/File[check_pacemaker]: Dependency file[/usr/local/lib/config.xml] has 1 failures
notice: //nagios::check::pacemaker/File[check_pacemaker]: Dependency file[/usr/local/lib/check.xsl] has 1 failures
warning: //nagios::check::pacemaker/File[check_pacemaker]: Skipping because of failed dependencies
err: //ha::varnish/File[/etc/varnish/default.vcl]: Failed to retrieve current state of resource: Fileserver module 'ha' not mounted Could not describe /ha/test/default.vcl: Fileserver module 'ha' not mounted at /etc/puppet/site/modules/ha/manifests/init.pp:290
notice: //ha::varnish/Service[varnish]: Dependency file[/etc/varnish/default.vcl] has 1 failures
warning: //ha::varnish/Service[varnish]: Skipping because of failed dependencies
err: //bacula::fd/File[/etc/bacula/pki/test-fd.pem]: Failed to retrieve current state of resource: Fileserver module 'bacula' not mounted Could not describe /bacula/test/fd.pem: Fileserver module 'bacula' not mounted at /etc/puppet/site/modules/bacula/manifests/init.pp:123
notice: //bacula::fd/Service[bacula-fd]: Dependency file[/etc/bacula/pki/test-fd.pem] has 1 failures
warning: //bacula::fd/Service[bacula-fd]: Skipping because of failed dependencies
notice: //bacula::fd/File[/etc/bacula/pki/test.non.3dart.com-fd.pem]: Dependency file[/etc/bacula/pki/test-fd.pem] has 1 failures
warning: //bacula::fd/File[/etc/bacula/pki/test.non.3dart.com-fd.pem]: Skipping because of failed dependencies
err: //bacula::fd/File[/etc/bacula/pki/master.cert]: Failed to retrieve current state of resource: Fileserver module 'bacula' not mounted Could not describe /bacula/master.pem: Fileserver module 'bacula' not mounted at /etc/puppet/site/modules/bacula/manifests/init.pp:116
err: //nagios::check::puppet/File[monitor_puppet]: Failed to retrieve current state of resource: Fileserver module 'puppet' not mounted Could not describe /puppet/monitor.py: Fileserver module 'puppet' not mounted at /etc/puppet/site/modules/nagios/manifests/init.pp:692
notice: //puppet::puppet/Service[puppet]: Dependency file[/etc/sysconfig/puppet] has 1 failures
warning: //puppet::puppet/Service[puppet]: Skipping because of failed dependencies
info: Sent transaction report in 0.26 seconds
notice: Finished catalog run in 11.38 seconds

from puppetmaster POV:

Dec 14 16:28:31 punch puppetmasterd[3361]: Starting Puppet server version 0.25.1
Dec 14 16:28:31 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:31 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:31 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:31 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'puppet' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'nagios' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'nagios' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'ha' not mounted
Dec 14 16:28:32 punch puppetmasterd[3361]: Fileserver module 'bacula' not mounted
Dec 14 16:28:33 punch puppetmasterd[3361]: Fileserver module 'bacula' not mounted
Dec 14 16:28:33 punch puppetmasterd[3361]: Fileserver module 'puppet' not mounted
Dec 14 16:28:33 punch puppetmasterd[3361]: No report named 'log #'
Dec 14 16:28:33 punch puppetmasterd[3361]: RRD library is missing; cannot store metrics
Dec 14 16:28:33 punch puppetmasterd[3361]: RRD library is missing; cannot graph metrics
Dec 14 16:28:33 punch puppetmasterd[3361]: RRD library is missing; cannot store metrics
Dec 14 16:28:33 punch puppetmasterd[3361]: RRD library is missing; cannot graph metrics
Dec 14 16:28:33 punch puppetmasterd[3361]: RRD library is missing; cannot store metrics
Dec 14 16:28:33 punch puppetmasterd[3361]: RRD library is missing; cannot graph metrics
Dec 14 16:28:33 punch puppetmasterd[3361]: Report rrdgraph failed: Could not find graphs for daily

yes, there is “puppet” environment defined in my puppet.conf, and at this time it’s the same as main/production (ie. all modules are identical in all environments where clients belong).

[puppet]
    manifest = /etc/puppet/site/manifests/site.pp
    modulepath = /etc/puppet/site/modules

is it possible that modulepath is being defaulted on successive rack puppetmaster runs? i think @ln -s /etc/puppet/site/modules /etc/puppet/modules@ stopped those from occuring (50 puppet runs on test client, no single fileserver error, no errors in puppetmaster log from other 50-few clients).

is anybody working on it or we should move towards mongrel?

Konrad —

Yes, though as Christian noted above (and as the title suggests) this ticket is really a collection of only partially related issues. To keep from getting bogged down we tend to peel off specific points from such tickets and resolve them separately, and over time the initial ticket doesn’t see as much activity. That’s what’s happening here. We hope to have several of the points addressed in 0.25.2 (expected in the next day or so; 0.25.2rc3 is available for testing now) except for those (like sqlite’s problems with multiple users) which are beyond our scope.

This ticket has been marked for review in the 0.25.3 process to make sure no issues fell through the cracks; if any did, separate tickets will be opened for them and this ticket will be updated to reflect the related tickets, then probably closed.

— Markus

i’m perfectly ok with tracking individual tickets, however i cannot see any “related tickets” list or section here.