Puppet

And for the record, the problem was that modulepath was cached at the wrong time.

I’m still seeing this problem with the puppetd/puppetmasterd combination. I have a custom modulepath in puppet.conf, and modules in the non-default path aren’t found by puppetmasterd until 15 seconds after it’s started — the default value of filetimeout. This is especially noticeable with Passenger, due to its on-demand puppetmasterd spawning and shutdown of idle instances. Reverting commit d397f8 fixes it in my setup.

At a guess, it looks like d397f8’s late loading:

     def module_directories
-        Puppet.settings.value(:modulepath, Puppet[:environment]).find_all do |dir|
-            FileTest.directory?(dir)
-        end.collect do |dir|
-            Dir.entries(dir)
+        # We have to require this late in the process because otherwise we might have
+        # load order issues.
+        require 'puppet/node/environment'
+        Puppet::Node::Environment.new.modulepath.collect do |dir|
+            Dir.entries(dir).reject { |f| f =~ /^\./ }.collect { |f| File.join(dir, f) }
         end.flatten.collect { |d| [File.join(d, "plugins"), File.join(d, "lib")] }.flatten.find_all do |d|
             FileTest.directory?(d)
         end

Thwarts ea584787’s modulepath clearing (doesn’t clear it ‘cause it isn’t loaded yet).

@@ -494,7 +494,15 @@ class Puppet::Util::Settings
         @sync.synchronize do # yay, thread-safe
             @values[type][param] = value
             @cache.clear
+
             clearused
+
+            # Clear the list of environments, because they cache, at least, the module path.
+            # We *could* preferentially just clear them if the modulepath is changed,
+            # but we don't really know if, say, the vardir is changed and the modulepath
+            # is defined relative to it. We need the defined? stuff because of loading
+            # order issues.
+            Puppet::Node::Environment.clear if defined?(Puppet::Node) and defined?(Puppet::Node::Environment)
         end

I can’t reproduce this, either with puppetd and puppetmasterd, or with the standalone puppet.

What, specifically, are you trying to load? A manifest? Plugin?

I think the problem is unrelated to environments – autoload instances would get expired differently than environments. That is, I think both environments and autoloader instances are caching information, and we’re only clearing the environments.

What about if you apply this patch to your master?

diff --git a/lib/puppet/util/settings.rb b/lib/puppet/util/settings.rb
index f2c513b..e879cf9 100644
--- a/lib/puppet/util/settings.rb
+++ b/lib/puppet/util/settings.rb
@@ -511,6 +511,7 @@ class Puppet::Util::Settings
             # is defined relative to it. We need the defined? stuff because of loading
             # order issues.
             Puppet::Node::Environment.clear if defined?(Puppet::Node) and defined?(Puppet::Node::Environment)
+            Puppet::Util::Cacher.expire
         end
 
         return value

Do you still see the problem?

Markus Roberts wrote:

John —

Can you reproduce the problem with 0.25.2? If so, please post enough details to allow us to do the same; if not post that so we can close the ticket.

— Markus

Markus, We’re also experiencing this bug and can reproduce with 0.25.2. The one line patch Luke gave for lib/puppet/util/settings.rb did not make any difference.

This only causes us trouble when run via passenger. Running puppetmasterd with webrick does not exhibit the bug.

Our versions are:

  puppet: 0.25.2
  ruby: 1.8.7
  passenger: 2.2.8
  apache: httpd-2.2.3-22.el5.centos

config.ru file is:

# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $:.unshift('/opt/puppet/lib')

$0 = "puppetmasterd"
require 'puppet'

# if you want debugging:
#ARGV << "--debug"

ARGV << "--rack"
require 'puppet/application/puppetmasterd'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Application[:puppetmasterd].run

[puppetmaster] section of puppet.conf is:

[puppetmasterd]
    autosign = $confdir/autosign.conf
    modulepath = /export/home/puppet/modules
    manifestdir = /export/home/puppet/manifests
    templatedir = /export/home/puppet/templates

One other interesting note: if you turn on —trace in the config.ru, then the problem goes away. Ex:

ARGV << "--trace"

Let me know if there is any additional info I can provide. It should be pretty easy to reproduce if using passenger.

Steve

I tried 0.25.4 today and it still exhibits the same behavior (I saw that it had been pushed to 0.25.5, but figured I’d try anyhow).

One point that I made in my previous post that may have slipped through the cracks, but I think is interesting:

if you enable —trace in the config.ru file, the problem goes away. And I can’t see that any extra trace logging is happening. No noticeable performance hit either.

Again, if there is anything I can do to help a developer reproduce this bug I’ll be happy to help.

Steve

As of puppet-0.25.4, the reproduction recipe in Bug #2832 (which is marked as a duplicate of this bug) still shows the problem. I use the default web server built in to puppetd, not passenger.

Hi, Same problem here with passenger 2.2.5 and puppet 0.25.4. Instead of the —trace option, I use the —ignorecache option in the config.ru as workaround.

Aurélien

I’m also seeing this behavior on puppet 0.24.8 on ubuntu 9.10 (karmic koala). Since I’m a newbie I don’t know what the equivalent of a “config.ru” file is, so I can’t apply the workarounds mentioned above.

Does anyone have a hint on another workaround, or a hint on where I can find config.ru on my system? (“locate” couldn’t find it even after running updatedb, so I’m pretty certain that there are no config.ru files on my system).

Does anyone know who the “anonymous” user that keeps (incorrectly, so far as I can tell) closing this ticket is?

That’s Redmine. I’m turning the feature off. More trouble than good.

I was testing upgrading a dev puppetmaster from puppet 0.24.8 to 0.25.4 this weekend, and had the same issues.

debug: Calling fileserver.list
err: /File[/var/puppet/lib]: Failed to generate additional resources during transaction: None of the provided sources existdebug: Calling fileserver.describe
err: /File[/var/puppet/lib]: Failed to retrieve current state of resource: No specified source was found from puppet://pm.sharif.example.com/plugins
[SNIP]
err: Could not retrieve catalog: Could not find class server at /etc/puppet/manifests/nodes.pp:25 on node lenny64.sharif.example.com

Relevant info: Apache + passenger 2.2.5 puppet 0.25.4 from Debian backports (yes this is all very old, but it’s what I had already downloaded when I was playing on the airplane)

I used the boilerplate config.ru in the distribution. I tried the work around of adding ARGV << “—trace” in config.ru and suddenly it all works.

/etc/puppet/puppet.conf on the puppetmaster:

[main]
  factpath = $vardir/lib/facter
  logdir = /var/log/puppet
  pidfile = /var/run/puppet/puppetd.pid
  pluginsync = true
  puppetdlog = $logdir/puppetd.log
  puppetport = 8139
  runinterval = 14400
  splay = true
  server = pm.sharif.example.com
  rundir = /var/run/puppet
  vardir = /var/lib/puppet
  ssldir = /var/lib/puppet/ssl
  verbose = true
  debug = true

[puppetmasterd]
  modulepath = /var/lib/puppet/modules:/etc/puppet/manifests/modules:/etc/puppet/manifests/modules/base:/etc/puppet/manifests/modules/site:/etc/puppet/manifests/modules/deploy
  reports = log
  ssl_client_header = SSL_CLIENT_S_DN
  ssl_client_verify_header = SSL_CLIENT_VERIFY
  templatedir = /var/lib/puppet/templates
  pidfile = /var/run/puppet/puppetmasterd.pid

fyi:

puppet apply --modulepath=/home/duritong/dev/puppet/modules/public/:/home/duritong/dev/puppet/modules/site/ manifests/site.pp
or
puppet --modulepath=/home/duritong/dev/puppet/modules/public/:/home/duritong/dev/puppet/modules/site/ manifests/site.pp

Works on Debian Puppet 2.6.7, I’m not sure if this is still a bug