Puppet

I tested it and puppet now doesn’t anymore with a stacktrace, but it still exits directly after printing the error. Wouldn’t it be better to just fail the resource on which the timeout appears as it is done at some other places.

Peter Meier wrote:

I tested it and puppet now doesn’t anymore with a stacktrace, but it still exits directly after printing the error. Wouldn’t it be better to just fail the resource on which the timeout appears as it is done at some other places.

Another note on that: This seems to happen mainly during the creation of the internal types, so when the catalog is being parsed and the individual resource type objects are being created and puppet is for example asking the master for metadata about a file.

This means that it is definitely not the same as a usual timeout during fetching the source of a file resource, so failing the resource and anything else depending on it doesn’t yet seem to be possible.

However, still puppet usually seems to be stalled for a very long time if that error happens and it then prints out for example:

[...]
info: Caching catalog for foo.bar.ch
err: Could not run Puppet configuration client: Connection timed out Could not retrieve file metadata for puppet:///modules/apache/vhosts.d/CentOS.Final/0-default.conf: Connection timed out at /srv/puppet/development/modules/public/apache/manifests/vhost/file.pp:72
#

If I hook tcpdump on the master in front of nginx I can see that ningx sent something to the client. However the client remains silent about that and using strace I can see that the client is waiting for an answer and is then running into the http timeout after a very long time (more than the usual configtimeout and the usual 60s).

It occures to me that this behavior has been somehow introduced into 2.6 as I only remember resource timeouts (like fetching a file’s source) before 2.6. Furthermore, this is getting a bit annoying as I have a lot of clients on rather high latency networks and if they exit after that timeout the whole run is aborted. :/

But as I also upgraded other things (such as client’s/master’s ruby version or the different gems on the master) when upgrading puppet to 2.6, so the problem might be also somewhere else and debugging is therefore a bit difficult. Anyway, I will dig into that problem a bit further and my next step would be to dump the traffic on both sites, so I can see if really a package is getting lost or ruby/puppet-client is stalled somewhere else. If there are any other ideas around how to track things down I would be happy for further suggestions.

So besides looking for the cause for that timeout, I think puppet should somehow not abort its run, when such a timeout happens.

James Turnbull wrote:

This needs a rethink based on Peter’s comments.

I could get rid off most of the timeouts symptoms by introducing the fair proxy scheduler as described in nginx – mongrel – fair proxy balancer. However I still think that the run shouldn’t be aborted.

Markus Roberts wrote:

My understanding is that this isn’t a timeout on a resource, it’s a timeout on communicating with the puppetmaster (e.g. getting the catalog).

No, it’s also timeout when getting metadata such as file-checksums in the pre-apply stage. This is related to the questions I asked in this thread: http://groups.google.com/group/puppet-dev/browse_thread/thread/1c8ac2c2d6fab46

Btw: I also filed #4704 and I think this one is related to that one. But I failed it as a separate bug as it’s a different cause (timeout vs. non-existant source). If you think they have both the same root-cause and should be fixed in the same way you can close one as duplicate.