The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #10819

Puppet agent hangs when 'listen = true' on Centos 5.7

Added by Chip Schweiss over 2 years ago. Updated over 1 year ago.

Status:DuplicateStart date:11/14/2011
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:agent
Target version:-
Affected Puppet version:2.6.9 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.


Description

If the ‘listen = true’ is set in puppet.conf, the client will start, create /var/lib/puppet/state/puppetdlock and stall forever.

Once the service is started ‘puppetd —test’ always returns:

notice: Ignoring --listen on onetime run
notice: Run of Puppet configuration client already in progress; skipping

If the service is stopped ‘puppetd —test’ runs normally.

The syslog has only two lines:

Mon Nov 14 12:30:17 -0600 2011 Puppet (notice): Reopening log files
Mon Nov 14 12:30:18 -0600 2011 Puppet (notice): Starting Puppet client version 2.6.9

Operation is fine on Centos 5.6 until the kernel update from Centos 5.7 is applied. (kernel-2.6.18-274.7.1.el5)

Removing ‘listen = true’ and operation returns to normal.


Related issues

Related to Puppet - Bug #10418: Puppet agent hangs when listen is true and reading from /... Closed 11/01/2011
Related to Puppet - Bug #12181: Puppet 2.7.10 run stopps after loading facts. Closed 01/26/2012
Duplicates Puppet - Bug #7141: puppetd runs fail in 'daemon' mode when stat'ing /proc files Closed 04/18/2011 07/05/2011

History

#1 Updated by Chip Schweiss over 2 years ago

I have found this problem is specific to kernel-2.6.18-274.7.1.el5. Downgrading the kernel to kernel-2.6.18-274.3.1.el5 fixes the problem.

There is a bug in Redhat’s bugzilla, but is now locked out. It can be seen in Google’s cache: http://webcache.googleusercontent.com/search?q=cache:Z3wAuF4G04sJ:https://bugzilla.redhat.com/show_bug.cgi%3Fformat%3Dmultiple%26id%3D751214+2.6.18-274+puppet&cd=2&hl=en&ct=clnk&gl=us&client=firefox-a

The problem may go away with the next kernel release, but puppet should not be waiting forever for a file read.

#2 Updated by Kelsey Hightower over 2 years ago

Chip,

Thanks for reporting this issue. There needs to be a discussion on the best way to handle such issues.

#3 Updated by Josh Cooper over 2 years ago

  • Status changed from Unreviewed to Duplicate

The fix for this requires kernel changes. A ticket has been logged with Red Hat to fix the kernel regression upstream at https://bugzilla.redhat.com/show_bug.cgi?id=751214.

#4 Updated by Xavier Krantz about 2 years ago

Hi every one !

I’m facing the same issue with Puppet 2.6.13 on a CentOS 5.7 OpenVZ Container, running on a Debian 2.6.26-2-openvz-amd64 kernel…

So, is that really a kernel issue linked to the 2.6.18-274.17.1.el5 version ?

And more over, I have others CentOS 5.7 clients. Some physical (with the 2.6.18-274.17.1.el5 kernel), and others into containers.

The issue seems not be very reproducible, desipte every agent have the same “puppet.conf” file withe the “listen = true” option activated…

Thanks for you time !

#5 Updated by Chip Schweiss about 2 years ago

I’m beginning to wonder the same thing. The problem of an instant hang up stopped at kernel 2.6.18-274.17.1.el5. But out of 44 nodes, I get about half a dozen per day that hang up randomly. Some on Centos 5.7 with kernel kernel 2.6.18-274.17.1.el5 and others on Centos 6.2 with kernel 2.6.32-220.2.1.el6.x86_64.

It seems more common on the Centos 6.2 nodes.

#6 Updated by Mark Fleishman about 2 years ago

Has there been any updates for this problem? I have a new installation of Puppet 2.7 on RHEL 6.1 and 6.2 and I am experiencing the same problem with listen=true using Dashboard and puppet kick. The kernels that I have installed are: 2.6.32-220.el6.x86_64, 2.6.32-220.7.1.el6.x86_64.

#7 Updated by eric sorenson over 1 year ago

Please follow the activity on this on #10418.

Also available in: Atom PDF