Puppet agent hangs when 'listen = true' on Centos 5.7
|Affected Puppet version:||2.6.9||Branch:|
If the ‘listen = true’ is set in puppet.conf, the client will start, create /var/lib/puppet/state/puppetdlock and stall forever.
Once the service is started ‘puppetd —test’ always returns:
notice: Ignoring --listen on onetime run notice: Run of Puppet configuration client already in progress; skipping
If the service is stopped ‘puppetd —test’ runs normally.
The syslog has only two lines:
Mon Nov 14 12:30:17 -0600 2011 Puppet (notice): Reopening log files Mon Nov 14 12:30:18 -0600 2011 Puppet (notice): Starting Puppet client version 2.6.9
Operation is fine on Centos 5.6 until the kernel update from Centos 5.7 is applied. (kernel-2.6.18-274.7.1.el5)
Removing ‘listen = true’ and operation returns to normal.
#1 Updated by Chip Schweiss about 2 years ago
I have found this problem is specific to kernel-2.6.18-274.7.1.el5. Downgrading the kernel to kernel-2.6.18-274.3.1.el5 fixes the problem.
There is a bug in Redhat’s bugzilla, but is now locked out. It can be seen in Google’s cache: http://webcache.googleusercontent.com/search?q=cache:Z3wAuF4G04sJ:https://bugzilla.redhat.com/show_bug.cgi%3Fformat%3Dmultiple%26id%3D751214+2.6.18-274+puppet&cd=2&hl=en&ct=clnk&gl=us&client=firefox-a
The problem may go away with the next kernel release, but puppet should not be waiting forever for a file read.
#4 Updated by Xavier Krantz almost 2 years ago
Hi every one !
I’m facing the same issue with Puppet 2.6.13 on a CentOS 5.7 OpenVZ Container, running on a Debian 2.6.26-2-openvz-amd64 kernel…
So, is that really a kernel issue linked to the 2.6.18-274.17.1.el5 version ?
And more over, I have others CentOS 5.7 clients. Some physical (with the 2.6.18-274.17.1.el5 kernel), and others into containers.
The issue seems not be very reproducible, desipte every agent have the same “puppet.conf” file withe the “listen = true” option activated…
Thanks for you time !
#5 Updated by Chip Schweiss almost 2 years ago
I’m beginning to wonder the same thing. The problem of an instant hang up stopped at kernel 2.6.18-274.17.1.el5. But out of 44 nodes, I get about half a dozen per day that hang up randomly. Some on Centos 5.7 with kernel kernel 2.6.18-274.17.1.el5 and others on Centos 6.2 with kernel 2.6.32-220.2.1.el6.x86_64.
It seems more common on the Centos 6.2 nodes.
#6 Updated by Mark Fleishman over 1 year ago
Has there been any updates for this problem? I have a new installation of Puppet 2.7 on RHEL 6.1 and 6.2 and I am experiencing the same problem with listen=true using Dashboard and puppet kick. The kernels that I have installed are: 2.6.32-220.el6.x86_64, 2.6.32-220.7.1.el6.x86_64.