The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #13038

Hung agent processes

Added by Erik Dalén over 2 years ago. Updated about 2 years ago.

Status:ClosedStart date:03/09/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Affected Puppet version:2.7.11 Branch:
Keywords:select hung process stuck

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.


Description

Since upgrading to 2.7.11 (from 2.6.2) we have experienced a number or hosts where the puppet process has gotten stuck. Doing a strace shows the following:

Process 15272 attached – interrupt to quit select(0, NULL, NULL, NULL, {0, 386506}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout) select(0, NULL, NULL, NULL, {1, 0}C Process 15272 detached

lsof output:

lsof -p 15272

COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME puppet 15272 root cwd DIR 254,0 4096 917506 /root puppet 15272 root rtd DIR 254,0 4096 2 / puppet 15272 root txt REG 254,0 5424 794665 /usr/bin/ruby1.8 puppet 15272 root mem REG 254,0 80712 1180151 /lib/libresolv-2.11.2.so puppet 15272 root mem REG 254,0 22928 1180172 /lib/libnss_dns-2.11.2.so puppet 15272 root mem REG 254,0 37984 922400 /usr/lib/ruby/1.8/x86_64-linux/zlib.so puppet 15272 root mem REG 254,0 10784 925695 /usr/lib/ruby/1.8/x86_64-linux/shadow.so puppet 15272 root mem REG 254,0 8648 922394 /usr/lib/ruby/1.8/x86_64-linux/digest/md5.so puppet 15272 root mem REG 254,0 45824 922412 /usr/lib/ruby/1.8/x86_64-linux/socket.so puppet 15272 root mem REG 254,0 93936 793016 /usr/lib/libz.so.1.2.3.4 puppet 15272 root mem REG 254,0 1693344 797851 /usr/lib/libcrypto.so.0.9.8 puppet 15272 root mem REG 254,0 349248 797852 /usr/lib/libssl.so.0.9.8 puppet 15272 root mem REG 254,0 298664 922401 /usr/lib/ruby/1.8/x86_64-linux/openssl.so puppet 15272 root mem REG 254,0 15304 922415 /usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so puppet 15272 root mem REG 254,0 12712 922397 /usr/lib/ruby/1.8/x86_64-linux/digest.so puppet 15272 root mem REG 254,0 12808 922392 /usr/lib/ruby/1.8/x86_64-linux/digest/sha1.so puppet 15272 root mem REG 254,0 14392 791358 /usr/lib/gconv/UTF-16.so puppet 15272 root mem REG 254,0 17064 922409 /usr/lib/ruby/1.8/x86_64-linux/iconv.so puppet 15272 root mem REG 254,0 19352 922402 /usr/lib/ruby/1.8/x86_64-linux/strscan.so puppet 15272 root mem REG 254,0 271400 922421 /usr/lib/ruby/1.8/x86_64-linux/nkf.so puppet 15272 root mem REG 254,0 12680 922406 /usr/lib/ruby/1.8/x86_64-linux/syslog.so puppet 15272 root mem REG 254,0 47616 1180155 /lib/libnss_files-2.11.2.so puppet 15272 root mem REG 254,0 43552 1180152 /lib/libnss_nis-2.11.2.so puppet 15272 root mem REG 254,0 89064 1180168 /lib/libnsl-2.11.2.so puppet 15272 root mem REG 254,0 31616 1180165 /lib/libnss_compat-2.11.2.so puppet 15272 root mem REG 254,0 5344 922420 /usr/lib/ruby/1.8/x86_64-linux/fcntl.so puppet 15272 root mem REG 254,0 11032 922399 /usr/lib/ruby/1.8/x86_64-linux/etc.so puppet 15272 root mem REG 254,0 16560 922398 /usr/lib/ruby/1.8/x86_64-linux/thread.so puppet 15272 root mem REG 254,0 119984 922403 /usr/lib/ruby/1.8/x86_64-linux/syck.so puppet 15272 root mem REG 254,0 21120 922413 /usr/lib/ruby/1.8/x86_64-linux/stringio.so puppet 15272 root mem REG 254,0 1432968 1180158 /lib/libc-2.11.2.so puppet 15272 root mem REG 254,0 530736 1180167 /lib/libm-2.11.2.so puppet 15272 root mem REG 254,0 35104 1180169 /lib/libcrypt-2.11.2.so puppet 15272 root mem REG 254,0 14696 1180157 /lib/libdl-2.11.2.so puppet 15272 root mem REG 254,0 31744 1180160 /lib/librt-2.11.2.so puppet 15272 root mem REG 254,0 131260 1180159 /lib/libpthread-2.11.2.so puppet 15272 root mem REG 254,0 927840 794548 /usr/lib/libruby1.8.so.1.8.7 puppet 15272 root mem REG 254,0 128744 1180170 /lib/ld-2.11.2.so puppet 15272 root mem REG 254,0 26048 793220 /usr/lib/gconv/gconv-modules.cache puppet 15272 root 0r FIFO 0,8 0t0 765633332 pipe puppet 15272 root 1u REG 254,0 0 393323 /tmp/tmpfs2dR5B (deleted) puppet 15272 root 2u REG 254,0 0 393323 /tmp/tmpfs2dR5B (deleted) puppet 15272 root 3r FIFO 0,8 0t0 765635635 pipe puppet 15272 root 4w FIFO 0,8 0t0 765635635 pipe puppet 15272 root 5u unix 0xffff8801b88f0f00 0t0 765635637 socket


Related issues

Duplicates Puppet - Bug #10418: Puppet agent hangs when listen is true and reading from /... Closed 11/01/2011

History

#1 Updated by Erik Dalén over 2 years ago

More easily read output:

strace:

strace -p 15272
Process 15272 attached - interrupt to quit
select(0, NULL, NULL, NULL, {0, 386506}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0})     = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}^C <unfinished ...>
Process 15272 detached

lsof:

# lsof -p 15272
COMMAND   PID USER   FD   TYPE             DEVICE SIZE/OFF      NODE NAME
puppet  15272 root  cwd    DIR              254,0     4096    917506 /root
puppet  15272 root  rtd    DIR              254,0     4096         2 /
puppet  15272 root  txt    REG              254,0     5424    794665 /usr/bin/ruby1.8
puppet  15272 root  mem    REG              254,0    80712   1180151 /lib/libresolv-2.11.2.so
puppet  15272 root  mem    REG              254,0    22928   1180172 /lib/libnss_dns-2.11.2.so
puppet  15272 root  mem    REG              254,0    37984    922400 /usr/lib/ruby/1.8/x86_64-linux/zlib.so
puppet  15272 root  mem    REG              254,0    10784    925695 /usr/lib/ruby/1.8/x86_64-linux/shadow.so
puppet  15272 root  mem    REG              254,0     8648    922394 /usr/lib/ruby/1.8/x86_64-linux/digest/md5.so
puppet  15272 root  mem    REG              254,0    45824    922412 /usr/lib/ruby/1.8/x86_64-linux/socket.so
puppet  15272 root  mem    REG              254,0    93936    793016 /usr/lib/libz.so.1.2.3.4
puppet  15272 root  mem    REG              254,0  1693344    797851 /usr/lib/libcrypto.so.0.9.8
puppet  15272 root  mem    REG              254,0   349248    797852 /usr/lib/libssl.so.0.9.8
puppet  15272 root  mem    REG              254,0   298664    922401 /usr/lib/ruby/1.8/x86_64-linux/openssl.so
puppet  15272 root  mem    REG              254,0    15304    922415 /usr/lib/ruby/1.8/x86_64-linux/racc/cparse.so
puppet  15272 root  mem    REG              254,0    12712    922397 /usr/lib/ruby/1.8/x86_64-linux/digest.so
puppet  15272 root  mem    REG              254,0    12808    922392 /usr/lib/ruby/1.8/x86_64-linux/digest/sha1.so
puppet  15272 root  mem    REG              254,0    14392    791358 /usr/lib/gconv/UTF-16.so
puppet  15272 root  mem    REG              254,0    17064    922409 /usr/lib/ruby/1.8/x86_64-linux/iconv.so
puppet  15272 root  mem    REG              254,0    19352    922402 /usr/lib/ruby/1.8/x86_64-linux/strscan.so
puppet  15272 root  mem    REG              254,0   271400    922421 /usr/lib/ruby/1.8/x86_64-linux/nkf.so
puppet  15272 root  mem    REG              254,0    12680    922406 /usr/lib/ruby/1.8/x86_64-linux/syslog.so
puppet  15272 root  mem    REG              254,0    47616   1180155 /lib/libnss_files-2.11.2.so
puppet  15272 root  mem    REG              254,0    43552   1180152 /lib/libnss_nis-2.11.2.so
puppet  15272 root  mem    REG              254,0    89064   1180168 /lib/libnsl-2.11.2.so
puppet  15272 root  mem    REG              254,0    31616   1180165 /lib/libnss_compat-2.11.2.so
puppet  15272 root  mem    REG              254,0     5344    922420 /usr/lib/ruby/1.8/x86_64-linux/fcntl.so
puppet  15272 root  mem    REG              254,0    11032    922399 /usr/lib/ruby/1.8/x86_64-linux/etc.so
puppet  15272 root  mem    REG              254,0    16560    922398 /usr/lib/ruby/1.8/x86_64-linux/thread.so
puppet  15272 root  mem    REG              254,0   119984    922403 /usr/lib/ruby/1.8/x86_64-linux/syck.so
puppet  15272 root  mem    REG              254,0    21120    922413 /usr/lib/ruby/1.8/x86_64-linux/stringio.so
puppet  15272 root  mem    REG              254,0  1432968   1180158 /lib/libc-2.11.2.so
puppet  15272 root  mem    REG              254,0   530736   1180167 /lib/libm-2.11.2.so
puppet  15272 root  mem    REG              254,0    35104   1180169 /lib/libcrypt-2.11.2.so
puppet  15272 root  mem    REG              254,0    14696   1180157 /lib/libdl-2.11.2.so
puppet  15272 root  mem    REG              254,0    31744   1180160 /lib/librt-2.11.2.so
puppet  15272 root  mem    REG              254,0   131260   1180159 /lib/libpthread-2.11.2.so
puppet  15272 root  mem    REG              254,0   927840    794548 /usr/lib/libruby1.8.so.1.8.7
puppet  15272 root  mem    REG              254,0   128744   1180170 /lib/ld-2.11.2.so
puppet  15272 root  mem    REG              254,0    26048    793220 /usr/lib/gconv/gconv-modules.cache
puppet  15272 root    0r  FIFO                0,8      0t0 765633332 pipe
puppet  15272 root    1u   REG              254,0        0    393323 /tmp/tmpfs2dR5B (deleted)
puppet  15272 root    2u   REG              254,0        0    393323 /tmp/tmpfs2dR5B (deleted)
puppet  15272 root    3r  FIFO                0,8      0t0 765635635 pipe
puppet  15272 root    4w  FIFO                0,8      0t0 765635635 pipe
puppet  15272 root    5u  unix 0xffff8801b88f0f00      0t0 765635637 socket

#2 Updated by Michael Stahnke over 2 years ago

  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Erik Dalén

Are you on RHEL or CentOS or something similar? We’ve seen issues with specific kernels on EL 5 systems.

#3 Updated by Erik Dalén over 2 years ago

Michael Stahnke wrote:

Are you on RHEL or CentOS or something similar? We’ve seen issues with specific kernels on EL 5 systems.

This is on Debian Squeeze (Linux 2.6.32).

The agent is run from cron.

#4 Updated by Erik Dalén over 2 years ago

  • Assignee deleted (Erik Dalén)

This seems to have been solved by upgrading to 2.7.12. So you could close it I guess.

#5 Updated by Patrick Carlisle over 2 years ago

  • Status changed from Needs More Information to Closed

#6 Updated by eric sorenson about 2 years ago

Marking this a dup of #10418 – Looks like the same kernel bug, though in a different distro. Please see if updating to facter-1.6.12 works for you.

Also available in: Atom PDF