The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #1539

Puppet consuming lots of background CPU time

Added by John Wiegley about 6 years ago. Updated almost 3 years ago.

Status:AcceptedStart date:08/31/2008
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:plumbing
Target version:-
Affected Puppet version:0.24.5 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket may be automatically exported to the PUP project on JIRA using the button below:


Description

When I run puppetd and/or puppetmasterd, I’m seeing a continual background utilization of around 0.3% for puppetd and 0.9% for puppetmasterd. This is on OS X 10.5.4.

I’ve run ‘sample’ on the puppetd process, and I’m seeing that there’s a bit of a busy wait going on. Attached is the data (the numbers to the left of the function names are call counts during the 60 second analysis period).

My expectation is that between the half-hour catalog checks, puppetd would consume 0.0%, with maybe a tiny bump every minute or so for config file checks (and that this would be configurable, as I rarely change mine on the servers where I run puppetd).

ruby_2901.jROjFi.sample.txt Magnifier (2.67 KB) John Wiegley, 08/31/2008 05:31 pm

strace.txt Magnifier - Strace output (188 KB) Henrik Pedersen, 01/02/2009 12:25 am

History

#1 Updated by James Turnbull about 6 years ago

  • Category set to plumbing
  • Status changed from Unreviewed to Accepted
  • Assignee set to Andrew Shafer

#2 Updated by Mathieu Gagné almost 6 years ago

Hi,

I’m having the same issue and can’t find out what’s causing it. The client constantly consumes 3-5% of the CPU.

Here is some information about the environment: OS: Debian GNU/Linux lenny/sid Kernel: 2.6.26-1-amd64 Ruby: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux] Puppet: 0.24.5

Any help will be appreciated. Thanks.

#3 Updated by Henrik Pedersen almost 6 years ago

Hi

We are seeing the same issue on several debian servers. A full strace for a 10 second period is attached, but it mainly contains lines like this:

01:20:05 —– SIGVTALRM (Virtual timer expired) @ 0 (0) —– 01:20:05 sigreturn() = ? (mask now []) <0.000015> 01:20:05 select(4, [3], [], [], {5, 836000}) = ? ERESTARTNOHAND (To be restarted) <0.018253>

I hope this helps.

Testes on:

OS: Debian GNU/Linux etch Kernel: 2.6.26-bpo.1-686 Ruby: ruby 1.8.5 (2006-08-25) [i486-linux] puppet: 0.24.6

#4 Updated by Sheldon Hearn almost 4 years ago

  • Assignee deleted (Andrew Shafer)

Henrik Pedersen wrote:

We are seeing the same issue on several debian servers. A full strace for a 10 second period is attached, but it mainly contains lines like this:

01:20:05 —– SIGVTALRM (Virtual timer expired) @ 0 (0) —– 01:20:05 sigreturn() = ? (mask now []) <0.000015> 01:20:05 select(4, [3], [], [], {5, 836000}) = ? ERESTARTNOHAND (To be restarted) <0.018253>

Still present in puppet-2.6.2 on Debian Linux.

Puppetd does this in response to an unluckily timed network partitioning. We had a network problem in a DC last night. Out of the 402 hosts in that DC, only 7 got bitten by this bug.

Lsof confirms that the fd select() is spinning on is the one for the connection to the puppetmaster. Netstat shows the connection ESTABLISHED on the client, and absent on the puppetmaster.

Now that we use mcollective to schedule puppet runs, this is the last remaining problem that forces us to do any puppet babysitting on our network.

#5 Updated by Brice Figureau almost 3 years ago

I know this is an old bug reports, but if someone gets it again:

This is a bug in ruby 1.8.7pl72 as shipped in debian lenny: http://timetobleed.com/ruby-threading-bugfix-small-fix-goes-a-long-way/

I believe it was fixed in a later than 72 patchlevel, or by compiling MRI without pthreads (it’s better to use the latest 1,8.7 available though). I think it is related to this ruby bug entry: http://redmine.ruby-lang.org/issues/show/2553

Also available in: Atom PDF