The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #1781

Slow execution

Added by Mathieu Gagné over 7 years ago. Updated about 6 years ago.

Status:RejectedStart date:11/29/2008
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:plumbing
Target version:-
Affected Puppet version:0.24.8 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Hi,

We are currently evaluating Puppet to see if it could be used as part of our management system.

I’m currently facing what I consider serious performance issues. In fact, Puppet seems to be very slow in general.

Let me show you a first example:

$ time puppet —version 0.24.5

real 0m0.718s user 0m0.576s sys 0m0.140s

During that time, there is hundreds, if not thousands “rt_sigprocmask(SIG_BLOCK, NULL, [], 8)” calls being made. All this to show the version.

Is it normal? If yes, why? Can it be improved?

As for my second example, I installed Puppet with a basic manifest: the “sudo” example given in the tutorial. The file is synced to the node, great.

However, the puppet client is constantly using about 3-4% of the CPU. Again, hundreds of “rt_sigprocmask()” calls per second are being made. Is it normal?

I’m just wondering if there is any problem or if it’s a “normal” behavior.

To be honest, this is currently a show stopper for me as I’m not interested in running a tool which would just drain the CPU for no apparent reason.

Here is some information about the environment: OS: Debian GNU/Linux lenny/sid Kernel: 2.6.26-1-amd64 Ruby: ruby 1.8.7 (2008-08-11 patchlevel 72) [x86_64-linux] Puppet: 0.24.5

Can you provide some help/hints about it? Any help will be appreciated. Thanks.

History

#1 Updated by James Turnbull over 7 years ago

  • Category set to plumbing
  • Status changed from Unreviewed to Accepted

I don’t see the changing of the signals using .6 on Debian or Ubuntu or Red Hat – so I don’t think that’s normal.

As for your performance issues – well there are some known issues with file serving and performance – and those with large-scale file serving installations report some memory leakage and performance issues.

Luke – any ideas on the signal stuff?

#2 Updated by Luke Kanies over 7 years ago

  • Status changed from Accepted to Rejected

Startup time is not a reasonable judge of performance, in my opinion.

Those signals you’re seeing are Ruby’s relatively silly way of sleeping, so it’s perfectly normal, and there’s no way I can do anything about that.

If you’re concerned about the performance between runs, you can run puppet out of cron just as easily as the daemon.

#3 Updated by Mathieu Gagné over 7 years ago

Hi,

Thanks for your answer.

luke wrote:

Those signals you’re seeing are Ruby’s relatively silly way of sleeping, so it’s perfectly normal, and there’s no way I can do anything about that.

Ok, the problem is probably somewhere else then.

If you’re concerned about the performance between runs, you can run puppet out of cron just as easily as the daemon.

So turning a blind eye on a problem I find important would be the solution? You can certainly understand that this “solution” is just a patch, doesn’t solve anything at all under the hood and is not acceptable.

There is certainly slow Ruby’s components being used by Puppet and a profiling would be one of the way to determine what is causing all this CPU usage while waiting and doing nothing.

On the other hand, I think version 1.9 of Ruby would improve overall performances. Will Puppet be compatible with Ruby 1.9?

Don’t get me wrong, I like Puppet and what it can bring overall. However there is certainly room for improvements and I hope we can find a way to make it better.

Thanks for your time.

#4 Updated by Luke Kanies over 7 years ago

mgagne wrote:

So turning a blind eye on a problem I find important would be the solution? You can certainly understand that this “solution” is just a patch, doesn’t solve anything at all under the hood and is not acceptable.

It’s not so much that it’s a solution, as that it’s what we can do right now. I’ve spent a ton of time profiling Puppet, looking for ways to make it faster, but I haven’t always succeeded. We’re always looking for help making it faster, and we’d be glad to make any changes toward better performance, but everything’s a tradeoff.

There is certainly slow Ruby’s components being used by Puppet and a profiling would be one of the way to determine what is causing all this CPU usage while waiting and doing nothing.

I’ve done this before, but maybe it’s time to do it again. If you’re willing to spend some time profiling, I’m willing to sew what I can do to trim the code you find is causing the slowdowns.

On the other hand, I think version 1.9 of Ruby would improve overall performances. Will Puppet be compatible with Ruby 1.9?

I don’t actually know; I haven’t had a chance to test it yet. I expect there will be at least some minor incompatibilities, but it shouldn’t be anything significant from what I’ve seen.

Don’t get me wrong, I like Puppet and what it can bring overall. However there is certainly room for improvements and I hope we can find a way to make it better.

Clearly, and I’ll always do what I can to help people improve Puppet. Let me know if I can help at all with the profiling.

#5 Updated by Mathieu Gagné over 7 years ago

Hi Luke,

I found this blog post which seems to describe the exact same problem I’m currently experimenting: http://timetobleed.com/ruby-threading-bugfix-small-fix-goes-a-long-way/

Do you think it is related in some way?

#6 Updated by Luke Kanies over 7 years ago

mgagne wrote:

Hi Luke,

I found this blog post which seems to describe the exact same problem I’m currently experimenting: http://timetobleed.com/ruby-threading-bugfix-small-fix-goes-a-long-way/

Do you think it is related in some way?

Hmm, could be, but we don’t really create any threads on our own, so we don’t have threads die. Well, we have an EventLoop thread, but it never goes away, AFAIK.

It’d certainly be interesting to see how it performed with that patch applied.

#7 Updated by Pavel Veretennikov over 7 years ago

  • Affected Puppet version changed from 0.24.5 to 0.24.6

I have the same problem. I have two almost identical servers, one affected and one unaffected:

Unaffected [14:52 root@wn-2 ~]$ rpm -qa | egrep “puppet|ruby” ruby-libs-1.8.5-5.el5_2.5 ruby-shadow-1.4.1-7.el5 ruby-1.8.5-5.el5_2.5 puppet-0.24.6-1.el5

[14:44 root@wn-2 ~]$ strace puppet —version 2>&1 | grep rt_sigprocmask | wc -l 0

Affected [14:54 root@bg-1 ~]$ rpm -qa | egrep “puppet|ruby” rubygems-0.9.4-1.el5 ruby-libs-1.8.5-5.el5_2.5 ruby-1.8.5-5.el5_2.5 ruby-rdoc-1.8.5-5.el5_2.5 ruby-shadow-1.4.1-7.el5 puppet-server-0.24.6-1.el5 ruby-irb-1.8.5-5.el5_2.5 puppet-0.24.6-1.el5

[14:54 root@bg-1 ~]$ strace puppet —version 2>&1 | grep rt_sigprocmask | wc -l 492362

#8 Updated by Mathieu Gagné over 7 years ago

vermut wrote:

I have the same problem. I have two almost identical servers, one affected and one unaffected: […]

Is one of your server multi-core?

#9 Updated by Deepak Giridharagopal about 7 years ago

  • Affected Puppet version changed from 0.24.6 to 0.24.8

I’m seeing this with 0.24.8, using RHEL5 and their stock version of ruby (1.8.5).

This is a single-core VM. The “puppet” process spends 40-60% time in “system time”, which lengthens the time a single puppet run takes from a few minutes to > 20 minutes.

#10 Updated by Mathieu Gagné almost 7 years ago

Hi,

The “bug” is caused by “—with-pthread” within Ruby, an option often disabled by the Ruby community.

http://timetobleed.com/fix-a-bug-in-rubys-configurein-and-get-a-30-performance-boost/ https://bugs.launchpad.net/ubuntu/+source/ruby1.8/+bug/307462

I rebuilt a new Debian package with —disable-pthread —disable-tcl-thread, installed it and my CPU stopped melting.

Also available in: Atom PDF