The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #14355

performance issue with 2.7.14

Added by konrad rzentarzewski almost 2 years ago. Updated almost 2 years ago.

Status:InvestigatingStart date:05/08/2012
Priority:NormalDue date:
Assignee:Carl Caum% Done:

0%

Category:compiler
Target version:-
Affected Puppet version:2.7.14 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket may be automatically exported to the PUP project on JIRA using the button below:


Description

since our upgrade to 2.7.14 at may 3rd load average on all our puppetmaster has increased 2 times.

please investigate.

we use official rpms from puppetlabs, ie.

<16>root@punch(2)~$ rpm -qi puppet
Name        : puppet                       Relocations: (not relocatable)
Version     : 2.7.14                            Vendor: (none)
Release     : 1.el5                         Build Date: Wed 02 May 2012 08:52:02 PM CEST
Install Date: Thu 03 May 2012 12:29:04 AM CEST      Build Host: rpm-builder.puppetlabs.lan
Group       : System Environment/Base       Source RPM: puppet-2.7.14-1.el5.src.rpm
Size        : 3174857                          License: ASL 2.0
Signature   : RSA/SHA1, Wed 02 May 2012 09:01:02 PM CEST, Key ID 1054b7a24bd6ec30
URL         : http://puppetlabs.com
Summary     : A network tool for managing many disparate systems
Description :
Puppet lets you centrally manage every important aspect of your system using a
cross-platform specification language that manages all the separate elements
normally aggregated in different files, like users, cron jobs, and hosts,
along with obviously discrete elements like packages, services, and files.

2.7.14.png - example load graph from one of our production puppetmasters (33.8 KB) konrad rzentarzewski, 05/08/2012 05:07 am

RunTime.tiff - Runtime of puppet agent. (52.1 KB) Steve Traylen, 05/09/2012 06:59 am

puppetmaster.tiff - Puppetmaster CPU utilization. (77.6 KB) Steve Traylen, 05/09/2012 06:59 am

puppetdb.png - runtime decrease after changed storeconfigs = active_record => puppetdb (76.2 KB) konrad rzentarzewski, 05/21/2012 12:53 pm

History

#1 Updated by Daniel Pittman almost 2 years ago

  • Status changed from Unreviewed to Investigating
  • Assignee set to Carl Caum

Carl, I think this is probably something you want to look into.

#2 Updated by Steve Traylen almost 2 years ago

Just confirming similar behavior when we went from 2.7.11 to 2.7.14:

Running:

time puppet master --debug  --color=false  --compile steve01.cern.ch 

for each of the intermediate versions we skipped. 2.7.11 –> 2.7.12 seems to be the significant one.

This is RHEL6 with stock ruby: ruby 1.8.7 (2011-06-30 patchlevel 352) [x86_64-linux]

2.7.14

real 0m23.511s, user 0m18.509s, sys 0m4.284s, notice: Compiled catalog for steve01.cern.ch in environment devel in 19.10 seconds

2.7.13

real 0m21.459s, user 0m16.895s, sys 0m3.882s, notice: Compiled catalog for steve01.cern.ch in environment devel in 17.13 seconds

2.7.12

real 0m21.481s, user 0m16.786s, sys 0m3.989s, notice: Compiled catalog for steve01.cern.ch in environment devel in 17.15 seconds

2.7.11

real 0m15.248s, user 0m12.150s, sys 0m2.440s, notice: Compiled catalog for steve01.cern.ch in environment devel in 10.95 seconds

The plots show the step on monday when we did the 2.7.11 –> 2.7.14 update.

#3 Updated by konrad rzentarzewski almost 2 years ago

it seems it’s somehow related to storeconfigs code – puppetdb seem to mitigate it. i don’t know if Steve is using stored configs, i think it should be important bit of information.

#4 Updated by Steve Traylen almost 2 years ago

Hi, strored configs are enabled but we are hardly using them , less that 5 things shared amongst 5 hosts… But maybe its just having it enabled.

The new rc’s release notes mention a lot of speedups in compilation.

puppetdb .. the future and it’s postgres. :–)

#5 Updated by Carl Caum almost 2 years ago

Are you using module metadata? Do any of your modules contain a metadata.json file in the module’s root directory? If so, try out the patch outlined in this thread.

http://groups.google.com/group/puppet-dev/browse_thread/thread/5634e66c7d1d0001/25fa5711d7ba655c?lnk=gst&q=%22Once+again%2C+a+patch+is+attached.+Please+test+this+and+provide+feedback.%22#25fa5711d7ba655c

We will be including this patch in 2.7.15.

Also available in: Atom PDF