The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #21337

Security fix using safe_yaml caused a drastic performance hit in 2.7.22 (and probably others)

Added by Andrew Gaffney almost 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:
Priority:NormalDue date:
Assignee:Charlie Sharpsteen% Done:

0%

Category:reports
Target version:-
Affected Puppet version:3.2.2 Branch:
Keywords:safe_yaml performance

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

I’ve been testing out 2.7.22 as an update to 2.7.19 due to CVE-2013-3567. With nothing else changed other than the version, performance goes right down the crapper. I downgraded to 2.7.21 and performance returned to normal.

Here is a test puppet run with ‘time’ and ‘—summarize’ against a 2.7.22 master and then a 2.7.19 master.

2.7.22
======
notice: Finished catalog run in 114.74 seconds 
Changes: 
            Total: 3 
Events: 
          Success: 3 
            Total: 3 
Resources: 
      Out of sync: 3 
          Changed: 3 
            Total: 3079 
          Skipped: 6 
Time: 
        Resources: 0.00 
       Filebucket: 0.00 
            Group: 0.00 
          Yumrepo: 0.01 
             Cron: 0.02 
             User: 0.07 
          Package: 0.71 
             Exec: 10.27 
   Config retrieval: 108.95 
         Last run: 1371671599 
            Total: 185.09 
          Service: 3.57 
             File: 61.47 
Version: 
           Config: 1371671173 
           Puppet: 2.7.19 

real    6m2.831s 
user    1m37.205s 
sys 0m15.160s 

2.7.19
======
notice: Finished catalog run in 77.05 seconds 
Changes: 
            Total: 3 
Events: 
          Success: 3 
            Total: 3 
Resources: 
      Out of sync: 3 
          Changed: 3 
            Total: 3079 
          Skipped: 6 
Time: 
        Resources: 0.00 
       Filebucket: 0.00 
            Group: 0.01 
          Yumrepo: 0.01 
             Cron: 0.02 
             User: 0.07 
          Package: 0.38 
         Last run: 1371671871 
             File: 23.13 
          Service: 3.89 
   Config retrieval: 52.85 
             Exec: 9.73 
            Total: 90.09 
Version: 
           Config: 1371671735 
           Puppet: 2.7.19 

real    3m6.728s 
user    1m39.787s 
sys 0m16.342s

Related issues

Related to Puppet - Feature #21427: Deprecate YAML for network data transmission Closed

History

#1 Updated by Andrew Gaffney almost 3 years ago

That didn’t format quite how I intended. The bottom line is that the total time to run puppet doubled with 2.7.22.

#2 Updated by Charlie Sharpsteen almost 3 years ago

  • Description updated (diff)

#3 Updated by RoMan Shipovskij almost 3 years ago

We are using two masters, first for catalog compilation, second for report collection (~900-1000 clients per minute). After upgrading from 3.2.1 to 3.2.2 on first master nothing changed, but on second cpu usage increase from ~30% to ~80%, that more then two times.

#4 Updated by Andrew Gaffney almost 3 years ago

This only impacting report processing seems to be verified in my setup.

I just did some work to offload report processing completely from puppet (nginx treats the PUT request like a DAV request and dumps reports to a temp dir, and then a custom foreman rake job processes them). I was then able to upgrade to 2.7.22 again without any apparent issues.

#5 Updated by Roger Kennedy almost 3 years ago

  • Status changed from Unreviewed to Closed
  • Assignee set to Roger Kennedy

From the EOL announcement for 2.6 (https://groups.google.com/forum/#!topic/puppet-users/l7mzoeojKq8), Puppet 2.7 is in a security-fix only stage. The true EOL for 2.7 is in October (https://groups.google.com/forum/#!msg/puppet-users/8JEy7wY5VPs/F7TCazGDNvoJ), so you should plan on upgrading.

Aside: Both of my 2.7=>3.x upgrades were basically painless. It’s really not as scary as it sounds!

#6 Updated by Andrew Gaffney almost 3 years ago

  • Status changed from Closed to Re-opened

And what about Roman saying he saw the same exact thing in 3.2.2?

#7 Updated by Andrew Gaffney almost 3 years ago

Also, people like me with years of legacy puppet setup that use dynamic variable scoping all over the place don’t have a simple upgrade path.

#8 Updated by Roger Kennedy almost 3 years ago

  • Affected Puppet version set to 3.2.2

Sorry, I missed the part about still being in 3.2.

#9 Updated by Charlie Sharpsteen almost 3 years ago

  • Category set to reports
  • Status changed from Re-opened to Closed
  • Assignee changed from Roger Kennedy to Charlie Sharpsteen
  • Keywords set to safe_yaml performance

This is a side-effect of the security work done to fix CVE-2013-3567 whereby specially constructed YAML payloads could be used to execute arbitrary code on the Puppet master. Unfortunately, the safe_yaml library we are using to lock out remote code execution comes with a heavy performance penalty that is showing up when the master is processing YAML reports.

We are approaching this problem by shifting the preferred report format to JSON. The work required to accomplish this has been completed as part of #21427 and should be released in Puppet 3.3.0.

Also available in: Atom PDF