The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com
https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:
Security fix using safe_yaml caused a drastic performance hit in 2.7.22 (and probably others)
|Assignee:||Charlie Sharpsteen||% Done:|
|Affected Puppet version:||3.2.2||Branch:|
Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com
I’ve been testing out 2.7.22 as an update to 2.7.19 due to CVE-2013-3567. With nothing else changed other than the version, performance goes right down the crapper. I downgraded to 2.7.21 and performance returned to normal.
Here is a test puppet run with ‘time’ and ‘—summarize’ against a 2.7.22 master and then a 2.7.19 master.
2.7.22 ====== notice: Finished catalog run in 114.74 seconds Changes: Total: 3 Events: Success: 3 Total: 3 Resources: Out of sync: 3 Changed: 3 Total: 3079 Skipped: 6 Time: Resources: 0.00 Filebucket: 0.00 Group: 0.00 Yumrepo: 0.01 Cron: 0.02 User: 0.07 Package: 0.71 Exec: 10.27 Config retrieval: 108.95 Last run: 1371671599 Total: 185.09 Service: 3.57 File: 61.47 Version: Config: 1371671173 Puppet: 2.7.19 real 6m2.831s user 1m37.205s sys 0m15.160s 2.7.19 ====== notice: Finished catalog run in 77.05 seconds Changes: Total: 3 Events: Success: 3 Total: 3 Resources: Out of sync: 3 Changed: 3 Total: 3079 Skipped: 6 Time: Resources: 0.00 Filebucket: 0.00 Group: 0.01 Yumrepo: 0.01 Cron: 0.02 User: 0.07 Package: 0.38 Last run: 1371671871 File: 23.13 Service: 3.89 Config retrieval: 52.85 Exec: 9.73 Total: 90.09 Version: Config: 1371671735 Puppet: 2.7.19 real 3m6.728s user 1m39.787s sys 0m16.342s
#3 Updated by RoMan Shipovskij almost 3 years ago
We are using two masters, first for catalog compilation, second for report collection (~900-1000 clients per minute). After upgrading from 3.2.1 to 3.2.2 on first master nothing changed, but on second cpu usage increase from ~30% to ~80%, that more then two times.
#4 Updated by Andrew Gaffney almost 3 years ago
This only impacting report processing seems to be verified in my setup.
I just did some work to offload report processing completely from puppet (nginx treats the PUT request like a DAV request and dumps reports to a temp dir, and then a custom foreman rake job processes them). I was then able to upgrade to 2.7.22 again without any apparent issues.
#5 Updated by Roger Kennedy almost 3 years ago
- Status changed from Unreviewed to Closed
- Assignee set to Roger Kennedy
From the EOL announcement for 2.6 (https://groups.google.com/forum/#!topic/puppet-users/l7mzoeojKq8), Puppet 2.7 is in a security-fix only stage. The true EOL for 2.7 is in October (https://groups.google.com/forum/#!msg/puppet-users/8JEy7wY5VPs/F7TCazGDNvoJ), so you should plan on upgrading.
Aside: Both of my 2.7=>3.x upgrades were basically painless. It’s really not as scary as it sounds!
#9 Updated by Charlie Sharpsteen almost 3 years ago
- Category set to reports
- Status changed from Re-opened to Closed
- Assignee changed from Roger Kennedy to Charlie Sharpsteen
- Keywords set to safe_yaml performance
This is a side-effect of the security work done to fix CVE-2013-3567 whereby specially constructed YAML payloads could be used to execute arbitrary code on the Puppet master. Unfortunately, the safe_yaml library we are using to lock out remote code execution comes with a heavy performance penalty that is showing up when the master is processing YAML reports.
We are approaching this problem by shifting the preferred report format to JSON. The work required to accomplish this has been completed as part of #21427 and should be released in Puppet 3.3.0.