The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #1395

puppet memory usage

Added by Peter Meier almost 8 years ago. Updated almost 3 years ago.

Status:Needs More InformationStart date:07/06/2008
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:plumbing
Target version:-
Affected Puppet version:0.25.1 Branch:
Keywords:memory leaks

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

In a recent discussion about using puppetd with xinetd (“main thread”:http://groups.google.com/group/puppet-users/browse_thread/thread/50c94008469fc395/11687cd0a633f8f4 ) it came out that there is some investigation needed on puppetd’s memory usage.

Different people noticed that running puppetd is using a lot of footprint (links follow), while other noticed that it uses nearly none (links follow)

Notices about high memory usage:

VIRT/RES
"141m/61m":http://groups.google.com/group/puppet-users/msg/75a8bb3b6652bae6
"127m/82m":http://groups.google.com/group/puppet-users/msg/a4a43d3edd585c03 64 bit
"166m/83m":http://groups.google.com/group/puppet-users/msg/1825ed57ff84287f
"183m/96m":http://groups.google.com/group/puppet-users/msg/ecf2cd36b6e1e3a6 version 0.24.4, 64 bit

Notices about low memory usage:

"nothing specific":http://groups.google.com/group/puppet-users/msg/aac9fade1b906bd9 version 0.24.4
"57m/46m":http://groups.google.com/group/puppet-users/msg/680b2b2760aa8890 version 0.23.1

Personally I can add another memory usage on a 32bit machine, which is rather low: 85m virt and 69m res, running 0.24.4

Luke then “noticed”:http://groups.google.com/group/puppet-users/msg/573a76e32fba3eb0 that we should investigate on memory usage in the near future.

Besides the memory consumption of puppetd, also the memory consumption of puppetmaster was discussed. There Duncan Hill noticed that splitting the fileserver off the the main puppetmaster “improved the situation heavily”:http://groups.google.com/group/puppet-users/msg/a7083b931b5c4cda , however he “noticed”:http://groups.google.com/group/puppet-users/msg/f68347fdb69b8496 that the manifest puppetmaster still consumes a lot of memory.

There have been also “other reports about memory usage trashing the master and the client”:http://groups.google.com/group/puppet-users/browse_thread/thread/26aa8025de704cc4/1c9608f58360cdbb, however this might not be related.

Quick Summary:

We have seen different environments reporting a lot of memory usage. Most of them seem to be 64bit machines using 0.24.4. One report about low memory usage was a 0.23.1 version.

Personal additional assumption: And for a first glance it looks like that 64bit machines double the memory consumption. (Take care of this assumption)

vmadt138-test1_pupped_leak.txt Magnifier (54.5 KB) Sergio Ballestrero, 02/13/2012 02:49 am


Related issues

Duplicated by Puppet - Bug #2108: Puppetd holding memory between runs Duplicate 03/25/2009

History

#1 Updated by Ohad Levy almost 8 years ago

I can also add my 2c, puppetmaster uses huge amount of memory on a 32bit platform as well, currently a puppetmaster with about 150 clients, would need to be restarted every two or three days as each puppetmaster process crosses the 2gb usage. using a tool like monit can prevent puppet from abusing memory, however, I would prefer to find the root cause of the memory abuse (file server operations?)

#2 Updated by Andrew Shafer almost 8 years ago

  • Status changed from Unreviewed to Accepted
  • Assignee set to Andrew Shafer

I’ll be stylin' and profilin'

#3 Updated by Duncan Hill almost 8 years ago

A data point that may help track down where the issue lies.

We run puppetmaster in a VMware server 32-bit CentOS 5 VM, with 384 MB of memory allocated – tiny, but perfectly functional.

We split puppetmaster into two (I’m the D. Hill referenced in the ticket description) – one handling file serving, the other handling manifest serving. The file serving puppetmaster holds steady at 54M VIRT, 39M RES. The manifest serving puppetmaster hold steady at 74M VIRT, 55M RES until storeconfigs is enabled, with a backing server of PostgreSQL 8.1.11 (provided in the base CentOS 5 installation).

Once storeconfigs is enabled, the memory usage starts climbing, almost as if there’s a leak. Once the VM starts swapping, we have no choice but to stop puppetmaster, wait a few minutes for the load to come back to a sensible level, and start puppetmaster again. Rinse, repeat.

I’ve checked file handles with lsof, and the number seems to hold steady around 50 – 60 files, whether the memory usage is 50 MB or 350 MB.

#4 Updated by Andrew Shafer almost 8 years ago

I found and fixed one leak in puppetd.

Puppetd was leaking report objects into an array.

I was running puppetd on one minute intervals and it grew about 20 MB over 8 hours.

Now running on 30 runintervals it has grown ~160K in about the same period, but it isn’t gaining Ruby objects.

The patch is available from my github (littleidea) in the report_leak branch.

I don’t want to close this issue, but I do want to give people something to work with.

I made extensive use of the memory_profiler.rb from: http://reductivelabs.com/trac/puppet/ticket/1131 (It will basically start a separate thread that keeps an inventory of ObjectSpace prints an audit to a file every so often. Make sure to add the ‘start’ after puppet daemonizes)

This told me I was leaking Puppet:Transaction::Reports.

I modified it a bit as I went but I got mileage doing stuff like this to mark when the new Object was created:

if o.class == Puppet::Transaction::Report and !report_ids.include?(o.object_id)

file.puts('New report')
file.puts(o.inspect)
report_ids << o.object_id

end

And this to figure out what the structure of the whatever was holding it if o.inspect =~ /Puppet::Transaction::Report/ and (o.class == Hash or o.class == Array)

file.puts(o.inspect)

end

I used this in conjunction with some basic dtrace scripts, mainly of the form:

sudo dtrace -s find_sync.d -p <pid#>

where find_sync.d looks like ruby$target:::object-create-start /copyinstr(arg0) == “Sync”/ {

printf("%s was allocated in file %s, line number %d\n", copyinstr(arg0), copyinstr(arg1), arg2);

}

That will tell you the line number and file where the Sync objects get created, as an entry point to start looking at code.

I’ll keep revisiting performance and memory problems, but I have to balance that with other work. I can solve problems much faster if I have a manifest or setup that can reliably reproduce the issue. Alternatively, if you are having memory problems try using the memory_profiler and dtrace with your setup.

I didn’t look at storeconfigs yet, but I think there are more perf problems their than just memory.

If anyone in the community has strong dtrace fu, or learns a new trick share with the list please. The ruby probes are somewhat simplistic and I think we probably need to get deeper knowledge to solve some of the issues.

#5 Updated by Andrew Shafer almost 8 years ago

Also, I have a relatively simple setup right now, but I’m not getting growth in the puppetmasterd.

If someone can give me a setup that consistently produces the problems, I’ll figure it out much faster.

help me help you :/

#6 Updated by Andrew Shafer almost 8 years ago

Oh yeah, when is Linux getting something like dtrace?

#7 Updated by Alexander Schaber over 7 years ago

andrew wrote:

Also, I have a relatively simple setup right now, but I’m not getting growth in the puppetmasterd.

If someone can give me a setup that consistently produces the problems, I’ll figure it out much faster.

help me help you :/

Hello,

I’ve got a similar problem, and I’d gladly provide you will all the required files. The only Problem, even though I compressed them with bzip2. The Package is 44M. I’m using the version shipped with Ubuntu 8.04: 2.24.4.

The Package contains all configuration files as well as the data files I want puppet to distribute: www.alexanderschaber.de/puppet.tar.bz2 (It will be fully uploaded in about 10 minutes)

#8 Updated by Alexander Schaber over 7 years ago

Correcting .. it’s 36MB .. ah and btw.: it’s fully uploaded now..

#9 Updated by Oliver Hookins over 7 years ago

I’m getting significant memory usage on 0.24.5 on puppetd clients that are sitting around waiting for the CA to sign their cert. I’m not sure if this is the same issue as reported but thought it worth mentioning. One machine running Debian Etch has had puppetd running as a daemon for only 5 days and it has already swallowed a significant chunk of memory:

# ps u -C ruby
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     17715  9.6 60.9 750952 626340 ?       Dsl  Sep12 689:18 ruby /usr/sbin/puppetd -w 0

about 600MB! I’m also finding puppetd tends to use a lot of CPU time while waiting for the CA to sign the cert, and attempts to talk to the puppetmaster a lot more frequently than it should… but perhaps that’s a story for another ticket.

#10 Updated by Markus Roberts over 6 years ago

  • Category set to plumbing
  • Assignee changed from Andrew Shafer to Markus Roberts
  • Target version set to 0.25.2
  • Affected Puppet version changed from 0.24.4 to 0.25.1
  • Keywords set to memory leaks

In doing automated testing for several other bugs I’ve been able to reproduce the memory leak in 0.25.1

I’m attempting to isolate and diagnose.

#11 Updated by konrad rzentarzewski over 6 years ago

i’m also getting leaks with storeconfigs and postgresql driver (puppetmaster=0.25.1; passenger=2.2.2). now i’m killing threads with >768mb addressed (my server is getting killed without it). webrick also was going as high as 2gig (but it was uncapable of handling the load). i haven’t tried without stored configs. currently 57 nodes connects to this master.

2009-12-08 18:02:01 killed: pid=14966 shared=1500mb (limit: 1024mb) private=1123mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:03:02 killed: pid=15061 shared=1480mb (limit: 1024mb) private=1108mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:03:02 killed: pid=15169 shared=3046mb (limit: 1024mb) private=1673mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:05:02 killed: pid=14927 shared=1844mb (limit: 1024mb) private=1541mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:35:02 killed: pid=20125 shared=3076mb (limit: 1024mb) private=1594mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 18:39:01 killed: pid=21464 shared=1389mb (limit: 1024mb) private=976mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:03:03 killed: pid=24896 shared=1490mb (limit: 1024mb) private=1078mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:05:01 killed: pid=24795 shared=2634mb (limit: 1024mb) private=1689mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:06:02 killed: pid=24908 shared=2623mb (limit: 1024mb) private=1616mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:08:02 killed: pid=24844 shared=1503mb (limit: 1024mb) private=1254mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:11:02 killed: pid=26456 shared=1391mb (limit: 1024mb) private=1077mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 19:34:01 killed: pid=29484 shared=2502mb (limit: 1024mb) private=1760mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:06:01 killed: pid=1890 shared=1732mb (limit: 1024mb) private=1179mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:07:02 killed: pid=1691 shared=989mb (limit: 1024mb) private=817mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:08:01 killed: pid=2834 shared=1470mb (limit: 1024mb) private=1031mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:11:01 killed: pid=3270 shared=1867mb (limit: 1024mb) private=1575mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:36:01 killed: pid=6839 shared=1518mb (limit: 1024mb) private=1101mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:39:01 killed: pid=7218 shared=1534mb (limit: 1024mb) private=1054mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:41:01 killed: pid=7271 shared=2603mb (limit: 1024mb) private=1538mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:43:01 killed: pid=7737 shared=1514mb (limit: 1024mb) private=1266mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 20:45:02 killed: pid=8394 shared=971mb (limit: 1024mb) private=832mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 21:05:02 killed: pid=10905 shared=1729mb (limit: 1024mb) private=1257mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 21:12:01 killed: pid=10966 shared=2635mb (limit: 1024mb) private=1534mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:44:11 killed: pid=11703 shared=2615mb (limit: 1024mb) private=1409mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:44:11 killed: pid=11732 shared=4178mb (limit: 1024mb) private=1970mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 22:46:01 killed: pid=12141 shared=1389mb (limit: 1024mb) private=917mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:08:01 killed: pid=16876 shared=1494mb (limit: 1024mb) private=1120mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:16:01 killed: pid=17667 shared=1742mb (limit: 1024mb) private=1068mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:16:01 killed: pid=17682 shared=1799mb (limit: 1024mb) private=1002mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:18:01 killed: pid=18584 shared=2151mb (limit: 1024mb) private=1318mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:21:01 killed: pid=19036 shared=1659mb (limit: 1024mb) private=1137mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:40:01 killed: pid=21413 shared=1522mb (limit: 1024mb) private=1109mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:41:02 killed: pid=21262 shared=1060mb (limit: 1024mb) private=775mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:48:01 killed: pid=22471 shared=1831mb (limit: 1024mb) private=1148mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-08 23:50:01 killed: pid=23236 shared=1475mb (limit: 1024mb) private=1042mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:11:01 killed: pid=26442 shared=1777mb (limit: 1024mb) private=1234mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:20:02 killed: pid=27678 shared=1843mb (limit: 1024mb) private=1115mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:22:01 killed: pid=28141 shared=2267mb (limit: 1024mb) private=1471mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
2009-12-09 00:23:02 killed: pid=28457 shared=1174mb (limit: 1024mb) private=851mb (limit: 512mb) process=Rack: /usr/share/puppet/rack/puppetmasterd
/usr/local/bin/ruby /usr/local/bin/passenger-memory-stats | grep "/rack/"|grep "[0-9] MB"|while read procpid threads shared mb private mb crap ; do
        shared=$(echo "$shared"|cut -f1 -d.)
        private=$(echo "$private"|cut -f1 -d.)
        if [ $shared -gt 1512 ] || [ $private -gt 768 ] ; then
                kill -9 $procpid
                echo "$(date '+%Y-%m-%d %H:%M:%S') killed: pid=$procpid shared=${shared}mb (limit: 1024mb) private=${private}mb (limit: 512mb) process=$crap" >>/tmp/spuchlaki.log
        fi
done

#12 Updated by Markus Roberts over 6 years ago

Konrad —

Can you quantify how long (in wall clock time & puppet runs) a puppetmaster goes before reaching this point? And is the process gradual/continuous or abrupt?

Also, I note that the limits specified in the log file lines (1GB shared, 512MB private) are not the actual limits enforced by the script.

And finally, would you be willing to run test/instrumented versions to help pin the problem down?

— Markus

#13 Updated by konrad rzentarzewski over 6 years ago

it’s countinous while it starts, goes ~30mb/sec after it started and never finishes (server memory exhausted or killed by script). i’m not sure – but can verify – it occurs after the run (ie. when no pupppet is connected).

can you provide info about test procedures/versions?

i know about limits, that’s why i provided a script (i forgot to change the number that goes to log). limit was 768 since 18:00.

#14 Updated by konrad rzentarzewski over 6 years ago

EDIT: typo

#15 Updated by Markus Roberts over 6 years ago

I’m not sure – but can verify – it occurs after the run (ie. when no puppet is connected).

That would be very interesting if you can confirm. I’m not sure what it means though—there shouldn’t be anything much happening at that point. That’s generally the sign of a great clue.

If you can reliably reproduce this I’d love to work with you to nail it down.

#16 Updated by konrad rzentarzewski over 6 years ago

interesting screenshot below. i’ve taken it just after doing “service passenger stop” – so leaked processed were not killed (despite the apache process exited) and they keept their active mysql connections (see netstat and mysqladmin). btw. i’ve switched postgres to mysql as stored configs backend, as it appears to work more stable (no sudden disconnects) and leak less.

Every 2.0s: passenger-memory-stats | grep -A50 "Passenger processes" ; netstat -nalp|grep ESTAB ; tail -10 /var/log/httpd/passenger_access.log ; mysqladmin -u puppet --password...  Thu Dec 10 23:22:45 2009

[44m[1m[33m----------- Passenger processes -----------
[37mPID    Threads  VMSize     Private    Name
-------------------------------------------[0m
1193   1        59.7 MB    0.1 MB     grep -A50 Passenger processes
26882  1        3028.7 MB  1756.4 MB  Rack: /usr/share/puppet/rack/puppetmasterd
26894  1        2974.9 MB  821.1 MB   Rack: /usr/share/puppet/rack/puppetmasterd
28462  1        1530.1 MB  719.4 MB   Rack: /usr/share/puppet/rack/puppetmasterd
### Processes: 4
### Total private dirty RSS: 3296.99 MB
tcp        0      0 127.0.0.1:3306              127.0.0.1:39207             ESTABLISHED 18452/mysqld
tcp        0      0 10.0.4.4:39634              10.0.100.2:3493             ESTABLISHED 2399/upsmon
tcp        0      0 127.0.0.1:39153             127.0.0.1:3306              ESTABLISHED 26894/puppetmasterd
tcp        0      0 127.0.0.1:39145             127.0.0.1:3306              ESTABLISHED 26882/puppetmasterd
tcp        0      0 127.0.0.1:39207             127.0.0.1:3306              ESTABLISHED 28462/puppetmasterd
tcp        0      0 10.0.4.4:22                 100.100.100.100:44106       ESTABLISHED 7157/sshd: kondi [p
tcp        0      0 127.0.0.1:3306              127.0.0.1:39145             ESTABLISHED 18452/mysqld
tcp        0      0 127.0.0.1:3306              127.0.0.1:39153             ESTABLISHED 18452/mysqld
172.16.101.98 - - [10/Dec/2009:23:20:59 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:02 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:04 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| Id  | User   | Host            | db     | Command | Time | State | Info                  |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| 287 | puppet | localhost:39145 | puppet | Sleep   | 486  |       |                       |
| 293 | puppet | localhost:39153 | puppet | Sleep   | 487  |       |                       |
| 328 | puppet | localhost:39207 | puppet | Sleep   | 414  |       |                       |
| 389 | puppet | localhost       |        | Query   | 0    |       | show full processlist |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+

aaah, and sth more – if left not killed, those processes appear to return some of its memory – weird.

Every 2.0s: passenger-memory-stats | grep -A50 "Passenger processes" ; netstat -nalp|grep ESTAB ; tail -10 /var/log/httpd/passenger_access.log ; mysqladmin -u puppet --password...  Thu Dec 10 23:29:20 2009

[44m[1m[33m----------- Passenger processes -----------
[37mPID    Threads  VMSize     Private    Name
-------------------------------------------[0m
6987   1        59.7 MB    0.1 MB     grep -A50 Passenger processes
26882  1        3028.7 MB  976.7 MB   Rack: /usr/share/puppet/rack/puppetmasterd
26894  1        2974.9 MB  1684.7 MB  Rack: /usr/share/puppet/rack/puppetmasterd
28462  1        2051.1 MB  729.6 MB   Rack: /usr/share/puppet/rack/puppetmasterd
### Processes: 4
### Total private dirty RSS: 3391.05 MB
tcp        0      0 127.0.0.1:3306              127.0.0.1:39207             ESTABLISHED 18452/mysqld
tcp        0      0 10.0.4.4:39634              10.0.100.2:3493             ESTABLISHED 2399/upsmon
tcp        0      0 127.0.0.1:39153             127.0.0.1:3306              ESTABLISHED 26894/puppetmasterd
tcp        0      0 127.0.0.1:39145             127.0.0.1:3306              ESTABLISHED 26882/puppetmasterd
tcp        0      0 127.0.0.1:39207             127.0.0.1:3306              ESTABLISHED 28462/puppetmasterd
tcp        0      0 10.0.4.4:22                 100.100.100.100:44106       ESTABLISHED 7157/sshd: kondi [p
tcp        0      0 127.0.0.1:3306              127.0.0.1:39145             ESTABLISHED 18452/mysqld
tcp        0      0 127.0.0.1:3306              127.0.0.1:39153             ESTABLISHED 18452/mysqld
172.16.101.98 - - [10/Dec/2009:23:20:59 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:00 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:01 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:02 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:03 +0100] "POST /RPC2 HTTP/1.1" 200 120 "-" "XMLRPC::Client (Ruby 1.8.6)"
172.16.101.98 - - [10/Dec/2009:23:21:04 +0100] "POST /RPC2 HTTP/1.1" 200 170 "-" "XMLRPC::Client (Ruby 1.8.6)"
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| Id  | User   | Host            | db     | Command | Time | State | Info                  |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+
| 287 | puppet | localhost:39145 | puppet | Sleep   | 890  |       |                       |
| 293 | puppet | localhost:39153 | puppet | Sleep   | 891  |       |                       |
| 328 | puppet | localhost:39207 | puppet | Sleep   | 818  |       |                       |
| 469 | puppet | localhost       |        | Query   | 0    |       | show full processlist |
+-----+--------+-----------------+--------+---------+------+-------+-----------------------+

(my server has died shortly after that – watchdog reboots it when free physical mem <2000mb)

#17 Updated by konrad rzentarzewski over 6 years ago

good news… it seems not to be related to storeconfigs. i have small function on puppetmaster called @:template_search@ and it seems to be locking itself and eating all memory.

# /var/lib/puppet/lib/puppet/parser/functions/template_search.rb
 
Puppet::Parser::Functions::newfunction(:template_search, :type => :rvalue, :doc =>
    "Find first existing file and run puppet template function on it. If you want
    to concatenate multiple templates try var = template_search(file1,file2) +
    template_search(file3,file4) and then use var in your resources.") do |vals|
            ret = nil
            errors = Array.new
            vals.each do |file|
                debug "in :template_search find %s" % file
                ret = nil
                begin
                    ret = function_template(file)
                rescue Exception=>e
                    if /could.not.find.template/i =~ e
                        debug "in :template_search not found %s" % file
                    else
                        raise Puppet::ParseError, "in :template_search exception in %s: %s" % [file,e]
                        errors.push(e)
                    end
                end
                if ret
                    debug "in :template_search found %s" % file
                    break
                end
            end
            if ret
                ret
            else
                if errors.length > 0
                    raise Puppet::ParseError, "Errors from :template function: %s" %
                        errors.join(", ")
                else
                    raise Puppet::ParseError, "Could not find any files from %s" %
                        vals.join(", ")
                end
            end
end

i call it with:

    class backend inherits exim {
        file {"exim.conf":
            path    => "/etc/exim/exim.conf",
            content  => template_search("exim/$hostname/exim-backend.erb", "exim/$environment/exim-backend.erb"),
        }

in logs it hangs on:

Dec 14 19:10:40 punch puppetmasterd[16865]: (Scope(Class[exim::backend])) in :template_search find exim/mailser01/exim-backend.erb

(where @mailser01@ is substituted @$hostname@)

any idea why this is happening?

#18 Updated by konrad rzentarzewski over 6 years ago

i’m making separate ticket for it (#2930) as it seems to be endless loop, not memory leak. i’ve fixed my functions and now my puppetmaster processes are stable (~100mb on x86_64 running passenger/ruby-enterprise).

#19 Updated by Markus Roberts over 6 years ago

  • Target version changed from 0.25.2 to 0.25.3

Absent something definite & reproducible or a strategy for finding such, there’s not much we can expect to accomplish on this ticket in the next few days.

#20 Updated by Markus Roberts over 6 years ago

  • Target version changed from 0.25.3 to 0.25.4

#21 Updated by James Turnbull over 6 years ago

  • Target version changed from 0.25.4 to 0.25.5

#22 Updated by James Turnbull about 6 years ago

  • Target version changed from 0.25.5 to 49

#23 Updated by Kevin Cai about 6 years ago

This issue is much more critical!

It can be reproducible in a period running of 30+ days, I have about 20+ machines running puppetd (version 0.25.4), after 30 days, 80% of their puppetd’s memory usage are up to 50%

restarting puppetd process can resolve the issue.

puppetd memory usage (the host has 1GB RAM & 1GB swap)

ELAPSED %CPU %MEM CMD

33-20:01:14 3.4 60.7 /usr/bin/ruby /usr/sbin/puppetd

#24 Updated by James Turnbull about 6 years ago

Kevin

  1. What platform?
  2. What Ruby version?

#25 Updated by James Turnbull over 5 years ago

  • Target version deleted (49)

#26 Updated by Anonymous over 4 years ago

  • Assignee deleted (Markus Roberts)

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

#27 Updated by Anonymous over 4 years ago

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

#28 Updated by Steve Shipway over 4 years ago

I am getting excessive memory use on a puppet agent: Puppet v2.7.6 It seems that, if I use the File resource with recursive enabled (even if I explicitly limit the recursion) then it slowly eats up more and more memory. This only affects a couple of our puppet clients, and they are the only ones to be using the file recurse attribute.

Initial use is about 39M resident (131M total). However after a couple of weeks this reaches 1.2GB resident for 1.4G total!

Could this be a memory leak of some kind related to this bug?

Steve

#29 Updated by Ben Hughes over 4 years ago

  • Description updated (diff)
  • Status changed from Accepted to Unreviewed

#30 Updated by Luke Kanies over 4 years ago

  • Description updated (diff)
  • Status changed from Unreviewed to Accepted

#31 Updated by Anonymous over 4 years ago

If anyone can reproduce this and give a simple recipe, I will bless their little cotton socks.

If you can’t, but want the fun and excitement of instrumenting your production systems, http://timetobleed.com/memprof-a-ruby-level-memory-profiler/ looks like a useful guide to tracking down what is consuming all that memory.

It would also be interesting if anyone experiencing this, and using the long running Puppet agent, could confirm that it doesn’t happen with a puppet agent --onetime run – that it is a leak, and not static memory consumption caused by … something.

Finally, if you could tell me the size of your state.yml file, that would be super-awesome.

#32 Updated by Sergio Ballestrero about 4 years ago

Running on Scientific Linux CERN 5.7, 64bit, puppet-2.6.12-2.el5, augeas-0.9.0-1.el5, ruby-1.8.5-22.el5_7.1.x86_64, we see the long running puppetd having a virtual memory footprint of ~130MB when it starts, in ~1 week growing to 300MB and continuing more slowly.

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
 3349 root      15   0  345m 252m 3280 S  0.0  8.4 193:21.19 puppetd 

On a puppet agent --onetime I see the same memory consumption of a freshly started long-running puppetd:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND 
22657 root      15   0  224m 131m 3176 S 30.6  4.4   0:15.72 puppet

/var/lib/puppet/state/state.yaml is 125832 bytes.

This is happening practically on all our clients, so it’s very reproducible for me. I will now try a completely empty node definition, on a test VM, to see the effect on memory. For later reference, this VM now has been up 30 days, and I see

 2655 root      15   0  550m 438m 4464 S  0.0 43.7   1285:42 puppetd

and a 58328 bytes state.yaml. Did not decrease after changing the config, with puppetd still running. After a restart, it’s

 3151 root      15   0  158m  63m 3120 S  0.0  6.4   0:02.93 puppetd

and still 58328 bytes state.yaml.

#33 Updated by Yuri Arabadji about 4 years ago

It’s leaking and leaking badly.

PID     ELAPSED    VSZ   RSZ S TTY          TIME COMMAND
3263 15-00:13:32 368684 229712 S ?       01:47:21 /usr/local/rvm/rubies/ruby-1.8.7-p334/bin/ruby /usr/local/rvm/gems/ruby-1.8.7-p334/bin/puppetd

#34 Updated by Sergio Ballestrero about 4 years ago

On the “empty” config (mentioned in my previous entry) it’s leaking about 1MB per day – see attachment.

#35 Updated by Nigel Kersten about 4 years ago

  • Status changed from Accepted to Needs More Information
  • Assignee set to Anonymous

Daniel, do you have enough info here to reproduce?

#36 Updated by Anonymous about 4 years ago

Nigel Kersten wrote:

Daniel, do you have enough info here to reproduce?

I still can’t seem to reproduce this, no. I think it is going to take exactly reproducing the versions that folks have mentioned to try and track it down further. Generally, though, it feels like this might be more normal Ruby process growth over time … but not entirely clear.

#37 Updated by Mariusz Gronczewski about 4 years ago

For long running process (well, few hours, its restarted every night):

[13:36:21]:~ᛯ ps aux |grep pup
root     10610  4.7 69.4 1103676 728392 ?      DNsl 05:00  24:37 /usr/bin/ruby /usr/sbin/puppetd --report --verbose --ignorecache
[13:36:59]:~ᛯ ls -s -h /var/lib/puppet/state/state.yaml
1.4M /var/lib/puppet/state/state.yaml

on some machines (same node config) its >800m, would be more but we restart it nightly and even then sometimes puppet is killed by oomkiller

same catalog ran “manually” takes about 400m, most of state.yaml are files (few big dirs with recurse => true)

we’re running 2.7.9

#38 Updated by Oliver Hookins about 4 years ago

I found that just as of the last few days our Puppetmasters are consuming an additional 6GB of memory over about an 8 hour period, causing either httpd or ruby to be killed of by OOM killer. What is not clear to me is how to profile the running application.

If someone can give me some pointers on profiling for memory usage I would be able to quite easily set up a test installation and fire huge amounts of requests at it until it leaks enough memory to give some results.

We’re currently on Puppet 2.6.7, running through Passenger 2.2.7.

#39 Updated by Jelmer Jaarsma almost 4 years ago

we upgraded from puppet 2.6.7 to 2.7.14 on the 17th of May, since the 24th I’ve been getting Nagios warnings that memory usage is exceeding 400mb on my rhel5 / centos5 machines, I never had those before. state.yaml is 78k

#40 Updated by Khushil Dep over 3 years ago

  • Target version set to 2.7.x

So I see the same on puppet on client machines which have been alive for a long time – memory just seems to grow…

http://pastebin.com/mXfUhDNa

#41 Updated by Anonymous over 3 years ago

  • Target version deleted (2.7.x)

#42 Updated by Anonymous almost 3 years ago

  • Assignee deleted (Anonymous)

Also available in: Atom PDF