The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #2888

puppetd doesn't always cleanup lockfile properly

Added by Peter Meier over 4 years ago. Updated 8 months ago.

Status:Needs More InformationStart date:12/04/2009
Priority:NormalDue date:
Assignee:Colin Alston% Done:

0%

Category:plumbing
Target version:3.x
Affected Puppet version:3.3.0 Branch:https://github.com/puppetlabs/puppet/pull/1158
Keywords: customer

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1070


Description

ok I had the patch #2661 now running for some weeks and I had nearly no problems anymore. However from time to time (maybe once,twice a week) a random client doesn’t remove its lockfile (@/var/lib/puppet/state/puppetdlock@), hence future runs fail. I assume this might still happen due to a uncatched exception (as in #2261), however the problem is a) hard or nearly impossible to reproduce and b) it occurs really by random. The only thing I can see in the logs:

Nov 30 19:27:41 foobar puppetd[26228]: Finished catalog run in 98.79 seconds
Nov 30 20:00:02 foobar puppetd[3000]: Could not retrieve catalog from remote server: Error 502 on SERVER: ^M 502 Bad Gateway^M ^M 

502 Bad Gateway

^M
nginx/0.6.39
^M ^M ^M Nov 30 20:00:03 foobar puppetd[3000]: Using cached catalog Nov 30 20:00:03 foobar puppetd[3000]: Could not retrieve catalog; skipping run Nov 30 20:00:04 foobar puppetd[12169]: Run of Puppet configuration client already in progress; skipping Nov 30 20:30:04 foobar puppetd[21230]: Run of Puppet configuration client already in progress; skipping

as I run puppetd by cron twice an hour with —splay I assume that the run between 19:30 and 20:00 got delayed till 20:00. At this time (20:00) a puppetmaster restart happens and due to that the 502 occured. This was the run of pid 3000, the next run (pid 12169) failed, this could either be as pid 3000 was still running or because there was already no puppetd anymore running and the lock file haven’t been removed. However every future run failed as well as the lockfile wasn’t removed.

So somehow puppet doesn’t remove lockfiles properly under certain conditions.

PS: If you think it’s better to reopen the old bugreport, close this one and duplicate and re-open #2261


Related issues

Related to Puppet - Bug #5139: puppetdlock file can be empty Needs More Information 10/28/2010
Related to Puppet - Bug #10418: Puppet agent hangs when listen is true and reading from /... Closed 11/01/2011
Related to Puppet - Bug #12185: Puppet agents get stuck Closed 01/26/2012
Related to Puppet - Feature #12741: Add a hook for dumping thread state / backtrace of a runn... Accepted 02/20/2012
Related to Puppet - Bug #2661: puppetd exits if the master is unreachable. Closed 09/20/2009
Related to Puppet - Bug #504: Puppetd lock files seem to be staying around Closed
Related to Puppet - Bug #1254: puppetd client randomly hangs Closed
Related to Puppet - Feature #190: Clear lockfile if the associated PID no longer exists Closed
Related to Puppet - Bug #4153: notice: Run of Puppet configuration client already in pro... Rejected 07/07/2010
Related to Puppet - Bug #5246: Puppetd does not remove it's pidfile when it exits Closed 11/09/2010
Related to Puppet - Feature #3757: --enable and --disable should be improved Closed 05/12/2010
Related to Puppet - Feature #4836: puppetd --disable improvements Closed 09/23/2010
Related to Puppet - Bug #17454: Puppet agent deadlocks when sent two HUP signals at almos... Investigating
Related to MCollective Plugins - Bug #15472: rewrite puppet agent Investigating 03/28/2011
Related to Puppet - Bug #16491: Create a standalone API to manage the Puppet lock files, ... Investigating 09/19/2012

History

#1 Updated by James Turnbull over 4 years ago

  • Category set to executables
  • Status changed from Unreviewed to Investigating
  • Assignee set to Markus Roberts

#2 Updated by Markus Roberts over 4 years ago

  • Assignee changed from Markus Roberts to Jesse Wolfe
  • Target version set to 0.25.3

This may be related to the test isolation ticket.

#3 Updated by Markus Roberts over 4 years ago

  • Target version changed from 0.25.3 to 0.25.4

#4 Updated by James Turnbull over 4 years ago

  • Target version changed from 0.25.4 to 0.25.5

#5 Updated by James Turnbull over 4 years ago

  • Target version changed from 0.25.5 to 49

#6 Updated by eric sorenson almost 4 years ago

I see this a fair bit, but my hosts crash more than most I think.

Ideally both the $statedir/puppetdlock and $rundir/puppetd.pid test-and-exit conditions would add a sanity check to see if there actually are any other puppetd pids running, and, if not, conclude they died unnatural deaths and ought to be overriden by the current instance.

#7 Updated by Jesse Wolfe almost 4 years ago

  • Category changed from executables to plumbing
  • Status changed from Investigating to Accepted
  • Assignee deleted (Jesse Wolfe)
  • Target version changed from 49 to 52

#8 Updated by James Turnbull over 3 years ago

  • Target version deleted (52)

#9 Updated by Jason Antman over 2 years ago

Is there any update on this? I’m running a 2.6.12 client and just saw this after an otherwise successful run – report to dashboard and all.

#10 Updated by Jason Antman over 2 years ago

Right now, 7 of my 28 puppet clients (2.6.12, from EPEL) have this issue. Most of them haven’t run in 7 days, even though they’re set with runinterval=1800 and splay=true. The puppetd process appears to be hung in uninterruptible sleep (“Ssl”). The last log message I see on the client (from 7 days ago) is a finished catalog run (success, no errors, no changes). /var/lib/puppet/state/puppetdlock exists and contains the correct PID, but has a mtime of 7 days ago, 30 minutes after the last finished run (or, when the next run should have been).

If I try a “puppet kick” from the master, all of a sudden the client “wakes up” and outputs partial run output, and then a finished (successful) run in 3.72 seconds, but only with part of the changes that should have happened… almost like it froze during a run 7 days ago, and then after the kick, just picked up where it thinks it left off – complete with a report sent to Dashboard showing a 3 second run, 1 changed resource, and success.

I still have a client in this hung state. I don’t know much about ruby, but please advise if there’s any debugging I can do to assist in this.

Also, the clients in question are CentOS 6.2 running 2.6.32-220 kernel, x86_64, with the EPEL puppet-2.6.12-1 package and ruby 1.8.7.352.

#11 Updated by Chris Price over 2 years ago

Jason,

In the original report, it looks like the client was continuing to log “Run of Puppet configuration client already in progress” at the expected intervals. Just to make sure I understand correctly, you are not seeing this sort of message, correct?

If not, then it seems possible that your problem may not have to do with the lock files not being removed; we might want to create a new ticket.

I’m poking around to see if I can find any way that we might be able to get some useful information out of the client that you still have in the hung state.

#12 Updated by Jason Antman over 2 years ago

Chris Price wrote:

Jason,

In the original report, it looks like the client was continuing to log “Run of Puppet configuration client already in progress” at the expected intervals. Just to make sure I understand correctly, you are not seeing this sort of message, correct?

If not, then it seems possible that your problem may not have to do with the lock files not being removed; we might want to create a new ticket.

Chris, the original reporter said he was running puppet via cron, so it sounds like he’s talking about one puppet process leaving the lockfile and the next process seeing the stale lockfile with the old PID, and giving that message. I, on the other hand, run puppetd as a daemon, so I never get that message. It just sits there. And sits… and sits… and sits.

I’m open to doing absolutely any debugging you want. Next week, time allowing, I’m going to roll a few test VMs of identical configuration, run them with all debugging enabled, and hope to reproduce the problem. In the mean time, if there’s any way I can do the ruby equivalent of a memory/heap dump, if that would help, I’ll try it.

Also, given the behavior I explained, I’m inclined to think that puppetd was in the process of applying change(s) when it hung.

#13 Updated by Chris Price over 2 years ago

Thanks, I missed that point about the original report being run via cron.

I agree with your assessment about it sounding like it was in the middle of something when it hung—which means I am still suspicious about whether your issue has to do with the lockfiles or not. Checking into a few more things on this end, will post any relevant information I come up with.

#14 Updated by Chris Price over 2 years ago

I haven’t had any luck finding any info about doing a thread dump on a running ruby process without planning for it ahead of time, but if you are able to repro on your new test VMs, this looks like it could be very helpful:

http://pivotallabs.com/users/steve/blog/articles/746-inspect-running-ruby-processes-using-xray-and-kill-3

You should be able to add the “require ‘xray/thread_dump_signal_handler’” line at the beginning of the puppetd script.

#15 Updated by Chris Price over 2 years ago

There are a few commits that came in in version 2.7.4 (2ab563408159c06f897cae38addb372e1f1ad63f, f7e526b86a015a63995a3300a3d438f3f3b4272f), and a commit that came in in version 2.7.10 (b434e3b9c31de63365e1a0f53d152be7e6e988ec) that deal with lock files. On the surface, they appear to mostly be cleanup and handling Windows-specific issues, so I don’t know that they would have any effect on the behavior reported here. Still, if you are able to come up with a reliable repro scenario it would be interesting to see if it behaves any differently with 2.7.x.

The code that creates the lock file (Puppet::Agent::Locker, lib/puppet/agent/locker.rb:13) uses an “ensure” block to remove the lock file, so if an exception was occurring during the agent run it should not prevent the file from being removed. This seems like another data point indicating that there is something hanging or getting into an infinite loop during the agent run, and the failure to remove the lock file seems like it’s just a side effect rather than the root of the problem.

#16 Updated by Jason Antman over 2 years ago

I’m pretty sure the problem I was describing is better tracked in Issue #12185, which seems to more accurately explain the behavior, and more or less the same fix that I’m using: puppetd seems to actually be stuck in a socket read for the REST API, and sending any data at all to port 8139 will get puppet to resume.

Re: your comments above, essentially the feature you mention in Issue #12741, I’m giving it a try now and will post if I have any useful results.

#17 Updated by Jo Rhett about 2 years ago

This problem is not solved, and is happening actively with 2.7.14 on centos5 and centos6 systems in our environment.

[root@qa1-web02 home]# date
Fri Jun  8 18:44:40 UTC 2012
[root@qa1-web02 home]# cat /var/lib/puppet/state/puppetdlock 
30747
[root@qa1-web02 home]# ps auwx |grep 30747
root     15605  0.0  0.0  61212   776 pts/1    S+   18:44   0:00 grep 30747
root     30747  0.0  0.0 129268  1976 ?        Ss   Jun07   0:00 /usr/bin/ruby /usr/sbin/puppetd
[root@qa1-web02 home]# grep 30747 /var/log/messages
Jun  7 18:34:31 qa1-web02 puppet-agent[30747]: Reopening log files
Jun  7 18:34:31 qa1-web02 puppet-agent[30747]: Starting Puppet client version 2.7.14
Jun  7 18:34:31 qa1-web02 puppet-agent[30747]: Run of Puppet configuration client already in progress; skipping

I had thought we had fixed this some versions ago, but it’s happening here with vengeance. (new 2.7.14 setup so I have no history for when it broke again)

#18 Updated by Jo Rhett about 2 years ago

For reference purposes, we have about 60 nodes puppetized and it has happened to about 30 of them within the last 3 days.

#19 Updated by Chris Price about 2 years ago

  • Assignee set to Anonymous

Daniel, just wanted to make sure you were aware that this issue has some new comments.

#20 Updated by Anita Krueger about 2 years ago

Just wanted to note that we are seeing the exact same behavior with about 50% of our clients. Puppet is at 2.7.14 on RHEL 6 with selinux enabled on the master and clients. There are clients with RHEL 5 and they exhibit the same things.

What I found is that once the agent is kicked from the master, it returns to the normal runinterval until it gets stale again.

#21 Updated by Chris Price about 2 years ago

  • Assignee changed from Anonymous to Andrew Parker

#22 Updated by Chris Price about 2 years ago

Andy/Patrick; perhaps we should allocate some hardware to leave some long-running agent daemon processes running to try to help repro this?

#23 Updated by Jo Rhett about 2 years ago

Hm. I thought I posted a long update here yesterday, but it seems to be getting lost. Here’s some state information from a locked node:

  1. puppetdlock file is usually mtime 30 minutes later than the previous run
  2. Nothing was logged in /var/log/messages
  3. The previous run was aborted due to puppet already running (likely ‘puppet agent —test’ by hand)
[root@q1-m1 ~]# sudo puppet agent --test
notice: Ignoring --listen on onetime run
notice: Run of Puppet configuration client already in progress; skipping
[root@q1-m1 ~]# ls -la /var/lib/puppet/state
total 160
drwxr-xr-t  3 root   root     4096 Jun 18 21:13 .
drwxr-xr-x 10 puppet puppet   4096 Jun 18 17:11 ..
drwxr-xr-x  2 root   root     4096 Jun 18 17:09 graphs
-rw-r--r--  1 root   root   105644 Jun 18 20:43 last_run_report.yaml
-rw-r--r--  1 root   root      607 Jun 18 20:43 last_run_summary.yaml
-rw-r--r--  1 root   root        4 Jun 18 21:13 puppetdlock
-rw-r--r--  1 root   root     3318 Jun 18 20:43 resources.txt
-rw-rw----  1 root   root    26840 Jun 18 20:43 state.yaml
[root@q1-m1 ~]# date
Tue Jun 19 17:33:17 UTC 2012
[root@q1-m1 ~]# cat /var/lib/puppet/date
cat: /var/lib/puppet/date: No such file or directory
[root@q1-m1 ~]# cat /var/lib/puppet/state/puppetdlock 
5602[root@q1-m1 ~]# grep 5602 /var/log/messages
Jun 18 20:43:11 q1-m1 puppet-agent[5602]: Reopening log files
Jun 18 20:43:11 q1-m1 puppet-agent[5602]: Starting Puppet client version 2.7.14
Jun 18 20:43:11 q1-m1 puppet-agent[5602]: Run of Puppet configuration client already in progress; skipping
[root@q1-m1 ~]# 

Thus logic here says that your replication scenario is to run “puppet agent —test” just seconds before a normal puppet run to recreate the exact same scenario.

#24 Updated by Chris Price about 2 years ago

I am curious if anyone who is experiencing this issue might be able to repro it with ruby 1.9? If so, the “require ‘xray/thread_dump_signal_handler’” that I mentioned above could give us some backtraces that might be invaluable in attempting to debug this.

#25 Updated by Jo Rhett about 2 years ago

I don’t have Ruby 1.9, but it appears that having puppet daemon started by “puppet agent —test” is a 100% reproducible recipe for lockup. It locks puppetdlock on the first run attempt (ie 30 minutes) after it was started.

#26 Updated by Chris Price about 2 years ago

Wow, that’s very interesting because “—test” implicitly includes the no-daemon option. I don’t suppose there’s any way that we can get a VM snapshot or similar to help repro?

#27 Updated by Jo Rhett about 2 years ago

I don’t follow your logic. If you stop puppetd, and then run with —test a catalog that says to ensure puppet is running, the net result is always that puppetd is restarted. Were you expecting differently?

When puppetd first starts up, it immediately tried to do a run. It finds that the “—test” run is running, logs the message and backs off. The test run finishes and puppetdlock is removed. However for some reason 30 minutes later it creates puppetdlock, then goes into this lock state. Tracing the path of transactions involved here really shouldn’t be that hard.

#28 Updated by Jo Rhett about 2 years ago

FWIW, I’m replicating this on CentOS 5.8 and 6.2 systems, minimum installation and nothing more installed than is required to get puppet working. That is exactly

Installed:
  puppet.noarch 0:2.7.16-1.el5                                                                                                      

Dependency Installed:
  augeas-libs.x86_64 0:0.10.0-3.el5  facter.x86_64 0:1.6.10-1.el5       ruby.x86_64 0:1.8.7.358-5  ruby-augeas.x86_64 0:0.4.1-1.el5 
  ruby-libs.x86_64 0:1.8.7.358-5     ruby-shadow.x86_64 0:1.4.1-7.el5 

#29 Updated by Chris Price about 2 years ago

  • Target version set to 3.0.0

#30 Updated by Jo Rhett about 2 years ago

I’d like to protest that target version. This is affecting production 2.7.16 systems quite dramatically. I’m finding more and more systems that aren’t checking in. This is a production affecting bug — upgrading to 3.0 really isn’t a fair request to make of sites.

#31 Updated by Chris Price about 2 years ago

Apologies Jo, miscommunication on my part. Setting that as the target version doesn’t preclude this being fixed for a 2.7.x release as well; the reason that I set it for 3.0.0 is because we are in RC for that release right now, and so any ticket flagged that way will get more attention over the next few weeks than if it was set to any other target version. So this was basically just an extra effort to make sure that someone looks at this sooner rather than later.

#32 Updated by Jo Rhett about 2 years ago

Just a comment that I realized that a small subset of machines were running a different kernel, for reasons which would get a person talked to if they hadn’t already been fired for other reasons :( Those machines were running the very same known bad proc-handling kernel that I had reported and helped resolve late last year. Those have been fixed, and this issue is happening only randomly across many systems.

Running “puppet agent —test” is absolutely at the core of the issue however, since the machines that we don’t run it on don’t have the problem.

#33 Updated by Jeff McCune almost 2 years ago

  • Assignee changed from Andrew Parker to Jeff McCune

Working

I’m starting to work on this issue.

#34 Updated by Jeff McCune almost 2 years ago

I let ~10 nodes with Puppet 2.7.12 run in a 30 second loop in Daemonize mode.

I was unable to reproduce the issue when Puppet is only running on any of the nodes in daemon mode. Jo Rhett points out that puppet agent --test is at the core of the issue, so I’m going to leave this test group of 20 nodes and 1 puppet master running over night. I’m going to configure puppet agent --test to run periodically on half the nodes to see if I can reproduce the issue.

-Jeff

#35 Updated by Jeff McCune almost 2 years ago

Able to reproduce

Well, good news and bad news. I’m able to reproduce the issue, but I haven’t yet gotten Puppet into a harness where I can poke and prod it.

Here’s how I reproduced the issue relatively quickly in the end:

  • 21 Ubuntu 12.04 x86_64 nodes
  • PE 2.5.3 on all nodes
  • Puppet 2.7.12
  • Daemon mode configured on all agents. (/etc/init.d/pe-puppet-agent enabled and running)
  • runinterval = 30 on all agent nodes

So I have a four-core master and 20 nodes with persistent daemonized agent processes checking in with a 30 second runinterval setting.

I let this run for 3 days and the issue never arose.

I then configured 5 nodes to run puppet agent —test in a loop while the daemon was running. The problem came up on 4 out of 5 nodes in a matter of hours:

#! /bin/bash
# ~/test_2888.sh
export FACTER_INTERACTIVERUN=true
export FACTER_INTERACTIVERUNDATE=$(date)
export FACTER_INTERACTIVERUNSTAMP=$(date +%s)
/opt/puppet/bin/puppet agent --test

And the loop I ran in tmux on 5 of the node:

while sleep 5; do sudo ~jeff/test_2888.sh; done

The lock file protecting configuration runs became stale quite quickly:

[jeff@agent11] ~ 
$ ls -l /var/opt/lib/pe-puppet/state/puppetdlock
-rw-r--r-- 1 root root 0 Sep 10 22:12 /var/opt/lib/pe-puppet/state/puppetdlock
[jeff@agent11] ~ 
$ date
Tue Sep 11 00:06:03 GMT 2012

#36 Updated by Jeff McCune almost 2 years ago

A bit more information after diving into this.

First, complicated multi-node scenarios aren’t required to reproduce the issue. It was sufficient for me to start three different processes on my laptop. A webrick master, a daemonized agent with a 1 second runinterval, and BASH running a tight while loop with puppet agent —test inside the loop.

With these three processes running, I observed the file referenced by puppet agent --configprint puppetdlockfile in 2.7.x coming into and out of existence. Sometimes it contained the PID of the interactive agent. Sometimes it contained the PID of the daemon process. When it contained a PID the system worked as I expected it to. Only one of the two scheduled agent processes performed a configuration run.

It only took a few minutes for this scenario to run into the issue. When I observed the issue of no process performing a configuration run, I also observed the file referenced by puppet agent --configprint puppetdlockfile contained no PID and was zero length. 2.7.x currently reports this as Skipping run of Puppet configuration client; administratively disabled; use 'puppet agent --enable' to re-enable.

I’m going to proceed by tracking down whatever is creating the zero length file. I suspect changing to ensure the PID is written will fix the specific issue.

#37 Updated by Josh Cooper almost 2 years ago

Jeff McCune wrote:

I’m going to proceed by tracking down whatever is creating the zero length file.

Probably not related to your setup, but MCollective’s puppetd agent creates a zero-length puppetdlock file when disabling puppet.

#38 Updated by Andrew Parker almost 2 years ago

Josh and I were looking through the 3.x code today and came across what looked like possible races in handling the pid lock file. This code has been reworked quite a bit since 2.7.x, but those problems may still exist.

#39 Updated by Jeff McCune almost 2 years ago

Thanks to Josh figuring out that the Settings catalog is causing a managed file resource to create the puppetdlock file as an empty file. The resource gets added to the settings catalog here:

https://github.com/puppetlabs/puppet/blob/2.7.x/lib/puppet/util/settings.rb#L558

We can pry open the runtime and look around with something like this:

diff --git a/lib/puppet/util/settings.rb b/lib/puppet/util/settings.rb
index b0614e5..9695c51 100644
--- a/lib/puppet/util/settings.rb
+++ b/lib/puppet/util/settings.rb
@@ -553,13 +553,23 @@ class Puppet::Util::Settings
     @config.values.find_all { |value| value.is_a?(FileSetting) }.each do |file|
       next unless (sections.nil? or sections.include?(file.section))
       next unless resource = file.to_resource
+      # XXX We _could_ special-case the puppetdlock file at this point.  This
+      # seems like a bad idea.  Better, I think to enable FileSetting instances
+      # to opt out of the Settings catalog and honor that preference.
       next if catalog.resource(resource.ref)
 
+      # XXX This appears to be where the puppetdlock file gets added to the
+      # catalog.  The catalog creates a zero byte file when it manages the file
+      # resource.
+      binding.pry if resource.to_s =~ /puppetdlock/
       catalog.add_resource(resource)
     end
 
     add_user_resources(catalog, sections)
 
+    # XXX We could take this opportunity to wrap the catalog.apply method.
+    # This would allow us to figure out exactly when the Settings catalog gets
+    # applied.
     catalog
   end
 
-- 

#40 Updated by Jeff McCune almost 2 years ago

  • Status changed from Accepted to In Topic Branch Pending Review
  • Branch set to https://github.com/puppetlabs/puppet/pull/1158

Pull request for 2.7:

https://github.com/puppetlabs/puppet/pull/1158

This isn’t going to merge cleanly up into 3.x given the move from puppetdlockfile to agent_pidfile + agent_disabled_lockfile. I’m going to work on the 3.x pull request.

This ticket should not be closed until both 2.7.20 and 3.0.0 are released with these patches.

#41 Updated by Jeff McCune almost 2 years ago

  • Status changed from In Topic Branch Pending Review to Merged - Pending Release

This fix should be included in Puppet 2.7.20 and 3.0.0.

-Jeff

#42 Updated by Josh Cooper almost 2 years ago

These are my notes of other issues that we undercovered, but were not required to fix the deadlock.

  1. In Puppet::Agent#run, we check if we’re running? before acquiring a lock, so two agents can race and run in parallel unaware of the other.
  2. In Puppet::Util::Pidlock#lock and #unlock, we frequently check if the lockfile exists and if so read from it. But this is not atomic and can cause races. In cases, where we just want to read the file, we should call File.read, but be prepared to handle Errno::ENOENT.
  3. In cases, like lock, where we need to read the file, check if the pid still exists, and if not, overwrite it, then we should be using an advisory lock (see lib/puppet/external/lock.rb) to ensure the test & set is atomic with respect to other puppet processes.
  4. In addition, when writing to the pidfile, we don’t use replace_file, so partial reads, including an empty file, are possible.
  5. In Puppet::Util::Pidlock#clear_if_stale, we call lock_pid twice, which reads the lockfile twice unnecessarily.
  6. In addition, if the lockfile is empty, then we end up calling "".to_i, which returns 0, so then we call Process.kill(0, 0), which will actually succeed, even though there’s no pid 0. As a result, we may think the pidfile is still owned, when really it means we should be disabled.
  7. In the same method, Process.kill(0, pid) will raise EPERM if the process exists, but we don’t have permission to signal it. Normally this isn’t a problem for the agent since it’s running as root.

#43 Updated by Matthaus Owens almost 2 years ago

  • Target version changed from 3.0.0 to 2.7.20

Released in Puppet 3.0.0-rc7. Release pending 2.7.x.

#44 Updated by Matthaus Owens over 1 year ago

  • Status changed from Merged - Pending Release to Closed

Released in Puppet 2.7.20

#45 Updated by Charlie Sharpsteen over 1 year ago

  • Keywords set to customer

#47 Updated by Colin Alston 9 months ago

  • Status changed from Closed to Re-opened
  • Target version changed from 2.7.20 to 3.x
  • Support Urls deleted (https://support.puppetlabs.com/tickets/724)
  • Affected Puppet version changed from 0.25.1 to 3.3.0

This appears to have regressed in 3.3.0-1

root@qa:~# puppet agent --onetime --no-daemonize --ignorecache --logdest console
Notice: Run of Puppet configuration client already in progress; skipping  (/var/lib/puppet/state/agent_catalog_run.lock exists)
root@qa:~# ps -ef | grep puppet
root     20292 19918  0 09:33 pts/0    00:00:00 grep --color=auto puppet
root@qa:~# stat /var/lib/puppet/state/agent_catalog_run.lock
  File: `/var/lib/puppet/state/agent_catalog_run.lock'
  Size: 0               Blocks: 0          IO Block: 4096   regular empty file
Device: ca01h/51713d    Inode: 265302      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2013-11-07 20:21:01.623531262 +0200
Modify: 2013-10-11 18:49:01.755046969 +0200
Change: 2013-10-11 18:49:01.755046969 +0200
 Birth: -
root@qa:~# aptitude show puppet
Package: puppet
State: installed
Automatically installed: no
Version: 3.3.0-1puppetlabs1

#48 Updated by eric sorenson 9 months ago

  • Support Urls deleted (https://support.puppetlabs.com/tickets/724)

#50 Updated by eric sorenson 9 months ago

  • Status changed from Re-opened to Needs More Information
  • Assignee changed from Jeff McCune to Colin Alston

Colin — i’m unable to reproduce this. Do your logs indicate any abnormal exit before the run which left the lock file lying around? Are you seeing this systemically across your population on 3.3.0?

I’ve put this ticket’s status into “Needs more Information” and assigned it to you. Please either (a) update it with the information I’ve requested and re-assign it to me if you need more help, or (b) change the status to “Closed” if you were able to resolve the issue on your own.

#51 Updated by Jason Antman 8 months ago

Redmine Issue #2888 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1070

Also available in: Atom PDF