The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #7273

Add additional signals for restarting Puppet agent runs

Added by Randall Hansen almost 3 years ago. Updated 4 months ago.

Status:Needs More InformationStart date:04/28/2011
Priority:ImmediateDue date:
Assignee:eric sorenson% Done:

0%

Category:-
Target version:3.x
Affected Puppet version: Branch:
Keywords:backlog

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1152


Description

Sometimes it’s desirable to be able to abort the current Puppet agent run from within a manifest if certain conditions hold true.

Sometimes this should be result in a report, sometimes not.

Proposed functionality:

Add signal support that:

  1. aborts the current run, doesn’t send reports or do anything, starts again, reparsing client configs
  2. aborts the current run, sends a report with all the details and starts again, reparsing client configs.

Note that there is no mention of whether this is considered an error or not. Ideally we would find some other mechanism for triggering an error on the client, but it may be desirable to add additional signals to (1) and (2) above to allow specification of an error state.

Feedback welcome, particularly on the last point.


Related issues

Related to Puppet - Bug #5153: Ability to run --onetime in the background while a daemon... Accepted 10/29/2010
Related to Puppet - Bug #19153: service puppet ensure stopped kills off cron-run puppet w... Accepted
Duplicated by Puppet - Bug #1273: puppetd ignores SIGHUP during catalog run Duplicate
Duplicated by Puppet - Feature #19218: Would love a clean way to tell puppet agent to restart it... Duplicate

History

#1 Updated by Ryan Conway almost 3 years ago

Hey Randall,

So this is our issue. We have Puppet managing its own configuration file, for both the master and the agent. We define the service and subscribe to the relevant files like so:

service { "puppet":
    enable  => "true",
    ensure  => running,
    subscribe => [ File["/etc/init.d/puppet"], File["/etc/default/puppet"], File["/etc/puppet/puppet.conf"] ],
}

If any of those files are changed, when the agent next runs, the following is logged to /var/log/daemon.log:

May 18 17:48:27 puppetmaster puppet-agent[20088]: Starting Puppet client version 2.6.8
May 18 17:48:37 puppetmaster puppet-master[32372]: Compiled catalog for puppetmaster in environment production in 0.80 seconds
May 18 17:48:41 puppetmaster puppet-agent[20088]: (/Stage[main]/Puppetmaster/File[/etc/puppet/puppet.conf]/content) content changed '{md5}bba3dcc72ce50aece08416b62db4dda3' to '{md5}7ace02c1b78ba7e9e5bfaf95e197d179'
May 18 17:48:41 puppetmaster puppet-agent[20088]: Caught TERM; calling stop

The agent daemon has now stopped running. This example is from the agent running on our master, but the same thing happens on any node:

May 18 17:52:16 ryanpuppettest puppet-agent[4404]: (/Stage[main]/Puppetslave/File[/etc/puppet/puppet.conf]/content) content changed '{md5}ec16838eb75d99c156fc7a124965d9ed' to '{md5}e3eb04c7c07bf99d0f70d5f16d241dfa'
May 18 17:52:25 ryanpuppettest puppet-agent[4404]: Caught TERM; calling stop

If you need any more information, let me know! Ryan

#2 Updated by Ryan Conway almost 3 years ago

PS

When the puppet.conf changes on the master – as well as the agent failing as described, we also begin to suffer from the bug in #4487 / #5434.

#3 Updated by Ben Hughes almost 3 years ago

  • Status changed from Unreviewed to Investigating

This is because, I think (can confirm by running with debug), as it runs invoke-rc.d puppet stop, and then would run invoke-rc.d puppet start… but it can’t, as you’ve killed puppet from within puppet.

The way Nan suggests doing this is to make a cron job to start/restart puppet if say a file is present.

I am not 100% sure what would happen if you added “hasrestart => true” to it, either. I imagine it would still fail. Would need to test it.

#4 Updated by Nigel Kersten almost 3 years ago

Are you finding that you actually need to subscribe to the puppet config files?

I tend to not do this, and rely upon Puppet re-parsing the puppet configuration itself at the start of the next run.

#5 Updated by Ryan Conway almost 3 years ago

Hey,

I think the issue we have with that is we can’t wait an hour for the next run in order for the changed settings to take effect and any remaining settings be applied, as we’re using Puppet for servers that autoscale on EC2 as well. So they need to run their whole config successfully in one run. We work around this problem at the moment, but would be nicer if Puppet could handle restarting itself when you notify the Puppet service if that is possible?

#6 Updated by Nigel Kersten almost 3 years ago

Can you give us a little more detail on the kinds of changes you’re pushing?

#7 Updated by Ryan Conway almost 3 years ago

Sure – the main changes involve setting up the modules, for example our custom Facts module – without this, our EC2 instances will lack specific facts they need in order to finish installing everything on their first run.

#8 Updated by Nigel Kersten almost 3 years ago

If you’ve got facts delivered via pluginsync, then any facts in the modulepath will take effect in the same run that they’re initially synced on.

Can you elaborate a little bit more about the “custom Facts module” ?

#9 Updated by Ryan Conway almost 3 years ago

Hey there,

Sorry for the long delay – I’ve been away.

So, if the facts in our modules load in the same run, then that is fine, however we do make changes to the puppet.conf that need to also take effect immediately, for example delivering reports. Currently, as we try to notify the Puppet service, the agent then terminates as described above, and we have to manually handle this in our bootstrap script. It would be much cleaner and intuitive if Puppet could intelligently handle restarting itself in this situation, or had an option to re-parse and immediately re-run the catalog.

What are your thoughts on that?

#10 Updated by Noah Cantor almost 3 years ago

Ben Hughes wrote:

This is because, I think (can confirm by running with debug), as it runs invoke-rc.d puppet stop, and then would run invoke-rc.d puppet start… but it can’t, as you’ve killed puppet from within puppet.

The way Nan suggests doing this is to make a cron job to start/restart puppet if say a file is present.

I am not 100% sure what would happen if you added “hasrestart => true” to it, either. I imagine it would still fail. Would need to test it.

Nan’s solution sounds like a silly amount of overhead for a process which should just be able to reparse its config. Perhaps puppet should send itself a SIGHUP if the config file changes? After all, it runs in daemon mode all the time, anyway.

#11 Updated by Noah Cantor almost 3 years ago

I can think of several potential reasons to change puppet.conf and not want puppet to wait until the next run to pick up the information: A new report server; an updated run interval (current load is too high, or there’s some testing that needs to be done by the puppet agent); you want to move your puppetmaster from one URL to another (or IP).

I use puppet to manage my entire infrastructure. All systems, all services, all configurations, all packages, all files… everything that’s non-standard. So why is it that when I want to use puppet to manage itself (as seems incredibly intuitive), that it’s incapable of doing it?

I upgrade puppet via puppet. Sometimes that involves changes in puppet.conf. When I’m ready to roll out to the newest release, I tell my puppet installation that the new version is 2.7.1 (for example). When that gets rolled out, puppet dies. That’s completely counter-intuitive.

When I want to run tests on a handful of servers, I create the tests, run things manually, then add things to puppet and run puppet manually (via puppetd), then run things via puppet and the config file (/etc/init.d/puppet). In numerous cases, I have wanted to make rapid changes to systems. In order to properly test this via puppet, I want to change the frequency of system checks. Normally, they’re scheduled at 1/hr. When testing, I change them to 12/hr, in puppet.conf. Changing that on 15-20 servers, manually, would be a serious pain.

I think the main point is that Puppet can manage everything on a system — all files, packages, services, — except itself. Where is the logic in that? Should I install Chef to ensure that puppet gets restarted when the config changes?

#12 Updated by Ben Hughes almost 3 years ago

  • Status changed from Investigating to Needs Decision

You can change puppet.conf and it will re-read it, but it will only re-read it at the start of a run.

puppet agent --runinterval 120 --filetimeout 5 && sleep 25 && touch /etc/puppet/puppet.conf

Which gives

Jul  7 17:37:12 a4 puppet-agent[7596]: Finished catalog run in 11.22 seconds
Jul  7 17:37:22 a4 puppet-agent[7596]: Reparsing /etc/puppet/puppet.conf
Jul  7 17:39:45 a4 puppet-agent[7596]: Finished catalog run in 14.14 seconds

This is due to puppet agent sleeping during that period. I’m not sure if event-loop is capable of having a hook into it to monitor the config file. (not aided by event-loop’s website being down http://www.brockman.se/software/ruby-event-loop/).

One notions that it could be tunable of a re-run hook on the end of reparse in lib/puppet/settings.rb. We would need to be careful to not change current behaviour and be very careful to get stuck in an infinite loop of reparsing itself.

Something akin to:

diff --git i/lib/puppet/util/settings.rb w/lib/puppet/util/settings.rb
index f243b86..1f3a96c 100644
--- i/lib/puppet/util/settings.rb
+++ w/lib/puppet/util/settings.rb
@@ -415,6 +415,10 @@ class Puppet::Util::Settings
       Puppet.notice "Reparsing #{file.file}"
       parse
       reuse
+      if self[:runonconfigreparse] == true
+        # Thread safety goes here.
+        Puppet.run
+      end
     end
   end

Matt/Nigel, adding both of you as watchers as I’d like sign-off/comments from both of you. Thanks.

#13 Updated by Nigel Kersten almost 3 years ago

I’m torn here, but agree with others that there are good reasons to push out a new puppet config file that doesn’t take effect until the start of the next run as well as the need to have changes apply immediately.

A common solution here for people is to have a distinct set of manifests just for setting up Puppet itself, whether that is with an environment or tags. They then schedule Puppet to run once to bootstrap itself, and once for the real config.

What about sending a SIGHUP to the puppet agent process? Doesn’t that do what you need?

[edit] – Ultimately it sounds like we want to be able to tell Puppet, within a manifest, that the current run should be started again.

Would this send a report? Would failures matter at this point?

It’s not the most simple question in the world to answer.

#14 Updated by Nigel Kersten almost 3 years ago

  • Status changed from Needs Decision to Needs More Information
  • Assignee set to Nigel Kersten

#15 Updated by Daniel Pittman almost 3 years ago

On Fri, Jul 8, 2011 at 13:37, tickets@puppetlabs.com wrote:

I’m torn here, but agree with Noah that there are good reasons to push out a new puppet config file that doesn’t take effect until the start of the next run.

Actually, we have explicit direction that the Puppet core MUST be able to change the run mode dynamically, which implies that we are able to make sweeping changes to configuration on the fly. So, in the longer term any problems that would make this hard to achieve would need to be resolved anyhow.

Obviously, that doesn’t move our timeline around for doing it, but I think this is probably only a short-term issue.

(…and don’t get me wrong, I don’t think that this is a trivial thing to change; at the very least, logging and other dynamic data will have to be audited as part of this change to make sure they notice the dynamic updates and all!)

#16 Updated by Nigel Kersten almost 3 years ago

What if we just added two more signals?

a) abandons the current run, doesn’t send reports or do anything, starts again, reparsing client configs. b) stops the current run, sends a report with all the details and starts again, reparsing client configs.

#17 Updated by Daniel Pittman almost 3 years ago

What if we just added two more signals?

a) abandons the current run, doesn’t send reports or do anything, starts again, reparsing client configs. b) stops the current run, sends a report with all the details and starts again, reparsing client configs.

That seems, to me, to be another path to getting the desired behaviour. We could certainly use USR[12] for those purposes…

#18 Updated by Ryan Conway almost 3 years ago

That would be great yeah. So when Puppet runs for the first time, with a default config, would we be able to pass in a command-line argument to say which option to use?

#19 Updated by Nigel Kersten almost 3 years ago

Ryan Conway wrote:

That would be great yeah. So when Puppet runs for the first time, with a default config, would we be able to pass in a command-line argument to say which option to use?

No, it would be more that you’d send a signal to the agent/apply run from inside your manifests, and then the agent/apply run would respond appropriately.

so you’d have something like a bootstrap stage that set up your config file, and probably a notify to an exec that sent the signal, so if the config file changed, you respond appropriately.

does that make sense to everyone?

#20 Updated by Ryan Conway almost 3 years ago

Ah I see, that still feels suboptimal to me, in that we have to create a lot of extra code just to manage Puppet properly, when it should really (in my opinion) be capable of managing itself.

I really think we should be able to take the vanilla Puppet gem, point it at a master and change whatever configuration we like, all the while with Puppet handling notifies to restart itself properly.

#21 Updated by Nigel Kersten almost 3 years ago

Ryan Conway wrote:

Ah I see, that still feels suboptimal to me, in that we have to create a lot of extra code just to manage Puppet properly, when it should really (in my opinion) be capable of managing itself.

I really think we should be able to take the vanilla Puppet gem, point it at a master and change whatever configuration we like, all the while with Puppet handling notifies to restart itself properly.

The problem is that “properly” has multiple meanings. Sometimes it’s desirable to not apply settings until the next run, sometimes it’s desirable to abort the current run with no sense of success/failure, sometimes it’s desirable to abort with a failure status.

I’d definitely like to see a more ‘native’ solution, but feel that implementing signals is a good first step to getting there.

Particularly if we have a good signal type, this isn’t such a bad solution, imagine something like (completely off the top of my head):

signal { "puppetd":
  ensure => "USR1",
  refreshonly => "true",
  target => all,  # this could be first, last, all.
}

#22 Updated by Ryan Conway over 2 years ago

Yeah that looks like a good first step to me!

#23 Updated by Nigel Kersten over 2 years ago

  • Subject changed from Modifying puppet.conf causes 'reparsing config' and TERM signal results in shutdown of daemon to Add additional signals for restarting Puppet agent runs

(updated parent description)

#24 Updated by eric sorenson about 1 year ago

  • Assignee changed from Nigel Kersten to eric sorenson
  • Target version set to 3.x
  • Keywords set to backlog

Relating this to a bunch of other tickets around signalling running agents.

#25 Updated by eric sorenson about 1 year ago

  • Priority changed from Normal to High

#26 Updated by eric sorenson about 1 year ago

  • Priority changed from High to Immediate

#27 Updated by Anonymous 4 months ago

Redmine Issue #7273 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1152

Also available in: Atom PDF