Puppet

This isn’t trivial, but we’ll spend some time having looking into whether we can make this situation a bit better.

Underneath the two “starts” are doing different things, as a service with “hasrestart => true” shows:

info: /Stage[main]//File[/tmp/foobar]: Scheduling refresh of Service[apache2]
debug: Service[apache2](provider=debian): Executing '/etc/init.d/apache2 status'
debug: Service[apache2](provider=debian): Executing '/etc/init.d/apache2 start'
notice: /Stage[main]//Service[apache2]/ensure: ensure changed 'stopped' to 'running'
debug: Service[apache2](provider=debian): Executing '/etc/init.d/apache2 status'
debug: Service[apache2](provider=debian): Executing '/etc/init.d/apache2 restart'

See how one of these is invoking the start method to move from stopped => running, and the other is invoking the restart method due to receiving an event?

For some services quite different things happen in start compared to restart, and for many services you can’t restart them if they’re not running.

debug: Service[apache2](provider=debian): Executing '/etc/init.d/apache2 status'
debug: Service[apache2](provider=debian): Executing '/etc/init.d/apache2 status'
debug: /Stage[main]//Service[apache2]: Skipping restart; service is not running
notice: /Stage[main]//Service[apache2]: Triggered 'refresh' from 1 events

Hopefully that shows how this isn’t as simple a case as it might look.

I’m also seeing this (2.6.6 in this case). It’s causing problems as some init scripts (tomcat6 from jpackage, I’m looking at you) have race conditions which mean that starting then restarting the service in quick succession doesn’t work – by the time the “stop” process in the restart runs, the just-started daemon hasn’t started its management thread yet, and so the shutdown command fails.

I’m struggling to think of any software I know of which can’t be restarted if they’re not running, but it strikes me that the cleanest approach here (from a user PoV) would be to allow you to configure the service resource with something like a “canrestartifstopped” flag to represent that case.

Arguably, not having to wait for two lots of daemon startups would also allow puppet to run more quickly too :)

I’ve definitely run into init scripts that won’t “restart” when the service isn’t running, and I can see that argument, even if I think it’s kind of a pain, and ultimately we end up relying upon that infrastructure with our service providers.

It would appear that the service resource isn’t checked to see if its running until the notify is parsed, which is why the ordering happened as listed. You could enforce this ordering by adding “require => Service[‘nginx’]” to the file stanza, which would create

start service replace file restart service

What I think you’re asking for is “if the service hasn’t started yet, update the file first and then start the service”. I’m trying to think of a reason this couldn’t be the default behavior in case of a Notify to an unstarted service and I can’t think of one. If there are examples of when this would not be desirable then, then it would need explicit syntax. (not speaking for anyone besides myself)

Jo Rhett wrote:

What I think you’re asking for is “if the service hasn’t started yet, update the file first and then start the service”.

That is indeed the behavior that I desire out of puppet. As is, I keep having to create simple helper scripts (with bash, python, etc.) because puppet simply is not giving me enough control. Same goes for “require” which behaves more like the opposite of “before” than any sort of dependency control. That spreads the state-management logic around and generally makes a mess.

Nigel Kersten wrote:

I’m looking for more information on service providers where we want this behavior, as I’m pretty sure we’re not going to implement this for every provider.

I would want this for Fedora/RHEL, even if only available via setting some new parameter to the service type. Right now that would mean good old SysV init services. Fedora is going more and more to all native systemd init services, but so long as systemd keeps providing backwards compatibility through the ‘service’ and ‘chkconfig’ commands, I don’t think puppet has to make any special exceptions for that change.

Jo Rhett wrote:

It would appear that the service resource isn’t checked to see if its running until the notify is parsed, which is why the ordering happened as listed. You could enforce this ordering by adding “require => Service[‘nginx’]” to the file stanza, which would create

start service replace file restart service

This won’t work though as you end up creating a dependency loop as the service ends up depending on the file which depends on the service.

What I think you’re asking for is “if the service hasn’t started yet, update the file first and then start the service”.

“update the file first, start the service OR restart the service” instead of “update the file first, start the service and restart the service” which is what currently happens in the case it wasn’t running to begin with.

(I filed the duplicate #9656, this bites me hard as I have services which take in the order of minutes to start up, and I have to stop them before upgrading the packages so I always have to endure a double startup when the packages and config files change)

Personally I consider the init script buggy if restart fails when it wasn’t running, ditto for stop failing/exiting with !0 when it wasn’t running. Note that I consider “restart” and “reload” semantically different, restart to me is a just a shortcut for “stop; start” which at least for init scripts is how 99% of them implement the restart logic, while reload is the shortcut that just sends a signal or whatnot, and so requires the service to be running.

Having the same issue, please see code below. puppetmaster v2.7.14

## Error from puppet agent -t --debug
info: Websaas::Load_static_conf_files[tomcat-users.xml]: Scheduling refresh of Service[webservice-saas]
debug: Service[webservice-saas](provider=debian): Executing 'pgrep java'
debug: Service[webservice-saas](provider=debian): Executing '/etc/init.d/webservice-saas start'
notice: /Stage[main]/Websaas::Token_service/Service[webservice-saas]/ensure: ensure changed 'stopped' to 'running'
debug: /Stage[main]/Websaas::Token_service/Service[webservice-saas]: The container Class[Websaas::Token_service] will propagate my refresh event
debug: Service[webservice-saas](provider=debian): Executing 'pgrep java'
debug: Service[webservice-saas](provider=debian): Executing '/etc/init.d/webservice-saas stop'
err: /Stage[main]/Websaas::Token_service/Service[webservice-saas]: Failed to call refresh: Could not stop Service[webservice-saas]: Execution of '/etc/init.d/webservice-saas stop' returned 1:  at /etc/puppet/environments/passSplitNh/modules/Websaas/manifests/token_service.pp:115

Resource causing dual restart¶

websaas::load_static_conf_files { $static_conf_files:
    conf_dir    =>  "$webservice_config_path",
    notify      =>  Service['webservice-saas'],
    require     =>  File["${webservice_config_path}"],
}

service { 'webservice-saas':
    ensure      =>  running,
    enable      =>  true,
    status      =>  'pgrep java',
    provider    =>  debian,
    require     =>  Exec["install_package"],
}

Nigel Kersten wrote:

I’m looking for more information on service providers where we want this behavior, as I’m pretty sure we’re not going to implement this for every provider.

I don’t see how this behavior would be bad for any service provider. Where/how/why is it beneficial to ever start a service, update config files for that service, then restart that service a few seconds later? (Or many minutes later for those services that cannot start/stop quickly.)

This ticket is over a year old and there appears to be no forward progress on this issue, which is causing me great grief in trying to use puppet for my needs. I’m afraid I’m going to have start seriously looking at alternatives to puppet and discarding a heavy investment.

In one scenario, I’m trying to use puppet to configure bridging firewalls with VLAN support and to do so in the ideal modular way means puppet is needlessly bouncing several services, which then causes further problems downstream while waiting for switch ports to converge, clients behind those firewalls getting confused, etc. In short, this undesirable behavior is making puppet look very much like the wrong way to automate system configuration.

Having the same issue with configuring MongoDB:

130205-20:22:39 notice: /File[/var/lib/mongodb/journal]/ensure: created
130205-20:22:39 debug: Exec[mongo-journal-prealloc](provider=posix): Executing '/bin/dd if=/dev/zero of=/var/lib/mongodb/journal/prealloc.0 bs=1M count=1K && /bin/dd if=/dev/zero of=/var/lib/mongodb/journal/prealloc.1 bs=1M count=1K && /bin/dd if=/dev/zero of=/var/lib/mongodb/journal/prealloc.2 bs=1M count=1K && chmod 600 /var/lib/mongodb/journal/prealloc* && chown mongodb:mongodb /var/lib/mongodb/journal/prealloc*'
130205-20:22:39 debug: Executing '/bin/dd if=/dev/zero of=/var/lib/mongodb/journal/prealloc.0 bs=1M count=1K && /bin/dd if=/dev/zero of=/var/lib/mongodb/journal/prealloc.1 bs=1M count=1K && /bin/dd if=/dev/zero of=/var/lib/mongodb/journal/prealloc.2 bs=1M count=1K && chmod 600 /var/lib/mongodb/journal/prealloc* && chown mongodb:mongodb /var/lib/mongodb/journal/prealloc*'
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1024+0 records in
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1024+0 records out
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1073741824 bytes (1.1 GB) copied, 5.83921 s, 184 MB/s
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1024+0 records in
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1024+0 records out
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1073741824 bytes (1.1 GB) copied, 22.8184 s, 47.1 MB/s
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1024+0 records in
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1024+0 records out
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: 1073741824 bytes (1.1 GB) copied, 20.9342 s, 51.3 MB/s
130205-20:23:10 notice: /Stage[main]/Mongodb::Config/Exec[mongo-journal-prealloc]/returns: executed successfully
130205-20:23:10 debug: Service[mongod](provider=redhat): Executing '/usr/sbin/service-wait mongod status'
130205-20:23:10 debug: Puppet::Type::Service::ProviderRedhat: Executing '/sbin/chkconfig mongod'
130205-20:23:10 debug: Service[mongod](provider=redhat): Executing '/usr/sbin/service-wait mongod start'
130205-20:23:17 debug: Puppet::Type::Service::ProviderRedhat: Executing '/sbin/chkconfig mongod'
130205-20:23:17 debug: Puppet::Type::Service::ProviderRedhat: Executing '/sbin/chkconfig mongod on'
130205-20:23:17 notice: /Stage[main]/Mongodb::Service/Service[mongod]/ensure: ensure changed 'stopped' to 'running'
130205-20:23:17 debug: Service[mongod](provider=redhat): Executing '/usr/sbin/service-wait mongod status'
130205-20:23:17 debug: Service[mongod](provider=redhat): Executing '/usr/sbin/service-wait mongod restart'
130205-20:23:27 notice: /Stage[main]/Mongodb::Service/Service[mongod]: Triggered 'refresh' from 1 events

The code: https://github.com/Katello/katello/tree/master/katello-configure/modules/mongodb/manifests

It’s Puppet 2.6 from EPEL and I know this is the old one, but as guys describes it is still not fixed, just adding the log here for case.

Released in Puppet 3.2.0-rc1