Tukukkk

Consider the following code:

file { '/etc/systemd/system/docker.service.d/http-proxy.conf':

  ensure  => 'present',

  owner   => 'root',

  group   => 'root',

  mode    => '644',

  content => '[Service]

Environment="HTTP_PROXY=http://10.0.2.2:3128"

Environment="HTTPS_PROXY=http://10.0.2.2:3128"

',

  notify  => Exec['daemon-reload'],

  require => Package['docker-ce'],

}



exec { 'daemon-reload':

  command     => 'systemctl daemon-reload',

  path        => '/sbin',

  refreshonly => true,

}



service { 'docker':

  ensure    => 'running',

  subscribe => File['/etc/systemd/system/docker.service.d/http-proxy.conf'],

  require   => Exec['daemon-reload'],

}

I would like to edit some systemd service. In this instance, it is the environment for docker, but it could be any other need.

Since a systemd unit file has been changed, systemctl daemon-reload must be run for the new configuration to be picked up.

Running puppet apply fails:

Notice: Compiled catalog for puppet-docker-test.<redacted> in environment production in 0.18 seconds

Notice: /Stage[main]/Main/File[/etc/systemd/system/docker.service.d/http-proxy.conf]/ensure: defined content as '{md5}dace796a9904d2c5e2c438e6faba2332'

Error: /Stage[main]/Main/Exec[daemon-reload]: Failed to call refresh: Could not find command 'systemctl'

Error: /Stage[main]/Main/Exec[daemon-reload]: Could not find command 'systemctl'

Notice: /Stage[main]/Main/Service[docker]: Dependency Exec[daemon-reload] has failures: false

Warning: /Stage[main]/Main/Service[docker]: Skipping because of failed dependencies

Notice: Applied catalog in 0.15 seconds

The cause is immediately obvious: systemctl lives in /bin, not /sbin, as configured. However, fixing this, then running puppet apply again will neither cause the service to be restarted nor systemctl daemon-reload to be run:

Notice: Compiled catalog for puppet-docker-test.<redacted> in environment production in 0.19 seconds

Notice: Applied catalog in 0.16 seconds

Apparently, this happens because there were no changes to the file resource (since it was applied on the failed run), which would have refreshed the daemon-reload and then triggered the service to restart.

In order to force puppet to reload the service and restart it, I could change the contents of the file on disk, I could change the contents on the puppet code, but it feels like I'm missing some better way of doing this.

How to better recover from such scenario? Or, how to write puppet code that doesn't have this issue?

Puppet does not provide a mechanism for resuming a failed run. Doing so would not make much sense to me, since one would expect that for a resumption to have a different result would depend on the machine state being changed since the failure, and a machine-state change outside Puppet potentially invalidates the catalog that was being applied.

The agent does, by default, send run reports to the master, so in the event of a failed run, you should be able to determine from that what went wrong. Supposing that you don't want to scour reports to figure out how to recover from a failed run, however, you could consider compiling a recovery script.

For example, you know that any failure may have caused a daemon-reload to be missed, and it's harmless to perform one when it isn't required, so just put that in your script. You might also put in a restart of each service under management. Basically, you're looking for anything that has non-trivial refresh behavior (Execs and Services are the main ones that come to my mind).

It occurs to me that if you're extra clever then it might be possible to put that in the form of one or more Puppet classes, and to determine on the master whether the last run for the target node failed, so as to apply the recovery.

As for avoiding the problem in the first place, I can only suggest testing, testing, and more testing. To that end, if you don't have dedicated machines on which to test Puppet updates, then at least select a small number of normal machines that get updates first, so that any problems that arise are closely contained.