The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Certificates and Security

Overview

Puppet relies on standards wherever possible. In the case of security, it uses standard SSL certificates for client and master verification. This means that a client verifies that it’s talking with the right master and the master verifies that it’s talking with the right client. Because of the cost of buying signed certificates for every client and the complexity of managing your own certificate authority (CA), Puppet includes its own CA. This CA has been optimized for use with Puppet but could also be used to generate certificates for other purposes. The primary goal in certificate management within Puppet has been to keep it simple, and wherever possible to not make it even noticeable.

puppetca is the application for managing Puppet’s certification authority. It allows you to generate certificates, revoke them, sign them, remove them and list pending signature requests. By default the Puppet master, puppetmasterd, functions as a certificate authority.

Certificates

Certificates for clients and masters are generated automatically when puppetd or puppetmasterd are run for the first time, respectively. puppetd, the client side of Puppet, will accept the master’s certificate when it first connects to it and save it. From then on it will verify that the certificate it gets from the master verifies its identity.

You can manually copy the master certificate through a secure channel to the client if you want to, hence closing the small window for a man-in-the-middle attack.

Authentication

Certificates are the only method of authentication — if a client’s certificate can be verified using standard SSL verification mechanisms, then it is considered authenticated. Likewise for client validation of the master.

Encryption

Puppet also makes use of SSL to encrypt the traffic flow between master and clients.

Client Certificate Generation

Clients who do not yet have signed certificates will automatically generate a key pair and a certificate request, and then will connect to the master and provide it with the certificate request. If the master has autosign turned on (which is not necessarily recommended – see why autosign is dangerous?), then the autosign configuration file (which defaults to /etc/puppet/autosign.conf) is checked for whether the client’s name matches any contents. For instance, take the following configuration file:

hostname.domain.com
*.secure.domain.com

This configuration would autosign certificate requests for hostname.domain.com and any hosts coming from *.secure.domain.com.

This configuration file is read each time a signature is asked for, so changes to it can be short-lived and will be immediately noticed.

Master-Side Certificate Management

In the normal case, certificate auto-signing will be disabled. In these cases, certificates will have to be signed using the puppetca utility. Prior to the 1.0 release it is expected that there will be email notification of certificate requests waiting to be signed, but for now either the logs must be watched or puppetca —list can be used list waiting requests.

Once a request arrives, puppetca —sign <hostname> can be used to sign the request. Adding the —all flag will sign all outstanding requests. A list of all certificates ever issued by Puppet’s CA can be found in the file $cadir/inventory.txt.

All certificate files for a given host can be removed (for rebuilding hosts, for instance) with puppetca —clean <hostname>.

Certificates, once issued, can be revoked with puppetca —revoke <hostname|serial>. The server consults the certificate revocation list (CRL) every time a client tries to connect to the server; for revocations to take effect, the server must be restarted after the certificate revocation with puppetca.

Master-Side Client Certificate Generation

It is possible to generate certificates for clients on the master side, by using puppetca —generate <hostname>; this will also sign the newly generated certificate. You might want to do this if you want to automate the conversion of non-Puppet managed servers to Puppet. You can run a script that copies the generated client keys to the right place on the client, installs Puppet and then runs puppetd to get the client’s configuration. This would allow you to “Puppetize” a server with a single command. When you run puppetca —generate <hostname> three files are generated – $signeddir/hostname.pem, $certdir/hostname.pem and $privatekeydir/hostname.pem. You need to copy the private key, the client certificate, and the certificate for the master (ca.pem) to the corresponding directories on the client side. The client public key will be extracted from its certificate and put into $publickeydir when puppetd is run. The defaults for these directories should be subdirectories under /etc/puppet/ssl/ with self-explanatory names, look there to find the generated files.

This article provides an example of an implementation approach that uses USB keys to transfer the certificates from the master to client. Other approaches could include transferring the certificates during the KickStart, JumpStart), or similar OS post-installation script process.

Master-Side Client Certificate Revocation

As Puppet deploys a full PKI, it also allows to revoke a client certificate, if this client certificate or private key is compromised.

When a client certificate is revoked, its serial number is added to the current Certificate Revocation List (CRL for short). Since this file is signed by the CA certificate the CRL itself can be verified and can’t be tampered.

When a revoked client connects to the puppetmaster to check-out its configuration its connection is denied (provided you instructed your proxy if you use one to check the CRL, see Using Mongrel , Using Mongrel Nginx , Using Mongrel Pound , [[Using Passenger]]

The CRL if used to revoke only client certificate needs to be present only on the master. To revoke a certificate, issue on the master: puppetca —revoke <hostname>

Manual CA Configuration (optional)

If you are very paranoid you can consider setting up a separate CA manually rather than relying on puppet’s built-in CA. You can thereby restrict CA access to least privilege rather than letting puppetd and puppetmasterd do the work for you. Beware! This is for SSL experts only…

Adapt default permissions and locations for CA files and directories in your puppet installation’s default.rb along the following lines. If you don’t do this puppetmasterd will change permissions back to default each time you start it up. You have to install all keys and certificates before launching puppetmasterd/puppetd for the first time:

  • root CA key: neither readable nor writeable by puppetd/puppetmasterd or any other process, possibly installed on a separate and specially secured machine where you’ll do the signing of client certificate requests
  • trusted root CA certificate: readable but not writeable by puppetd/puppetmasterd
  • client CA keys: readable but not writeable by the process to be authenticated, not readable by any other process
  • certificate revocation list: readable but not writeable by puppetmasterd

Be aware though that a separate CA will not protect your puppet clients in all cases when your puppetmasterd has been compromised (see #1590 for a discussion). According to your individual security policy you’ll have to put other measures in place to keep malicious users from accessing puppetmasterd and puppetd processes or their configuration in the first place.

Access and Authorization

Puppet currently has few network functions, so security has so far been treated by them individually. It is expected that there will be some system-wide security hooks prior to the 1.0 release, but the certificate authentication already provides significant security.

Recommendations on approaches are heartily recommended.

Examples

Introduce a Client Machine to the Puppet Server

To introduce a client machine to the puppet server, first run puppetd on the client with the waitforcert parameter:

$ puppetd --waitforcert 30 --server puppetserver.domain.net -v

The client will request a certificate from the certificate authority facility on the server called puppetserver.domain.net.

Assuming that the machine puppetserver is not set to autosign any new certificate requests (this is the default), you’ll need to list the certificate requests from the command line on the server with an invocation like this:

$ puppetca --list

You’ll then be able to pick out the request by name (quite possibly, it will be the only request waiting) and instruct the certificate authority to sign the request:

$ puppetca --sign puppetclient-37.domain.net

Regenerate Certificates for Puppet Master

  • Stop puppet master.
  • Delete the puppet master’s certificate, private key, and public key:

      $ sudo find $(puppet master --configprint ssldir) -name "$(puppet master --configprint certname).pem" -delete
    
  • Edit the certname and certdnsnames settings in the puppet master’s /etc/puppet/puppet.conf file to match the puppet master’s actual hostnames.

  • Start a non-daemonized WEBrick puppet master instance, and wait for it to generate and sign a new certificate:

      $ sudo puppet master --no-daemonize --verbose
    
  • You should stop the temporary puppet master with ctrl-C after you see the “notice: Starting Puppet master version 2.6.9” message.

  • Restart the puppet master.

Common Caveats

  • When trying to run puppetd on a client and you get something like the following:
    warning: Certificate validation failed; consider using the certname configuration option
    err: Could not retrieve catalog: Certificates were not trusted: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed
    

    Find out what the puppet SSL directory is: Look for “ssldir” in /etc/puppet/puppet.conf – generally it’s either /var/puppet/ssl/ or /var/lib/puppet/ssl/

    Check that the clocks between the puppetmaster and client are in sync, if the client is behind when the certificate was generated, then the client sees that the start date of the certificate is in the future and therefore is invalid. Just for reference you can find out this info from the cert using the following:

    openssl x509 -text -noout -in /var/lib/puppet/ssl/certs/hostname.tld.pem | grep -A2 Validity
            Validity
                Not Before: Apr 19 23:21:29 2009 GMT
                Not After : Apr 18 23:21:29 2014 GMT
    

    This can also happen if you change the Puppet Master node. Then you need to clear the existing keys and certificates and reissue them. On the client delete the Puppet SSL directory and rerun puppet with “waitforcert” parameter. Make sure the master is clean as well (puppetca —clean )

%flipper — If Nick is rewriting, this will take one to two uninterrupted afternoons to replace. It should go into a new document in the /guides directory.