Puppet

h2. Background

We have a three tier deployment: top level is one puppetmaster that acts as CA and as master for the second level. At the second level we have two or more load balanced puppet masters. These second level puppet masters have their CA disabled and point to ca_server to the top level puppet master. All puppet masters have the certdnsnames option set as follows:

h3. Top level (puppeteer)

[puppetmasterd]
autosign = true
autosign = /etc/puppet/autosign.conf
certdnsnames = puppeteer.domain.com

# for Passenger
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

h3. Second level (puppet masters)

[puppetmasterd]
certdnsnames = puppet.domain.com
  
# for Passenger
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY

# CA
ca = false
ca_server = puppeteer.domain.com

[puppetd]
server = puppeteer.domain.com

h3. Third level (puppet client)

[main]
ca_server = puppeteer.domain.com

[puppetd]
server = puppet.domain.com

In order for this to work, we have to: * Start the puppeteer * Run puppetd on the second level puppetmasters to generate their certificates and have them signed by the puppeteer CA * Start the puppetmasterd on the second level masters

h2. Problem

The problem arises when the third level clients connect to any of the second level masters:

puppetd --test --debug
...
...
err: Could not retrieve catalog from remote server: hostname was not match with the server certificate
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

Investigating the issue we saw that the puppeteer CA did not apply the correct certdnsnames to the second level puppetmasters. Even worse, it applied its own aliases to ALL the certificates it signed!

puppetca --print puppetter-hostname.domain.com
puppeer-hostname.domain.com
...
X509v3 Subject Alternative Name: 
                DNS:puppeteer.domain.com, DNS:puppeteer-hostname.domain.com
...

puppetca --print any-puppet-master-hostname.domain.com
puppeer-hostname.domain.com
...
X509v3 Subject Alternative Name: 
                DNS:puppeteer.domain.com, DNS:any-puppet-master-hostname.domain.com
...

puppetca --print any-puppet-client-hostname.domain.com
puppeer-hostname.domain.com
...
X509v3 Subject Alternative Name: 
                DNS:puppeteer.domain.com, DNS:any-puppet-client-hostname.domain.com
...

We are currently working around the issue by assigning both puppeteer and puppet aliases to the puppeteer so when the certificates are generated the have both hostnames and the clients don’t complain.