The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #14837

Document the process of re-generating the PuppetDB cert if it's been revoked

Added by Gary Larizza almost 4 years ago. Updated almost 4 years ago.

Status:ClosedStart date:06/05/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:0.9.2
Keywords:docs Affected PuppetDB version:
Branch:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

As is usually the case, an accident like running puppet node clean puppetdb.host.lan will actually remove the cert for your PuppetDB host as well as cleaning the record out of storeconfigs (which is using PuppetDB). When you re-generate and re-sign the cert on the Puppet Master, you will get an error with PuppetDB. From the command line, any hosts that contact your master will respond with these errors:

root@gonzo-debian:~# puppet agent -t
info: Retrieving plugin
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/win_common_desktop_directory.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/iptables.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/root_home.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_vardir.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/facter_dot_d.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/concat_basedir.rb
info: Loading facts in /var/opt/lib/pe-puppet/lib/facter/puppet_ca_server.rb
err: Could not retrieve catalog from remote server: Error 400 on SERVER: Failed to submit 'replace facts' command for gonzo-debian.dc1.puppetlabs.net to PuppetDB at cookiemonster-centos.dc1.puppetlabs.net:8081: SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed.  This is often because the time is out of sync on the server or client
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

And your PuppetDB host will respond with these errors in its logs:

2012-06-05 15:30:20,529 WARN  [113216682@qtp-119195349-6] [mortbay.log] EXCEPTION 
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_revoked
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
    at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-06-05 15:30:26,712 WARN  [113216682@qtp-119195349-6] [mortbay.log] EXCEPTION 
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_revoked
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
    at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-06-05 15:30:39,701 WARN  [792223608@qtp-119195349-4] [mortbay.log] EXCEPTION 
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_revoked
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
    at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-06-05 15:32:37,242 WARN  [792223608@qtp-119195349-4] [mortbay.log] EXCEPTION 
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_revoked
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
    at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2012-06-05 15:32:37,967 WARN  [792223608@qtp-119195349-4] [mortbay.log] EXCEPTION 
javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_revoked
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1763)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1006)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1190)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1217)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1201)
    at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:675)
    at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

The solution is to run the /opt/puppet/sbin/puppetdb-ssl-setup command, modify /etc/puppetlabs/puppetdb/conf.d/jetty.ini, fix the permissions in /etc/puppetlabs/puppetdb, and restart puppetdb.

puppetdb-ssl-setup.txt Magnifier - puppetdb-ssl-setup script (1.84 KB) Jason Ashby, 06/19/2012 12:07 pm

History

#1 Updated by Deepak Giridharagopal almost 4 years ago

  • Status changed from Unreviewed to Accepted
  • Keywords set to docs

#2 Updated by Jason Ashby almost 4 years ago

My apologies if this is the wrong place or if I’m ‘doing it wrong’, but I encountered this issue and wanted to leave some notes. I can put this on github, but need to get myself up to speed on git first.

Per your solution above, I was unable to run puppetdb-ssl-setup. Instead, a bunch of options for the pkcs12 command were output to the terminal. This is because facter fqdn is not set on my puppet master, probably because I do not have domain mydomain.com set in /etc/resolv.conf.

The solution was to add a check if use facter hostname if fqdn is not set. In /usr/sbin/puppetdb-ssl-setup:

#
# facter fqdn may not be available. Use facter hostname instead.
#
fqdn=`facter fqdn`
if [ ! -n "$fqdn" ] ; then
  fqdn=`facter hostname`
fi

Also, to automate the additional changes per your solution above, I added these lines:

#
# add new password to jetty.ini
#
sed -e 's/^key-password.*/key-password = '"$password"'/' ${confdir}/conf.d/jetty.ini > ${tmpdir}/tmp.jetty
sed -e 's/^trust-password.*/trust-password = '"$password"'/' ${tmpdir}/tmp.jetty > ${confdir}/conf.d/jetty.ini

#
# permission fixes
# 
chown -R puppetdb:puppetdb ${confdir}/ssl
chown -R puppetdb:puppetdb ${confdir}/conf.d
chmod 640 ${confdir}/conf.d/jetty.ini

# moved cleanup to last line 
rm -rf $tmpdir

#3 Updated by Deepak Giridharagopal almost 4 years ago

Jason, that is extremely helpful. Thanks for the update, and we’ll see to getting these changes merged!

#4 Updated by Deepak Giridharagopal almost 4 years ago

  • Target version set to 1.0.0

#5 Updated by Jason Ashby almost 4 years ago

Hi Deepak, I just added pull request 192. I’m new to github, so let me know if you have any suggestions on pull requests, etc…

https://github.com/puppetlabs/puppetdb/pull/192

#6 Updated by Deepak Giridharagopal almost 4 years ago

Awesome, that helps a lot!

#7 Updated by Deepak Giridharagopal almost 4 years ago

  • Target version changed from 1.0.0 to 0.9.2

#8 Updated by Deepak Giridharagopal almost 4 years ago

  • Status changed from Accepted to Merged - Pending Release

#9 Updated by Matthaus Owens almost 4 years ago

  • Status changed from Merged - Pending Release to Closed

Released in PuppetDB 0.9.2

Also available in: Atom PDF