The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Feature #3143

Fully support multiple CAs and CA trust chains in Puppet

Added by Justin Baugh almost 5 years ago. Updated about 1 year ago.

Status:ClosedStart date:02/03/2010
Priority:HighDue date:
Assignee:-% Done:

0%

Category:SSL
Target version:-
Affected Puppet version:0.25.3 Branch:
Keywords:telly_deprecation customer

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.


Description

It would be very, very useful for Puppet to use existing enterprise-deployed PKI systems, rather than relying somewhat on the creation of its own self-signed CA.

Here is some background:

For a few months, I have been using 0.24.8 in production. I used the techniques described in the wiki (multiple CA certificates) to override the local CA certificate, setting localcacert in puppet.conf to the PEM bundle. This required a slight hack to prevent Puppet from overwriting the file, but it worked and verified correctly.

My PKI setup is fairly simple: I have a global root, which has a subordinate network CA, which has signed the Puppet CA (A->B->C).

I have recently upgraded to 0.25.4 and this has broken entirely (see http://pastebin.ca/1776458 for a client debug; the server side never sees the request because it doesn’t even get that far). I have tried both with a brand new client and an upgrade of an existing one – the errors are the same, failure to validate certificates.

From looking at cert_setup in network/http_pool.rb, Puppet will set the ca_file and SSL store as appropriate, but neither of these are seemingly sufficient to verify a trust chain. However, when I commented out usage of the certificate store and CA file (http://pastebin.ca/1776467) and used ca_path instead (set to an OpenSSL-hashed cert dir) containing A,B, and C’s PEM files, the verification succeeds, which confuses me greatly.

I also tried directly modifying the certificate store passed to the http object to load my certificates directly into the store (via add_file) and this also does not verify.

I notice that WEBrick deals with this issue by modifying the SSL context directly (extra_chain_cert). Is it possible for Puppet to do something similar? The property is not directly exposed from http.rb. When I have done testing with Puppet using WEBrick, modifying the appropriate WEBrick code to set the property to the appropriate trust chain (A and B) also works correctly, allowing the verification to succeed.

I apologize in advance if if I am overlooking something or doing it wrong.

In addition, I am happy to diagnose/debug/investigate this issue, if it is helpful, and write a patch for it if necessary, but my ruby/ruby openssl familiarity is minimal.


Related issues

Related to Puppet - Bug #3640: Added CRL disable option Closed 04/21/2010
Related to Puppet - Bug #3168: Cannot disable use of CRL in puppetd Investigating 02/06/2010
Related to Puppet - Bug #3120: 'localcacert' doesn't behave as described Closed 01/27/2010
Related to Puppet - Bug #3770: Puppet SSL verfication is broken with multiple chained ce... Duplicate 04/22/2010 04/22/2010
Related to Puppet - Feature #14550: Accept a CRL path on the agent Needs Decision 05/17/2012
Related to Puppet - Bug #3961: puppetca doesnt generate certificate in $certdir. Closed 06/08/2010
Related to Puppet - Bug #1525: local host fails to sync with mongrel/apache2 Closed 08/21/2008
Related to Puppet - Bug #9118: Puppet client does not update and does consult the crl du... Accepted 08/18/2011
Related to Puppet - Feature #7243: Additional data in Puppet CSRs (certdnsnames, and custom ... Closed 04/26/2011
Related to Puppet - Feature #6725: Serial # for x509 certificates Accepted 03/15/2011
Related to Puppet - Bug #4447: hostcert, hostcsr, hostprivkey, hostpubkey are not settings Needs More Information 08/03/2010
Related to Puppet - Bug #4273: SSL debug information not thread / process safe Closed 07/18/2010
Related to Puppet - Feature #3169: Add more debugging to SSL Cert verification Closed 02/09/2010
Related to Puppet - Bug #899: CRL signature failure when using apache/mongrel Closed
Related to Puppet - Feature #409: ca-bundle patch. Enable certificate authority chain veri... Closed
Related to Puppet - Feature #16842: CRLs on nodes other than the CA should occasionally update Duplicate 10/07/2012
Related to Puppet - Bug #4226: Puppet ca_name configuration setting should not default t... Closed 07/13/2010
Related to Puppet - Bug #16624: Using FreeIPA as CA with apache/mod_passenger fails to co... Needs More Information 09/27/2012
Related to Puppet - Bug #15561: Fix for CVE-2012-3867 is too restrictive Closed 07/17/2012
Duplicated by Puppet - Feature #15404: Fully support multiple CA's and CA trust chains in Puppet Duplicate 07/06/2012

History

#1 Updated by Justin Baugh almost 5 years ago

  • Priority changed from High to Normal

#2 Updated by James Turnbull almost 5 years ago

  • Status changed from Unreviewed to Investigating
  • Assignee set to Markus Roberts

#3 Updated by Nicholas Veeser almost 5 years ago

  • Affected Puppet version changed from 0.25.4 to 0.25.3

I will throw in my two cents, since I am up against the exact same problem. I am using a Cert chain that is part of an existing PKI solution.

The issue as I have diagnosed it is related to CRL being found

I have filed two bugs related to this:

  1. All puppetmasterd will run in CA mode http://projects.reductivelabs.com/issues/3141 As such, the server will respond to REST requests for a CRL, creating one if necessary.

  2. I cannot seem to find a way to tell the Client (Configurer/Agent) to stop looking for and using the CRL returned by its puppetmaster. http://projects.reductivelabs.com/issues/3168

Detail At some point, the Puppet::SSL::Host.ssl_store will initialize a OpenSSL::X509::Store with the certificate from Puppet[:localcert]

Optionally it will look for a CRL which is either cached locally or pulled from REST. If a CRL is found, it then enables “CRL checking” by flag in the SSL store (for use during SSL verification process). And given the above bugs, some a CRL can almost always be found.

if crl = Puppet::SSL::CertificateRevocationList.find("ca")
    Puppet.info "USING CRL: #{crl.content.issuer} Revoked: #{crl.content.revoked}"
    @ssl_store.flags = OpenSSL::X509::V_FLAG_CRL_CHECK_ALL|OpenSSL::X509::V_FLAG_CRL_CHECK
    @ssl_store.add_crl(crl.content)
end

I think (though I don’t know openssl that well), that for each cert in the verification chain, there must exist a CRL (named by ‘issuer’) signed by that cert. And with only 1 CRL and 3 certs in the CA chain, all verification will fail. For me, the failure happens when the current cert is Cert 2, (where 0 is the identity cert to verify, 1 signed 0, 2 signed 1, etc…) and the error returned during verify is (3): unable to get certificate CRL

Summary In the standard “Puppet is the CA” use case, the Cert chain is only one deep, so only one CRL (from the CA) is needed. If the cert chain is deeper, the CRL solution, as implemented breaks down, and Cert verification will always fail.

Sidenote FWIW I also filed this request: http://projects.reductivelabs.com/issues/3169

The code I added showed much more clearly what is happening in the X509 verification process which happens as part of SSL negotiation. ruby/OpenSSL does not seem to be very informative at this level. It may not be worth actually putting in the released code, but it may help you determine what is really going wrong.

#4 Updated by Nigel Kersten almost 4 years ago

  • Target version set to 2.7.x

#5 Updated by Nigel Kersten almost 4 years ago

To be clear, we’re going to spend some time investigating this in the immediate future because it’s important to a paying customer, and the results of that investigation will inform whether we can realistically resolve this in 2.6.x, or whether it needs to wait for another major version.

#6 Updated by Anonymous over 3 years ago

  • Assignee changed from Markus Roberts to Nigel Kersten

After investigating this the R&D team have concluded that we don’t have the ability to reduce this to a small, safe fix that would match the 2.6 branch guidelines, so our strong recommendations are:

  1. We should make this a statler feature, rather than a 2.6 feature.
  2. We need some sort of radical improvement; half our pain comes from the Ruby libraries for managing SSL, and their special behaviour in terms of mutating what we say into what they think we intended to say internally.

#7 Updated by Nigel Kersten over 3 years ago

  • Status changed from Investigating to Accepted
  • Assignee deleted (Nigel Kersten)
  • Priority changed from Normal to High

Perfect. Thank you for the research time.

#8 Updated by Nigel Kersten over 3 years ago

  • Target version changed from 2.7.x to 3.x

#9 Updated by Anonymous almost 3 years ago

I wanted to update the world: this has some technically challenging aspects, thanks to the Ruby OpenSSL bindings, and is especially security sensitive. That makes it a significant amount of work to get done, which is why this is taking longer than we might like to get resolved.

#10 Updated by Jeff McCune over 2 years ago

  • Tracker changed from Bug to Feature

#11 Updated by Jeff McCune over 2 years ago

As another update to the world, here is my take on this.

Basically, I think we need to add a bunch more settings that line up with Apache’s model of SSL servers and SSL clients. (They call them Proxies since they’re a web server.)

Overview

Puppet has accidentally supported multiple CA certificates and CA chaining up to and including Puppet 2.7. This support has never been “official” though and has always been based on work-arounds.

In order to fully support multiple certificate authorities and chains of trust Puppet needs to implement the following behaviors. All of these separate concerns are currently implemented in the localcacert behavior. Because Puppet uses a single self-signed CA certificate by default, a single settings to control these concerns makes sense. To fully support CA chaining and separate CA’s, however, we need the ability to separate out of the concerns of the localcacert setting.

Distinguish between client and server certificates

The default behavior of Puppet in 2.7 and previous versions is to create a single self signed CA certificate and use this certificate to issue all other certificates. All certificates issued by the Puppet CA are marked for client and server usage. This means a SSL certificate issued to a Puppet agent is able to be re-used as a SSL server certificate. This is a potential security risk as every agent possesses a signed certificate that could be used to serve up configuration catalogs to other agents. First and foremost, all SSL certificates

Distinguish between client and server SSL settings

Puppet is different from the traditional HTTPS model of SSL certificates in that both peers mutually authenticate each other using x.509 trust chains. In Puppet 2.7 and earlier the configuration options only allow a single CA certificate to be used for both sides of this handshake. This is the localcacert option.

If we are to fully support multiple CA’s then we need to expose the ability to configure Puppet agent’s and Puppet master’s with different trusted CA certificates. For example, I may want to issue all client (agent) certificates using the CA named “Puppet Agent Signing CA.” Similarly, I may want to issue all server (master) SSL certificates using a CA named “Puppet Master Signing CA” In this situation, I could simply point the localcacert setting for agents at one file and the localcacert setting for masters at another file but this poses a problem for the agent process running on the master itself. To disambiguate this situations I think we need configuration settings such as “ssl_client_cacert” and “ssl_server_cacert”

Distinguish between trusted certificates and authenticating certificates

When building a certificate chain we need to distinguish between CA certificates we trust and CA certificates we trust to authenticate peer connections. The Apache SSL documentation describes the situation pretty well:

This directive sets the optional all-in-one file where you can assemble the certificates of Certification Authorities (CA) which form the certificate chain of the server certificate. This starts with the issuing CA certificate of of the server certificate and can range up to the root CA certificate. Such a file is simply the concatenation of the various PEM-encoded CA Certificate files, usually in certificate chain order.
This should be used alternatively and/or additionally to SSLCACertificatePath for explicitly constructing the server certificate chain which is sent to the browser in addition to the server certificate. **It is especially useful to avoid conflicts with CA certificates when using client authentication. Because although placing a CA certificate of the server certificate chain into SSLCACertificatePath has the same effect for the certificate chain construction, it has the side-effect that client certificates issued by this same CA certificate are also accepted on client authentication. That's usually not one expect.**

Support verification depth

As soon as we officially support certificate chains, we need to tune how many intermediate certification authorities can be used to “fill in” the gap from a known authentic CA to a SSL certificate. This could default to a reasonable number (5-7?), but should probably be greater than 1 and less than 10. It should definitely be end-user tunable though.

#12 Updated by Jeff McCune over 2 years ago

  • Subject changed from Puppet should correctly support CA trust chains to Fully support multiple CAs and CA trust chains in Puppet

#13 Updated by Jeff McCune over 2 years ago

  • Assignee set to Jeff McCune

I’m taking this on as the final part of my support goalie week. It will almost certainly spill over into much of next week and beyond but I think it’s worth it for such an old and oft-requested feature.

As I work on this, please update this ticket with any impact data or use cases you have in mind.

Thanks, -Jeff

#14 Updated by Justin Baugh over 2 years ago

Hi,

Awesome to see this bug, er, feature request, finally being handled.

I think there are two big use cases here:

1) Allow puppet to be configured to support arbitrary CA paths and certificates / keys (in an instance where I already have an existing enterprise PKI and my PKI administrator gives me a Puppet CA Certificate signed by the existing CA path) 2) Allow puppet to spit out a CSR which can be signed by an upstream CA. This would be quite handy.

#15 Updated by Jeff McCune over 2 years ago

  • Assignee deleted (Jeff McCune)

Back into the backlog

In the Puppet / Facter planning meeting for today we decided to put this back into the backlog because it’s a larger change than an iteration week.

I’m going to work #3120 and make sure localcacert behaves in a manner compatible with properly supporting CA chains in Telly. This largely means making sure the following three behaviors have distinct settings:

  • The CA uses $localcacert to issue certificates. It should use cacert.
  • The agent uses $localcacert to authenticate master servers.
  • The master uses $localcacert to authenticate agent clients.

This ticket is not currently being actively worked on as a result of these decisions.

#16 Updated by eric sorenson over 2 years ago

  • Keywords set to telly_deprecation

Not technically a deprecation but I’m adding it to the list since we’re defining new config settings and changing behaviour.

#17 Updated by Charlie Sharpsteen over 1 year ago

  • Assignee set to Charlie Sharpsteen

#18 Updated by Charlie Sharpsteen over 1 year ago

  • Keywords changed from telly_deprecation to telly_deprecation customer

#20 Updated by Charlie Sharpsteen over 1 year ago

  • Status changed from Accepted to Closed
  • Assignee deleted (Charlie Sharpsteen)
  • Target version deleted (3.x)

With the 3.2.1 release, Puppet now supports three external CA setups:

  • Single self-signed CA which directly issues SSL certificates.
  • Single, intermediate CA issued by a root self-signed CA. The intermediate CA directly issues SSL certificates; the root CA doesn’t.
  • Two intermediate CAs, both issued by the same root self-signed CA, in the following configuration:
    • One intermediate CA issues SSL certificates for puppet master servers.
    • The other intermediate CA issues SSL certificates for agent nodes.
    • Agent certificates can’t act as servers, and master certificates can’t act as clients.

Full documentation can be found here:

http://docs.puppetlabs.com/puppet/3/reference/config_ssl_external_ca.html

#21 Updated by Mark Zeren over 1 year ago

  • Support Urls deleted (http://support.puppetlabs.com/tickets/245 https://support.puppetlabs.com/tickets/538)

Note we still have two pieces of commented out code associated with this closed bug. As of: ac95811e101c07767006f097f49469bbd71ab284 andy@puppetlabs.com – Merge pull request #1708 from jjulien/remove_rundir_patch

lib/puppet/defaults.rb:544: # intended for (#3143) and is not expected to be used until CA chaining is lib/puppet/defaults.rb:563: # intended for (#3143) and is not expected to be used until CA chaining is

#22 Updated by Adrien Thebo about 1 year ago

Mark,

Apologies for a lack of response on this issue; if you’re having issues with the external CA implementation that means you have to comment out code could you file a new bug for that?

Also available in: Atom PDF