The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #16769

Apache "SSLOptions +ExportCertData" causes "header too long" error

Added by eric sorenson over 3 years ago. Updated over 3 years ago.

Status:ClosedStart date:10/03/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:SSL
Target version:3.0.1
Affected Puppet version:3.0.0 Branch:
Keywords:ssl

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Reported on the mailing list at https://groups.google.com/forum/?fromgroups=#!topic/puppet-users/tpKvbor15iY

This was added as part of #7962.


Related issues

Related to Puppet - Feature #7962: Warn when a certificate approaches the expiration date Closed 06/16/2011

History

#1 Updated by eric sorenson over 3 years ago

  • Target version set to 3.0.1
  • Affected Puppet version set to 3.0.0
  • Keywords set to ssl

#2 Updated by Jeremy MAURO over 3 years ago

Hi,

Since the upgrade from puppet 2.7.18 to 3.0.0, I am using puppet+passenger on debian squeeze with the package from puppetlabs:


# dpkg -l 'puppet*' |grep ii
ii  puppet                                                 3.0.0-1puppetlabs1           Centralized configuration management - agent startup and compatibility scripts
ii  puppet-common                                          3.0.0-1puppetlabs1           Centralized configuration management
ii  puppet-dashboard                                       1.2.11-1puppetlabs1          Dashboard for Puppet
ii  puppetdb                                               1.0.0-1puppetlabs1           PuppetDB Centralized Storage.
ii  puppetdb-terminus                                      1.0.0-1puppetlabs1           Connect Puppet to PuppetDB by setting up a terminus for PuppetDB.
ii  puppetmaster                                           3.0.0-1puppetlabs1           Centralized configuration management - master startup and compatibility scripts
ii  puppetmaster-common                                    3.0.0-1puppetlabs1           Puppet master common scripts

# dpkg -l '*apache*' | grep ii
ii  apache2                                                2.2.16-6+squeeze8            Apache HTTP Server metapackage
ii  apache2-mpm-worker                                     2.2.16-6+squeeze8            Apache HTTP Server - high speed threaded model
ii  apache2-utils                                          2.2.16-6+squeeze8            utility programs for webservers
ii  apache2.2-bin                                          2.2.16-6+squeeze8            Apache HTTP Server common binary files
ii  apache2.2-common                                       2.2.16-6+squeeze8            Apache HTTP Server common files
ii  libapache2-mod-passenger                               2.2.11debian-2               Rails and Rack support for Apache2

# dpkg -l '*passenger*'
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                       Version                                    Description
+++-==========================================-==========================================-====================================================================================================
ii  libapache2-mod-passenger                   2.2.11debian-2                             Rails and Rack support for Apache2
un  passenger-doc                                                                   (no description available)
un  puppetmaster-passenger                                                          (no description available)

# gem list

*** LOCAL GEMS ***

rack (1.1.2)
rake (0.9.2.2)

Currently using the following apache configuration file:


Listen 8140
<VirtualHost *:8140>
        ServerName puppetmaster.fqdn

        ErrorLog /var/log/apache2/puppetmaster_error.log
        LogLevel warn
        SetEnvIf Remote_Addr "::1" dontlog
        CustomLog /var/log/apache2/puppetmaster_access.log combined env=!dontlog

        SSLEngine on
        SSLProtocol -ALL +SSLv3 +TLSv1
        SSLCipherSuite ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP

        SSLCertificateFile      /data/local/puppet/ssl/certs/puppetmaster.fqdn.pem
        SSLCertificateKeyFile   /data/local/puppet/ssl/private_keys/puppetmaster.fqdn.pem
        SSLCertificateChainFile /data/local/puppet/ssl/ca/ca_crt.pem
        SSLCACertificateFile    /data/local/puppet/ssl/ca/ca_crt.pem

        ## CRL checking should be enabled; if you have problems with
        ## Apache complaining about the CRL, disable the next line
        SSLCARevocationFile     /data/local/puppet/ssl/ca/ca_crl.pem
        SSLVerifyClient         optional
        SSLVerifyDepth          1
        SSLOptions              +StdEnvVars +ExportCertData

        # This header needs to be set if using a loadbalancer or proxy
        # RequestHeader unset X-Forwarded-For

        ## The following client headers allow the same configuration
        ## to work with Pound.
        RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
        RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

        RackAutoDetect On

        DocumentRoot /var/www/puppetmaster/public/
        RackBaseURI /
        <Directory /var/www/puppetmaster/public/>
                Options None
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>
</VirtualHost>

And the following config.ru:



# a config.ru, for use with every rack-compatible webserver.
# SSL needs to be handled outside this, though.

# if puppet is not in your RUBYLIB:
# $LOAD_PATH.unshift('/opt/puppet/lib')

$0 = "master"

# if you want debugging:
# ARGV << "--debug"

ARGV << "--rack"

# Rack applications typically don't start as root.  Set --confdir to prevent
# reading configuration from ~/.puppet/puppet.conf
ARGV << "--confdir" << "/etc/puppet" << "--config=/etc/puppet/conf/puppet.conf"

# NOTE: it's unfortunate that we have to use the "CommandLine" class
#  here to launch the app, but it contains some initialization logic
#  (such as triggering the parsing of the config file) that is very
#  important.  We should do something less nasty here when we've
#  gotten our API and settings initialization logic cleaned up.
#
# Also note that the "$0 = master" line up near the top here is
#  the magic that allows the CommandLine class to know that it's
#  supposed to be running master.
#
# --cprice 2012-05-22

require 'puppet/util/command_line'
# we're usually running inside a Rack::Builder.new {} block,
# therefore we need to call run *here*.
run Puppet::Util::CommandLine.new.execute

I having issue when trying to get new certificat:


info: Creating a new SSL key for linux-install.fqdn
err: Could not request certificate: Error 400 on SERVER: header too long
Exiting; failed to retrieve certificate and waitforcert is disabled

Problem is gone without the ‘+ExportCertData’ SSLOptions


SSLOptions              +StdEnvVars

#3 Updated by Chris Spence over 3 years ago

+1 to this ticket (and incidentally the workaround) – I ran into the same problem:

  • what version of passenger?

passenger (3.0.17)

  • what version of apache?

Server version: Apache/2.2.22 (Debian)

Do we think Passenger 3+ is ok?

#4 Updated by Anonymous over 3 years ago

Jeremy,

In an effort to help me understand the problem, could you describe what you need the ExportCertData option set in the Apache virtual host? This option causes the entire PEM encoded certificate to be copied into the environment, and our certificates are larger in Puppet 3.0.0 than they previously have been because the default key length has increased and the signing algorithm has changed from md5 to sha256.

Understanding the need for ExportCertData will help me come up with a fix or a work around to this issue.

Thanks, -Jeff

#5 Updated by Anonymous over 3 years ago

Jeff McCune wrote:

Jeremy,

In an effort to help me understand the problem, could you describe what you need the ExportCertData option set in the Apache virtual host? This option causes the entire PEM encoded certificate to be copied into the environment, and our certificates are larger in Puppet 3.0.0 than they previously have been because the default key length has increased and the signing algorithm has changed from md5 to sha256.

Understanding the need for ExportCertData will help me come up with a fix or a work around to this issue.

Thanks, -Jeff

Jeremy,

Sorry, please disregard. I see now that we need this option setin as part of #7962 and overlooked that information when I updated this ticket.

-Jeff

#6 Updated by eric sorenson over 3 years ago

  • Assignee set to Steven Lindberg

Hi Steven – Did you come across the issue reported in #16769 while testing the new code?

We’re getting several reports across the mailing list/ irc / bug tracker that SSLOptions +ExportCertData is not compatible with Passenger, which is the primary way most people run puppetmasters at scale. The issue is that Apache puts the cert data into environment variables and passes them to Passenger per the Rack specification, but Passenger has hard-coded length limits in the header sizes it will accept, so operations do not even reach the puppet master.

I think we need to figure out how to extract just the ValidityPeriod information from the certificate at the Apache layer, rather than passing the whole blob through and regenerating a whole X509 object at https://github.com/puppetlabs/puppet/blob/339ed9ec6fa7bdc37f4bcf0fb8e4a533badf746a/lib/puppet/network/http/rack/rest.rb#L82

Any thoughts?

#7 Updated by Steven Lindberg over 3 years ago

  • Assignee changed from Steven Lindberg to eric sorenson

Ah bummer… I didn’t think adding this option would have a significant impact. I never came across this issue on my development box:

Ubuntu 12.04 (uname -rm: 3.2.0-30-generic x86_64)
apache v2.2.22
ruby v1.8.7-p370
passenger v3.0.17
rack v1.4.1

My only guess is that a having more than just one cert in any certificate chain pushes it over that mysterious hard limit, although it doesn’t look like Jeremy is doing anything fancy certificate-wise… I would have guessed it has to do with using a recent version of passenger, but Chris is using the same version as I was.

In regards to extracting only the ‘ValidityPeriod’, looks like the SSL_CLIENT_V_END var will give just the client certificate’s expiration date without requiring the +ExportCertData option. Puppet::Network::HTTP::Handler will need to get updated to provide just that information, and the WEBRick handler as well. The warn_if_near_expiration() function also currently expects full-fledged certificates and will need modification.

However, as it stands the option is only necessary if the admin wants agent certificate expiration warnings to appear in the master logs. If omitted, master and CA cert expiration warnings will still appear in the master logs, and agent cert warnings will still appear in the agent logs, and therefore in reports. As I understood it, this covered situations where reports (and agent logs in general) were not utilized, and the puppetmaster log is the primary source of information about the status of nodes. Hopefully in larger clusters — ones that are likely going to be using passenger — admins can rely on reports to show the agent certificate expiration warnings (or maybe they don’t even care?), and the extra +ExportCertData option can just be omitted.

There’s a note above the config option in the example apache.conf stating what it is used for, perhaps it would be best to change the language to indicate that it is not crucial for operation?

steven

#8 Updated by Anonymous over 3 years ago

  • Category set to SSL
  • Status changed from Needs More Information to Investigating
  • Assignee changed from eric sorenson to Anonymous

#9 Updated by Anonymous over 3 years ago

I’m diving into this issue now. I’ve added Matthaus as a watcher, I’ll update this ticket with relevant information if the vhost configuration settings change for Apache so that the packages can be kept up to date.

#10 Updated by Anonymous over 3 years ago

I’m able to reproduce using Puppet Enterprise packages (Passenger 2.2.11) with puppet 3.0.x running out of source.

Trying to figure out how best to enable passenger debugging now.

#11 Updated by Anonymous over 3 years ago

Pull request

https://github.com/puppetlabs/puppet/pull/1222

The issue was that we weren’t dealing with the edge case of an agent with no signed certificate.

-Jeff

#12 Updated by Anonymous over 3 years ago

  • Status changed from Investigating to Merged - Pending Release
  • Assignee deleted (Anonymous)

Merged into 3.0.x

As 8df9825

Merged up into master.

Assuming this passes through CI without issue, this should be released with Puppet 3.0.1.

-Jeff

#13 Updated by Matthaus Owens over 3 years ago

  • Status changed from Merged - Pending Release to Closed

Released in Puppet 3.0.1-rc1

Also available in: Atom PDF