The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Version 1/3 - Next ยป - Current version
Anonymous, 03/13/2010 08:02 pm


Using Mongrel with Pound

STILL INCOMPLETE

Alternatively, see: Using Mongrel and Using Mongrel Nginx

This solution freshly baked by James Turnbull, Jeff McCune, and Nigel Kersten.

Date: 2007-11-15

Why choose Pound

  • Pound is very small and fast.
  • Pound gives you fine-grained control of your SSL setup and client authorization.
  • Pound is probably easier and faster to setup than Apache.
  • For very-large-installations (3,000+ clients), pound may (confirmation needed) scale better than mod_proxy_balancer.
  • Pound is able to do sessions based on IP.
  • Pound checks Certificate Revocation Lists

Why NOT Choose Pound

  • With older versions of Mongrel, you may need to patch pound or upgrade Mongrel. (See issue 2 below, regarding multi-line request headers.)

Overview

Out of the box, we encountered a number of issues getting Pound to proxy puppetd requests to mongrel and ultimately puppetmasterd.

  1. Pound formats the x.509 subject string differently than Apache does. Puppetmasterd uses the x.509 subject as the client name for authorization purposes.
  2. Pound adds a multi-line X-SSL-Certificate header, containing the PEM encoded certificate of the client. Mongrel does not accept multi-line HTTP headers.
  3. The header Pound uses for the x.509 Subject is X-SSL-Subject, while Apache uses X-Client-DN.
  4. Pound requires it’s server SSL certificates in a particular format and order.
  5. Pound is pretty aggressive about short timeouts on the client and back-end connections.

Issues 1 and 2 required small patches to puppet and pound, as documented in ticket:906 . Both patches have been submitted to the puppet and pound and will hopefully be included in future releases. As of puppet version 0.23.2 and Pound version 2.3.2, both patches still need to be applied to the code tree.

Issue 3 requires puppetmasterd be configured with the option ssl_client_header=HTTP_X_SSL_SUBJECT to work with Pound.

Issue 4 requires the PEM certificate and key files to be concatenated into a single file.

Issue 5 simply requires adjusting timeout values in the pound configuration file.

Patches

To ensure the solution works we need to apply patches to both Puppet and Pound.

Patching Puppet

There are patches available for the Puppet 0.23.2 version and the trunk.

If you are running Puppet 0.23.2 apply the following patch:

$ curl -o /tmp/puppet_pound.patch "http://reductivelabs.com/trac/puppet/attachment/ticket/906/puppet_pound_for_0.23.2.patch?format=raw"
$ cd puppet-0.23.2
$ patch -p0 < /tmp/puppet_pound.patch

If you are running the Puppet trunk version then apply the following patch:

$ curl -o /tmp/puppet.patch "http://reductivelabs.com/trac/puppet/attachment/ticket/906/0001-Fixed-puppet-master-parsing-bug-when-mongrel-is-used.patch?format=raw"
$ cd puppet-trunk/
$ patch -p1 < /tmp/puppet.patch

Patching Pound

You will also need to Patch the current version of Pound, 2.3.2. Download and unpack the Pound source:

$ wget http://www.apsis.ch/pound/Pound-2.3.2.tgz
$ tar -zxf Pound-2.3.2.tgz
$ cd Pound-2.3.2

Download and apply the Pound patch:

$ curl -o /tmp/pound.patch "http://reductivelabs.com/trac/puppet/attachment/ticket/906/0001-Pound-Single-Line-x509certificate-header.patch?format=raw"
$ patch -p1 < /tmp/pound.patch

Compile and install Pound:

$ ./configure --prefix=/usr --sysconfdir=/etc
$ make
$ make install

Here Pound is configured to install into /usr and use /etc as a configuration location.

Setting up Pound

Pound SSL Certificates

Pound needs a PEM file containing the SSL Private Key, the signed x.509 certificate, and any certificates in the chain of trust.

Note: Order IS IMPORTANT. Pound requires that it “walk” the chain from the top of the PEM file to the Bottom, so the root trusted authority should be at the end of the file, and the server’s certificate should be at the “top” of the PEM file.

To create this file of key, server cert, and CA certificate we concatenate together our existing Puppet certificates. Here we assume your certificates are stored in /etc/puppet/ssl.

$ cat /etc/puppet/ssl/private_keys/host_key.pem >> \
  /etc/puppet/ssl/pound/host_key_and_cert_chain.pem

$ cat /etc/puppet/ssl/certs/host_cert.pem >> \
  /etc/puppet/ssl/pound/host_key_and_cert_chain.pem

$ cat /etc/puppet/ssl/ca/ca_crt.pem >> \
  /etc/puppet/ssl/pound/host_key_and_cert_chain.pem

Pound Configuration

Please note that for those following along in the Apache configuration, client SSL verification is optional there, as Apache can set the X-Client-Verify header dynamically depending on the success or failure of the SSL verification.

In pound, we don’t have dynamic variable expansion, so we instead configure two listener ports 8140 and 8140. 8140 requires a valid, verified certificate as it will ALWAYS set X-Client-Verify: SUCCESS to proxied connections. 8140 does not set the X-Client-Verify header.

##################################################
# pound.cfg for use with mongrel => puppetmasterd
#
# Jeff McCune <jeff@northstarlabs.net>
#

User            "puppet"
Group           "puppet"

## Set log level
LogLevel        2

## Fork to the background
Daemon 1

## Check back-end mongrel servers every 30 seconds.
Alive           30

# Client timeout in seconds.  Pound will kill the connection if it does
# not receive any data after this many seconds.
Client 300

# How long should Pound wait for a response from the back-end (in seconds).
TimeOut 180

# Authenticated clients connect to port 8140
ListenHTTPS
  Address 0.0.0.0
  Port    8140

  # THIS IS VERY IMPORTANT.  Without it, clients may forge
  # the X-Client-Verify: SUCCESS header.
  # I believe this is correct, but have not verified it.
  HeadRemove "X-SSL-.*"
  HeadRemove "X-Client-Verify.*"

  ####    Pound's SSL certificate, key file, and CA chain of trust.
  Cert    "/etc/puppet/ssl/pound/host_key_and_cert_chain.pem"

  # This is the list of CA's we ultimately trust.
  # If a certificate's chain of trust is ROOTED with a CA in this list,
  # The client connection is authorized.
  VerifyList "/etc/puppet/ssl/ca/ca_crt.pem"

  # This is the list of CA's we will permit to exist in the CHAIN
  # from client certificate to ROOT CA's listed in VerifyList.
  # This list is sent to the client, so they may pick the correct
  # certificate to submit.
  CAlist "/etc/puppet/ssl/ca/ca_crt.pem"

  ## THIS IS WHERE ALL YOUR SECURITY STEMS FROM.  TAKE CARE!
  # ClientCert 0|1|2|3 depth
  #   0 - don't ask (default),
  #   1 - ask,
  #   2 - demand, and require valid chain to a root CA listed in VerifyList
  #   3 - demand (I think) but do not verify
  #   depth is the depth of verification for a client certificate (up to 9).
  ClientCert 2 4

  # Revocation is important.
  CRLlist    "/etc/puppet/ssl/ca/ca_crl.pem"

  # Pound, by default is too strict with SSL ciphers.
  Ciphers    "SSLv2:-LOW:-EXPORT:RC4+RSA"

  # If all of the above SSL stuff checks out...  Then
  # NOTE: I believe there is a bug in Pound where only one
  # header may be added.
  AddHeader "X-Client-Verify: SUCCESS"

  # Forward the connection back to the Mongrel servers.
  Service
    BackEnd
      Address 127.0.0.1
      Port    18140
    End
    BackEnd
      Address 127.0.0.1
      Port    18141
    End
    # etc...
    Session
      Type IP
      TTL  300
    End
  End
End

## Listen on port 8141 for CA signing requests.
ListenHTTPS
  Address    0.0.0.0
  Port       8141
  HeadRemove "X-SSL-.*"
  HeadRemove "X-Client-Verify.*"
  Cert    "/etc/puppet/ssl/pound/host_key_and_cert_chain.pem"
  VerifyList "/etc/puppet/ssl/ca/ca_crt.pem"
  CAlist "/etc/puppet/ssl/ca/ca_crt.pem"
  ClientCert 3 9
  CRLlist    "/etc/puppet/ssl/ca/ca_crl.pem"
  Ciphers    "SSLv2:-LOW:-EXPORT:RC4+RSA"

  Service
    BackEnd
      Address 127.0.0.1
      Port    18140
    End
    BackEnd
      Address 127.0.0.1
      Port    18141
    End
    Session
      Type IP
      TTL  300
    End
  End
End

Starting Pound

Now we can start Pound:

# pound -f /etc/pound/pound.cfg -p /var/run/pound.pid

Debugging Pound

Pound logs to syslog – on Debian this defaults to logging to /var/log/daemon.log. If you use syslog-ng you can direct your logging output elsewhere using a filter like:

destination df_pound { file("/var/log/pound.log"); };
filter f_pound { program("pound"); };
log {
        source(s_all);
        filter(f_pound);
        destination(df_pound);
};

Start puppetmasterd daemons

You need to run the puppetmasterd daemon using the Mongrel web server and run each instance on the port numbers indicated in our BackEnd clauses defined in our pound.cfg. You can use a script like this to start each instance:

#!/bin/bash
# name: mongrel_puppetmasterd
# Start a Puppet Master Server instance.

if ! [[ "$1" -gt 0 ]]; then
echo "ERROR: You must provide a port to run this puppet master on."
echo "Ensure your apache load balancer is configured to talk to these servers"
exit 1
fi

MASTERPORT="$1"
shift

puppetmasterd \
--pidfile=/var/run/puppetmasterd."${MASTERPORT}".pid \
--servertype=mongrel \
--ssl_client_header=HTTP_X_SSL_SUBJECT \
--masterport="${MASTERPORT}" \
$*

The script is initiated by passing the port number you wish to start the puppetmasterd on. To start puppetmasterd daemons for the Back End? instances we’ve defined in our pound.cfg you would:

# mongrel_puppetmasterd 18140
# mongrel_puppetmasterd 18141

Test clients

Now attempt to test a client connection.