The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Using Mongrel with Pound

STILL INCOMPLETE

Alternatively, see: Using Mongrel and Using Mongrel Nginx

This solution freshly baked by James Turnbull, Jeff McCune, and Nigel Kersten.

Date: 2007-11-15

Why choose Pound

  • Pound is very small and fast.
  • Pound gives you fine-grained control of your SSL setup and client authorization.
  • Pound is probably easier and faster to setup than Apache.
  • For very-large-installations (3,000+ clients), pound may (confirmation needed) scale better than mod_proxy_balancer.
  • Pound is able to do sessions based on IP.
  • Pound checks Certificate Revocation Lists

Why NOT Choose Pound

  • With older versions of Mongrel, you may need to patch pound or upgrade Mongrel. (See issue 2 below, regarding multi-line request headers.)

Overview

Out of the box, we encountered a number of issues getting Pound to proxy puppetd requests to mongrel and ultimately puppetmasterd.

  1. Pound formats the x.509 subject string differently than Apache does. Puppetmasterd uses the x.509 subject as the client name for authorization purposes.
  2. Pound adds a multi-line X-SSL-Certificate header, containing the PEM encoded certificate of the client. Mongrel does not accept multi-line HTTP headers.
  3. The header Pound uses for the x.509 Subject is X-SSL-Subject, while Apache uses X-Client-DN.
  4. Pound requires it’s server SSL certificates in a particular format and order.
  5. Pound is pretty aggressive about short timeouts on the client and back-end connections.

Issues 1 and 2 may require small patches to puppet and pound, as documented in #906.

Issue 3 requires puppetmasterd be configured with the option ssl_client_header=HTTP_X_SSL_SUBJECT to work with Pound.

Issue 4 requires the PEM certificate and key files to be concatenated into a single file.

Issue 5 simply requires adjusting timeout values in the pound configuration file.

Setting up Pound

Pound SSL Certificates

Pound needs a PEM file containing the SSL Private Key, the signed x.509 certificate, and any certificates in the chain of trust.

Note: Order IS IMPORTANT. Pound requires that it “walk” the chain from the top of the PEM file to the Bottom, so the root trusted authority should be at the end of the file, and the server’s certificate should be at the “top” of the PEM file.

To create this file of key, server cert, and CA certificate we concatenate together our existing Puppet certificates. Here we assume your certificates are stored in /etc/puppet/ssl.

$ cat /etc/puppet/ssl/private_keys/host_key.pem >> \
  /etc/puppet/ssl/pound/host_key_and_cert_chain.pem

$ cat /etc/puppet/ssl/certs/host_cert.pem >> \
  /etc/puppet/ssl/pound/host_key_and_cert_chain.pem

$ cat /etc/puppet/ssl/ca/ca_crt.pem >> \
  /etc/puppet/ssl/pound/host_key_and_cert_chain.pem

Pound Configuration

Please note that for those following along in the Apache configuration, client SSL verification is optional there, as Apache can set the X-Client-Verify header dynamically depending on the success or failure of the SSL verification.

In pound, we don’t have dynamic variable expansion, so we instead configure two listener ports 8140 and 8140. 8140 requires a valid, verified certificate as it will ALWAYS set X-Client-Verify: SUCCESS to proxied connections. 8140 does not set the X-Client-Verify header.

##################################################
# pound.cfg for use with mongrel => puppetmasterd
#
# Jeff McCune <jeff@northstarlabs.net>
#

User            "puppet"
Group           "puppet"

## Set log level
LogLevel        2

## Fork to the background
Daemon 1

## Check back-end mongrel servers every 30 seconds.
Alive           30

# Client timeout in seconds.  Pound will kill the connection if it does
# not receive any data after this many seconds.
Client 300

# How long should Pound wait for a response from the back-end (in seconds).
TimeOut 180

# Authenticated clients connect to port 8140
ListenHTTPS
  Address 0.0.0.0
  Port    8140

  # THIS IS VERY IMPORTANT.  Without it, clients may forge
  # the X-Client-Verify: SUCCESS header.
  # I believe this is correct, but have not verified it.
  HeadRemove "X-SSL-.*"
  HeadRemove "X-Client-Verify.*"

  ####    Pound's SSL certificate, key file, and CA chain of trust.
  Cert    "/etc/puppet/ssl/pound/host_key_and_cert_chain.pem"

  # This is the list of CA's we ultimately trust.
  # If a certificate's chain of trust is ROOTED with a CA in this list,
  # The client connection is authorized.
  VerifyList "/etc/puppet/ssl/ca/ca_crt.pem"

  # This is the list of CA's we will permit to exist in the CHAIN
  # from client certificate to ROOT CA's listed in VerifyList.
  # This list is sent to the client, so they may pick the correct
  # certificate to submit.
  CAlist "/etc/puppet/ssl/ca/ca_crt.pem"

  ## THIS IS WHERE ALL YOUR SECURITY STEMS FROM.  TAKE CARE!
  # ClientCert 0|1|2|3 depth
  #   0 - don't ask (default),
  #   1 - ask,
  #   2 - demand, and require valid chain to a root CA listed in VerifyList
  #   3 - demand (I think) but do not verify
  #   depth is the depth of verification for a client certificate (up to 9).
  ClientCert 2 4

  # Revocation is important.
  CRLlist    "/etc/puppet/ssl/ca/ca_crl.pem"

  # Pound, by default is too strict with SSL ciphers.
  Ciphers    "SSLv2:-LOW:-EXPORT:RC4+RSA"

  # If all of the above SSL stuff checks out...  Then
  # NOTE: I believe there is a bug in Pound where only one
  # header may be added.
  AddHeader "X-Client-Verify: SUCCESS"

  # Forward the connection back to the Mongrel servers.
  Service
    BackEnd
      Address 127.0.0.1
      Port    18140
    End
    BackEnd
      Address 127.0.0.1
      Port    18141
    End
    # etc...
    Session
      Type IP
      TTL  300
    End
  End
End

## Listen on port 8141 for CA signing requests.
ListenHTTPS
  Address    0.0.0.0
  Port       8141
  HeadRemove "X-SSL-.*"
  HeadRemove "X-Client-Verify.*"
  Cert    "/etc/puppet/ssl/pound/host_key_and_cert_chain.pem"
  VerifyList "/etc/puppet/ssl/ca/ca_crt.pem"
  CAlist "/etc/puppet/ssl/ca/ca_crt.pem"
  ClientCert 3 9
  CRLlist    "/etc/puppet/ssl/ca/ca_crl.pem"
  Ciphers    "SSLv2:-LOW:-EXPORT:RC4+RSA"

  Service
    BackEnd
      Address 127.0.0.1
      Port    18140
    End
    BackEnd
      Address 127.0.0.1
      Port    18141
    End
    Session
      Type IP
      TTL  300
    End
  End
End

Starting Pound

Now we can start Pound:

# pound -f /etc/pound/pound.cfg -p /var/run/pound.pid

Debugging Pound

Pound logs to syslog – on Debian this defaults to logging to /var/log/daemon.log. If you use syslog-ng you can direct your logging output elsewhere using a filter like:

destination df_pound { file("/var/log/pound.log"); };
filter f_pound { program("pound"); };
log {
        source(s_all);
        filter(f_pound);
        destination(df_pound);
};

Start puppetmasterd daemons

You need to run the puppetmasterd daemon using the Mongrel web server and run each instance on the port numbers indicated in our BackEnd clauses defined in our pound.cfg. You can use a script like this to start each instance:

#!/bin/bash
# name: mongrel_puppetmasterd
# Start a Puppet Master Server instance.

if ! [[ "$1" -gt 0 ]]; then
echo "ERROR: You must provide a port to run this puppet master on."
echo "Ensure your apache load balancer is configured to talk to these servers"
exit 1
fi

MASTERPORT="$1"
shift

puppetmasterd \
--pidfile=/var/run/puppetmasterd."${MASTERPORT}".pid \
--servertype=mongrel \
--ssl_client_header=HTTP_X_SSL_SUBJECT \
--masterport="${MASTERPORT}" \
$*

The script is initiated by passing the port number you wish to start the puppetmasterd on. To start puppetmasterd daemons for the Back End? instances we’ve defined in our pound.cfg you would:

# mongrel_puppetmasterd 18140
# mongrel_puppetmasterd 18141

Test clients

Now attempt to test a client connection.