The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Using Mongrel with Apache

Alternatively, see:

Puppet daemons default to using WEBrick for http serving, but puppetmasterd can be used with Mongrel instead. This requires setting up a web proxy in front of Mongrel to handle the SSL connections, and then you can start as many puppetmasterd instances as you need. Note that Puppet uses Mongrel differently than most — usually, you would have a mongrel_cluster command that managed multiple Mongrel instances. I planned to implement it this way, but it ended up being easier in the short term to stick with puppetmasterd. Please contact us if you want to work on getting mongrel_cluster to work with Puppet — but we generally recommend folks use Passenger or Unicorn instead these days…

Mongrel will work with any modern version of puppet; 0.23.1 is the minimum version with appropriate support. (Earlier versions couldn’t do certificate signing because they didn’t support the X-Client-Verify header.)

Mongrel 1.1.5 or later is strongly recommended, and while 0.3.15 will theoretically work we don’t test older versions of Mongrel extensively.

This document only describes setting up Apache as a proxy in front of Mongrel. Any other http proxy should work, as long as it supports validating client certificates.

Why You’d Do This

Mongrel scales much better than WEBrick, at least partially because it allows you to run multiple processes serving the same pool of clients on the same host. WEBrick only uses Ruby’s threading, which does not scale beyond one processor, and it appears that WEBrick starts dropping connections beyond about 2 concurrent connections.

If you’re getting connection-reset or End-of-file errors, you should try Mongrel. As more people try it and it proves to be stable, it will eventually become the preferred serving platform for the master.

Known Issues

We recommend Mongrel 1.1.5 or later, although it will theoretically work as far back as 0.3.15 or maybe 0.3.14. We have not extensively tested between those versions, however, and would strongly encourage you to upgrade.

SLES 10 ships with Mongrel 0.3.13, which is too old to work out of the box.

The Puppet file server (and other parts of puppet?) will not see connections coming from the puppet client but from the reverse proxy (ie. Apache). FileServingConfiguration (Security section) gives more detail.

There is a problem with Apache and the certificate revocation list (CRL) as generated by the puppet CA. If you encounter this problem, you are likely to see lines like this in your error log:

[Mon Nov 10 08:47:20 2008] [warn] Invalid signature on CRL
[Mon Nov 10 08:47:20 2008] [error] Certificate Verification: Error (8): CRL signature failure

Puppet will also complain:

err: Could not retrieve catalog: Certificates were not trusted: tlsv1 alert decrypt error

Yet everything looks good in your puppet.conf.

If so, you will need to create a CRL directory and point your Apache server to this path by changing

SSLCARevocationFile     /Library/Puppet/Generated/Server/SSL/ca/ca_crl.pem

To

SSLCARevocationPath     /Library/Puppet/Generated/Server/SSL/CRL_DIRECTORY

To create this directory, you should take the following steps. For this example, we will assume that the directory will be created in the standard SSL directory that puppet uses.

$mkdir /Library/Puppet/Generated/Server/SSL/crl
$cd /Library/Puppet/Generated/Server/SSL/crl
$ln -s /Library/Puppet/Generated/Server/SSL/ca/ca_crl.pem `openssl crl -in /Library/Puppet/Generated/Server/SSL/ca/ca_crl.pem -hash -noout`.0

Then, restart your Apache server.

If, for some reason, you are still getting the old error, you will need to disable CRL checking by commenting out all lines starting with ‘SSLCARevocation’.

Apache Configuration

Here is a complete apache configuration, intended for use only with puppet:

# Jeff McCune <mccune@math.ohio-state.edu>
# 2007-09-14
#
# Minimal Apache Configuration for Apache+Mongrel+Puppetmaster
#
#  Host System Setup and Configuration:
#  - Add puppet/puppet user/group
#  - Use the following configuration file.
#  - /Library/Puppet/Generated/Server is owned by puppet/puppet
#
#  - If you have a system which doens't provide Apache 2.2.X packages:
#    Apache may be built with:
#
#    ./configure --enable-so \
#       --enable-ssl=shared --enable-proxy=shared --enable-proxy_http=shared \
#       --enable-proxy_balancer=shared --enable-headers=shared \
#       --enable-authz_host=shared --enable-log_config=shared \
#       --prefix=/Library/Puppet/Resources/httpd-2.2.4
#
# - Finally, start the SSL proxy with:
#   /Library/Puppet/Resources/httpd-2.2.4/bin/httpd -f \
#   /Library/Puppet/Versioned/Server/cluster-orange/httpd/puppetmasterd.conf

Listen 8140
PidFile /Library/Puppet/Generated/Server/balancer.pid
User puppet
Group puppet

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule headers_module modules/mod_headers.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule authz_host_module modules/mod_authz_host.so
LoadModule log_config_module modules/mod_log_config.so

<Directory />
    Options FollowSymLinks
    AllowOverride None
    Order deny,allow
    Deny from all
</Directory>

<Proxy balancer://puppetmaster>
  BalancerMember http://127.0.0.1:18140 keepalive=on max=2 retry=30
  BalancerMember http://127.0.0.1:18141 keepalive=on max=2 retry=30
</Proxy>

<VirtualHost *:8140>
    SSLEngine on
    SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
    SSLCertificateFile      /Library/Puppet/Generated/Server/SSL/host_cert.pem
    SSLCertificateKeyFile   /Library/Puppet/Generated/Server/SSL/host_key.pem
    SSLCertificateChainFile /Library/Puppet/Generated/Server/SSL/ca/ca_crt.pem
    SSLCACertificateFile    /Library/Puppet/Generated/Server/SSL/ca/ca_crt.pem
    # Using the technique from above.
    SSLCARevocationPath     /Library/Puppet/Generated/Server/SSL/crl
    SSLVerifyClient require
    SSLVerifyDepth  1
    SSLOptions +StdEnvVars

    # The following client headers allow the same configuration to work with Pound.
    RequestHeader set X-SSL-Subject %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    <Location />
        SetHandler balancer-manager
        Order allow,deny
        Allow from all
    </Location>

    ProxyPass / balancer://puppetmaster:8140/ timeout=180
    ProxyPassReverse / balancer://puppetmaster:8140/
    ProxyPreserveHost on
    SetEnv force-proxy-request-1.0 1
    SetEnv proxy-nokeepalive 1

    ErrorLog  /Library/Puppet/Generated/Server/balancer_error.log
    CustomLog /Library/Puppet/Generated/Server/balancer_access.log combined
    CustomLog /Library/Puppet/Generated/Server/balancer_ssl_request.log \
                  "%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>

Note that you’ll have to customize it to match the name of your puppet server. Note also the cluster member declarations — this is where you tell Apache how to contact the Mongrel instances — you need a separate line for every instance, and (of course) each instance needs to be started on a separate port.

This configuration starts Apache on Puppet’s normal server port, so clients do not need to be reconfigured.

Another Apache configuration

Here’s another happy apache configuration, without all the OSXey stuff (‘’‘Note:’‘’ You still need to load the apache modules from the configuration above. Except for ‘'log_config’‘ which seems to be OS X specific):

Listen 8140

ProxyRequests Off

<Proxy balancer://puppetmaster>
    BalancerMember http://127.0.0.1:18140
</Proxy>

<VirtualHost *:8140>
    SSLEngine on
    SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA
    SSLCertificateFile /etc/puppet/ssl/certs/backup4.foo.com.pem
    SSLCertificateKeyFile /etc/puppet/ssl/private_keys/backup4.foo.com.pem
    SSLCertificateChainFile /etc/puppet/ssl/ca/ca_crt.pem
    SSLCACertificateFile /etc/puppet/ssl/ca/ca_crt.pem
    # Using the technique from above.
    SSLCARevocationPath     /Library/Puppet/Generated/Server/SSL/crl
    SSLVerifyClient require
    SSLVerifyClient require
    SSLVerifyDepth  1
    SSLOptions +StdEnvVars

    RequestHeader set X-Client-DN %{SSL_CLIENT_S_DN}e
    RequestHeader set X-Client-Verify %{SSL_CLIENT_VERIFY}e

    <Location />
       SetHandler balancer-manager
       Order allow,deny
       Allow from all
    </Location>

    ProxyPass / balancer://puppetmaster:8140/
    ProxyPassReverse / balancer://puppetmaster:8140/
    ProxyPreserveHost on

</VirtualHost>

Configuring Puppet

The only change to Puppet is that you need to start puppetmasterd with —servertype mongrel and choose a separate masterport for each instance. You will also need to specify a separate pidfile for each instance.

The attached example init script for RedHat systems, or this helper script might be useful:

#!/bin/bash
#
# Start a Puppet Master Server instance.
EZPUPPET_CONFDIR="${EZPUPPET_CONFDIR:=${EZPUPPET_BASE:=/Library/Puppet}/Versioned/Server/${EZPUPPET_SERVER:=cluster-orange}/puppet}"

if ! [[ "$1" -gt 0 ]]; then
  echo "ERROR: You must provide a port to run this puppet master on."
  echo "Start at 18140 and increment upward."
  echo "  The apache load balancer is configured to talk to these servers"
  echo "  so the port numbers are NOT arbitrary."
  exit 1
fi

MASTERPORT="$1"
shift

puppetmasterd \
  --confdir="${EZPUPPET_CONFDIR}" \
  --pidfile=${EZPUPPET_BASE}/Generated/Server/var/run/puppetmaster."${MASTERPORT}".pid \
  --servertype=mongrel \
  --masterport="${MASTERPORT}" \
  $*

As mentioned earlier, this would be easier if it were done with mongrel_start, but it will take a significant amount of work to get all of Puppet’s existing code working in that framework, and I’m unlikely to do it except as part of contract development. Clearly, patches that provided this would be accepted.

‘’‘NOTE’‘’: The current (as of at least 0.24.6) /etc/init.d/puppetmaster script on RedHat and compatible systems already supports multiple mongrel instances in a similar way to above. Simply specify your Mongrel instance ports in /etc/sysconfig/puppetmaster in BASH array format as per the following example:

PUPPETMASTER_PORTS=([0]=18140 [1]=18141 [2]=18142 [3]=18143 [4]=18144)

Generating Certificates

In version 0.23.2 I found that puppetmasterd would not generate certificates when started using Mongrel. I worked around the problem by using puppetca to generate the certificates as part of the install process, like this:

puppetca --generate <DOMAIN>

Client certificate verification

With the apache option:

SSLVerifyClient optional|require

you control how Apache wil check the client certification. You can either, set the different certificate request path to not require verification, or disable it (unless you have another ca server or another way of distributing the certificates). So:

  • by enabling require, you basically say – the client is already suppose to have a certificate that I could verify. If certificate is not signed in puppetmaster you will get the error:

    err: Could not request certificate: SSL_connect returned=1 errno=0 state=SSLv3 read finished A: sslv3 alert handshake failure

  • “optional” will simulate the behaviour of an default standalone puppetmaster (without apache).