The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

« Previous - Version 3/16 (diff) - Next » - Current version
James Turnbull, 03/14/2010 07:59 am


Using Stored Configurations

Puppet uses a technique called stored configuration, enabled with the storeconfigs configuration option, to store data in a database. This technique makes use of the Ruby on Rails framework and one of three database back-ends: SQLite (the default), MySQL or PostgreSQL. Additionally, a message queue can be used to improve a puppetmaster’s client response time. Currently stored configurations are principally used by exported resources.

It is important to note that once a node is in the stored configuration database it will not be automatically purged – even if you remove the corresponding node from your configuration. If you wish to purge nodes from your database you will need to do this manually. There is a script in the ext directory that purges nodes from stored configuration.

Installing Prerequisites

To make use of stored configuration you will need to install some prerequisites principally the database back-end you wish to use and the Rails framework.

Rails

Puppet 0.24.8 and older

Rails version 2.2.2 or earlier.

Puppet 0.25

Rails version 2.2.2 or greater.

Installation

We start by installing the rails Ruby Gem, either with the distribution’s package management (e.g. Debian/Ubuntu have a ‘rails’ package) or as a ruby gem.

To ensure a specific version of rails (not the latest available from RubyForge) specify the optional ‘-v’ parameter with the version number you wish to install:

gem install rails -v 2.2.2

MySQL

For MySQL you will need to install a number of packages. For Red Hat/Fedora the typical packages required would be:

mysql
mysql-devel
mysql-server

Or for Debian/Ubuntu you would need the following packages. libmysql-ruby provides the mysql gem, so installing via gems is unneccesary:

mysql-server
libmysql-ruby

You may also need to install the mysql Ruby Gem (not required on Debian/Ubuntu). You may need to specify the —with-mysql-config option to tell Gems where to find your MySQL libraries and headers:

# gem install mysql -- --with-mysql-config=/usr/bin/mysql_config

You may require the ruby-devel package in order to install this gem.

A word of warning – it is possible to be using stored configurations with MySQL without installing the mysql gem. This is because rails includes with it a simple Ruby based MySQL connector. Relying on this connector is strongly discouraged, as it has been observed to leak filehandles, leave open network and database connections, and eventually cause puppetmasterd to become completely unresponsive.

In order to check if you are using the unreliable Ruby based connector, examine the rails.log file produced by puppetmasterd. If you are using the Ruby one, you will find a message like this:

WARNING: You're using the Ruby-based MySQL library that ships with Rails.
This library is not suited for production. Please install the C-based MySQL
library instead (gem install mysql).

If you find that warning, make sure you install the mysql gem, which is based on the native MySQL libraries, before you begin using stored configurations in production.

SQLite3

For SQLite you will need to install the database package. For Red Hat/Fedora the package required would be:

sqlite

For Ubuntu/Debian, the packages required are (libsqlite3-ruby contains the appropriate gem):

sqlite3
libsqlite3-ruby

You may also need to install the sqlite3-ruby Ruby Gem (not required on Debian/Ubuntu):

# gem install sqlite3-ruby

PostgreSQL

For PostgreSQL you will need to install a number of packages. For Red Hat/Fedora the typical packages required would be:

postgresql
postgresql-server
postgresql-devel

You will also need to install the postgres Ruby Gem:

# gem install postgres

If you fail to install the postgres gem then you will receive the following error:

err: Could not retrieve configuration: Uncaught exception no such file to load -- postgres in method puppetmaster.getconfig

For PostgreSQL you will also need to create the database and create an appropriate role and password.

Configuring basic storeconfigs

Now we need to configure stored configuration in Puppet itself. There are three types of database adapters available for us, the default sqlite3, MySQL and PostgreSQL. For the latter back-end databases we also have available a number of options that let us specify a particular user, password and the location of a database socket. Let’s walk through the steps required to configure the most basic SQLite backend.

  • Configure puppet.conf:

    [puppetmasterd]
    storeconfigs = true
    dbadapter = sqlite3
    dblocation = /var/lib/puppet/storeconfigs.sqlite
    
  • “Thin” stored configuration (optional) – from 0.25.0 we can also use the “thin” stored configuration option. This only collects and stores to the database exported resources, tags and host facts. To use “thin” stored configuration configure it in your puppet.conf:

    [puppetmasterd]
    storeconfigs = true
    thin_storedconfigs = true
    dbadapter = sqlite3
    dblocation = /var/lib/puppet/storeconfigs.sqlite
    
  • Start the puppetmasterd daemon

  • Start the puppetd client

Now storeconfigs should be enabled and running.

Note that if you intend to use PuppetShow, you may need to include a dblocation setting under puppetmasterd, pointing to the default location. Also note that SQLite is not very happy with high volume usage (more than 5 or 10 hosts), and will report locking failures during operation.

Configuring storeconfigs for MySQL

  • Create a puppet database, and grant privileges. Replace password below with an appropriate password:

    # mysql -u root -p
    mysql> create database puppet;
    mysql> grant all privileges on puppet.* to puppet@localhost identified by 'password';
    
  • Configure your [puppetmasterd] section to reflect these settings:

    [puppetmasterd]
    storeconfigs = true
    dbadapter = mysql
    dbuser = puppet
    dbpassword = password
    dbserver = localhost
    dbsocket = /var/run/mysqld/mysqld.sock
    
  • Start the puppetmasterd daemon

  • Start the puppetd client

You can also find some information on [[My_Sql_Stored_Configuration_Patterns|Recipes/MySQLStored Configuration]] .

To optimize some often run Puppet queries on your MySQL database, use the following index:

create index exported_restype_title on resources (exported, restype, title(50));

Configuring storeconfigs for PostgreSQL

By default (on RHEL/CentOS, other distros unknown), you will have to su – to the postgres user to do any administrative work such as database creation.

  • Set up the database:

    # su - postgres
    $ psql template1
    template1=# create database puppet;
    CREATE DATABASE
    create user puppet with unencrypted password 'password';
    CREATE ROLE
    template1=# grant create on database puppet to puppet;
    

Note that Puppet requires the CREATE right to do automatic table creation to initialize the database.

  • Configure your [puppetmasterd] section to reflect these settings:

    [puppetmasterd]
    storeconfigs = true
    dbadapter = postgresql
    dbuser = puppet
    dbpassword = password
    dbserver = localhost
    dbname = puppet
    
  • Start the puppetmasterd daemon

  • Start the puppetd client

  • Check /var/log/messages (or wherever you have Puppet set to log) for the message for an ‘'Initialized database’‘ message when the first client checks in.

Installing Queuing Support for Storeconfigs

Puppet queueing is a feature which is designed to take some load off of the Puppet master by transferring the task of updating the database to a separate program which is named puppetqd (the Puppet Queue Daemon).

To add support for queuing to an existing storeconfigs configuration, you only need to install a queue broker, configure your Puppet master, and configure your puppetqd.

While many message queue services exist, currently Puppet only supports the Stomp protocol. We have tested using StompServer and ActiveMQ. While StompServer is easy to set up and configure, we suggest you use ActiveMQ — if you need queueing, you need it because of scalability, and StompServer simply isn’t built for scaling up to enterprise needs.

Queuing is available from release 0.25.0 onwards. It is NOT available in the 0.24.x releases.

Installing StompServer

First make sure you have the Ruby development package. On a Red Hat-based system, this would be:

yum install ruby-devel

Or on a Debian or Ubuntu-based system:

apt-get install ruby1.8-dev

StompServer is available as a gem, so you can install via:

gem install stompserver

Installing ActiveMQ

Download and install ActiveMQ: e.g.,:

$ wget http://www.apache.org/dyn/closer.cgi/activemq/apache-activemq/5.2.0/apache-activemq-5.2.0-bin.tar.gz
$ tar zxvf apache-activemq-5.2.0-bin.tar.gz

Configuring queuing storeconfigs

Configuring the queuing broker

Configuring StompServer

Configuration can either be done on the command line or via a config file. For example, the following command-line:

stompserver -q dbm

will start the stompserver on the default port of 61613 and use a DBM file as its backing store (i.e., while messages have not yet been picked up by puppetqd, they will be stored in a DBM file).

The configuration file:

$ cat /etc/puppet/puppet-queue.conf
:daemon: true
:working_dir: /tmp/stompserver
:storage: .queue
:queue: file
:auth: false
:debug: false
:group:
:user:
:host: 127.0.0.1
:port: 61613

configures the stompserver to use a file backing store.

Then run the stompserver:

stompserver -C /etc/puppet/puppet-queue.conf

Consult the stompserver’s README.txt for additional information on installation and configuration.

Configuring ActiveMQ

Create an activemq.xml file (based on the samples provided in the source), and enable stomp. The sample configuration uses the default backing store, AMQ, (which is a transactional journal in ActiveMQ 5) and only requires the one-line change below to enable stomp on port 61613:

<transportConnectors> <transportConnector name="stomp" uri="stomp://localhost:61613"/> </transportConnectors>

You can then start ActiveMQ:

activemq

You should consult the ActiveMQ documentation for additional information on configuring ActiveMQ.

Configuring Puppet

You need to configure both puppetmasterd and puppetqd to use queuing.

In this example, we’re keeping our stored configuration information in sqlite3 — if you’re using PostgreSQL, MySQL, or others, your dbadapter and dblocation information will differ).

Please refer to the basic stored configuration section on the wiki for details about creating the database for MySQL or PostgreSQL.

First we need to install the Stomp client. You can do this from a gem:

gem install stomp

Next, we need to configure Puppet itself. Note that the configuration of the Puppet master and the puppetqd is the same: this is because they both need to read information out of the same database (i.e., the database is a cache). Thus, a puppet.conf would contain:

[main]
  queue_type = stomp
  queue_source = stomp://localhost:61613
  dbadapter = sqlite3
  dblocation = /var/lib/puppet/storeconfigs.sqlite
[puppetmasterd]
  async_storeconfigs = true

Note that these can all be put onto a command line:

puppetmasterd --async_storeconfigs --queue_type=stomp --queue_source="stomp://localhost:61613" \
  --dbadapter sqlite3 --dblocation /var/lib/puppet/storeconfigs.sqlite

Also note that the dblocation is not needed for MySQL or PostgreSQL dbadapters.

You then need to run the puppetqd daemon.

puppetqd

Bootstrapping into stored configuration

If you have a lot of hosts, turning on stored configuration might crush your database server. The main problem is that data for every host needs to be INSERTed into the database. Once you get past that hurdle there’s much less SQL activity. Here are some notes about how we (usg.edu folks) got over the hurdle. The main bottleneck for us was CPU—IO wait wasn’t an issue.

First we tuned our MySQL server (all based on masterzen’s recommendations). We added these lines to the [mysqld] section of my.cnf:

innodb_buffer_pool_size=2G
innodb_log_file_size=256M
innodb_log_buffer_size=64M
innodb_additional_mem_pool_size=20M
innodb_flush_method = O_DIRECT

Next we stopped puppetd on half of our \~400 servers. This, plus the MySQL changes above, allowed our server to survive the initial stored configuration hit of our other 200 servers. After the first 200 were in stored configuration we ran puppetd —test —tags SMALLMODULE on the second half of our hosts. Later we restarted puppetd.

Our clients check in once an hour. About 15 hosts check in per minute. The puppetmasterd and mysqld run on the same server (along with a few other applications). Our server has 8 CPUs and 12G of RAM.

Here’s a fine blog post by masterzen about this issue: http://www.masterzen.fr/2009/03/18/omg-storedconfigs-killed-my-database/