Using Stored Configurations¶

Puppet uses a technique called stored configuration, enabled with the storeconfigs configuration option, to store data in a database. This technique makes use of the Ruby on Rails framework and one of three database back-ends: SQLite (the default; not suitable for production use), MySQL or PostgreSQL. Additionally, a message queue can be used to improve a puppetmaster’s client response time. Currently stored configurations are principally used by exported resources.

It is important to note that once a node is in the stored configuration database it will not be automatically purged – even if you remove the corresponding node from your configuration. If you wish to purge nodes from your database you will need to do this manually. There is a script in the ext directory that purges nodes from stored configuration.

Installing Prerequisites¶

To make use of stored configuration you will need to install some prerequisites principally the database back-end you wish to use and the Rails framework.

Rails¶

Puppet 0.24.8 and older

Rails version 2.2.2 or earlier.

Puppet 0.25 and later

Rails version 2.2.2 or greater.

Warning

Jeff McCune jeff@puppetlabs.com 2010-10-01

With Puppet version 0.25.5 and MySQL on Enterprise Linux 5 you must update the Active Record gem to version 2.3.5 in order to avoid the error:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: Mysql::
Error: MySQL server has gone away: SELECT * FROM `hosts` WHERE (`hosts`.`name` =
 'foo.puppetlabs.lan')  LIMIT 1
warning: Not using cache on failed catalog
err: Could not retrieve catalog; skipping run

In order to update the gem to this specific version, it is possible to install the gem into the puppet home directory:

sudo -H -E -u puppet -s
bash-3.2$ gem install activerecord -v 2.3.5
WARNING:  Installing to ~/.gem since /usr/lib/ruby/gems/1.8 and
          /usr/bin aren't both writable.
WARNING:  You don't have /var/lib/puppet/.gem/ruby/1.8/bin in your PATH,
          gem executables will not run.
Successfully installed activesupport-2.3.5
Successfully installed activerecord-2.3.5
2 gems installed
Installing ri documentation for activesupport-2.3.5...
Installing ri documentation for activerecord-2.3.5...
Installing RDoc documentation for activesupport-2.3.5...
Installing RDoc documentation for activerecord-2.3.5...
bash-3.2$ gem list

*** LOCAL GEMS ***

actionmailer (2.2.2)
actionpack (2.2.2)
activerecord (2.3.5, 2.2.3, 2.2.2)
activeresource (2.2.2)
activesupport (2.3.5, 2.2.3, 2.2.2)
fastthread (1.0.7)
passenger (2.2.11)
rack (1.0.1)
rails (2.2.2)
rake (0.8.7)
sqlite3-ruby (1.2.4)

Reference: http://projects.puppetlabs.com/issues/4694

Installation

We start by installing the rails Ruby Gem, either with the distribution’s package management (e.g. Debian/Ubuntu have a ‘rails’ package) or as a ruby gem.

To ensure a specific version of rails (not the latest available from RubyForge) specify the optional ‘-v’ parameter with the version number you wish to install:

gem install rails -v 2.2.2

MySQL¶

For MySQL you will need to install a number of packages. For Red Hat/Fedora the typical packages required would be:

mysql
mysql-devel
mysql-server

Or for Debian/Ubuntu you would need the following packages. libmysql-ruby provides the mysql gem, so installing via gems is unneccesary:

mysql-server
libmysql-ruby

You may also need to install the mysql Ruby Gem (not required on Debian/Ubuntu). You may need to specify the —with-mysql-config option to tell Gems where to find your MySQL libraries and headers:

# gem install mysql -- --with-mysql-config=/usr/bin/mysql_config

You may require the ruby-devel package in order to install this gem.

A word of warning – it is possible to be using stored configurations with MySQL without installing the mysql gem. This is because rails includes with it a simple Ruby based MySQL connector. Relying on this connector is strongly discouraged, as it has been observed to leak filehandles, leave open network and database connections, and eventually cause puppetmasterd to become completely unresponsive.

In order to check if you are using the unreliable Ruby based connector, examine the rails.log file produced by puppetmasterd. If you are using the Ruby one, you will find a message like this:

WARNING: You're using the Ruby-based MySQL library that ships with Rails.
This library is not suited for production. Please install the C-based MySQL
library instead (gem install mysql).

If you find that warning, make sure you install the mysql gem, which is based on the native MySQL libraries, before you begin using stored configurations in production.

SQLite3¶

For SQLite you will need to install the database package. For Red Hat/Fedora the package required would be:

sqlite

For Ubuntu/Debian, the packages required are (libsqlite3-ruby contains the appropriate gem):

sqlite3
libsqlite3-ruby

You may also need to install the sqlite3-ruby Ruby Gem (not required on Debian/Ubuntu):

# gem install sqlite3-ruby

PostgreSQL¶

For PostgreSQL you will need to install a number of packages. For Red Hat/Fedora the typical packages required would be:

postgresql
postgresql-server
postgresql-devel

You will also need to install the postgres Ruby Gem:

# gem install postgres

If you fail to install the postgres gem then you will receive the following error:

err: Could not retrieve configuration: Uncaught exception no such file to load -- postgres in method puppetmaster.getconfig

For PostgreSQL you will also need to create the database and create an appropriate role and password.

Configuring basic storeconfigs¶

Now we need to configure stored configuration in Puppet itself. There are three types of database adapters available for us, the default sqlite3, MySQL and PostgreSQL. For the latter back-end databases we also have available a number of options that let us specify a particular user, password and the location of a database socket.

Let’s begin by walking through the steps to configure the SQLite backend. Note that this backend is not suitable for production-level use — it can’t handle high volume traffic, and will report locking failures if used with more than five or ten hosts.

• Configure puppet.conf:

  [puppetmasterd]
  storeconfigs = true
  dbadapter = sqlite3
  

• “Thin” stored configuration (optional) – from 0.25.0 we can also use the “thin” stored configuration option. This only collects and stores to the database exported resources, tags and host facts. To use “thin” stored configuration configure it in your puppet.conf:

  [puppetmasterd]
  storeconfigs = true
  thin_storeconfigs = true
  dbadapter = sqlite3
  

• Start the puppetmasterd daemon

• Start the puppetd client

Now storeconfigs should be enabled and running.

Note that if you intend to use PuppetShow, you may need to include a dblocation setting under puppetmasterd, pointing to the default location.

Configuring storeconfigs for MySQL¶

• Create a puppet database, and grant privileges. Replace password below with an appropriate password:

  # mysql -u root -p
  mysql> create database puppet;
  mysql> grant all privileges on puppet.* to puppet@localhost identified by 'password';
  

• Configure your [puppetmasterd] section to reflect these settings:

  [puppetmasterd]
  storeconfigs = true
  dbadapter = mysql
  dbuser = puppet
  dbpassword = password
  dbserver = localhost
  dbsocket = /var/run/mysqld/mysqld.sock
  

• Start the puppetmasterd daemon

• Start the puppetd client

You can also find some information on [[My_Sql_Stored_Configuration_Patterns|Recipes/MySQLStored Configuration]] .

To optimize some often run Puppet queries on your MySQL database, use the following index:

create index exported_restype_title on resources (exported, restype, title(50));

Configuring storeconfigs for PostgreSQL¶

By default (on RHEL/CentOS, other distros unknown), you will have to su – to the postgres user to do any administrative work such as database creation.

• Set up the database:

  # su - postgres
  $ psql template1
  template1=# create database puppet;
  CREATE DATABASE
  create user puppet with unencrypted password 'password';
  CREATE ROLE
  template1=# grant create on database puppet to puppet;
  

Note that Puppet requires the CREATE right to do automatic table creation to initialize the database.

• Configure your [puppetmasterd] section to reflect these settings:

  [puppetmasterd]
  storeconfigs = true
  dbadapter = postgresql
  dbuser = puppet
  dbpassword = password
  dbserver = localhost
  dbname = puppet
  

• Start the puppetmasterd daemon

• Start the puppetd client

• Check /var/log/messages (or wherever you have Puppet set to log) for the message for an ‘'Initialized database’‘ message when the first client checks in.

Installing Queuing Support for Storeconfigs¶

Puppet queueing is a feature which is designed to take some load off of the Puppet master by transferring the task of updating the database to a separate program which is named puppetqd (the Puppet Queue Daemon).

To add support for queuing to an existing storeconfigs configuration, you only need to install a queue broker, configure your Puppet master, and configure your puppetqd.

While many message queue services exist, currently Puppet only supports the Stomp protocol. We have tested using StompServer and ActiveMQ. While StompServer is easy to set up and configure, we suggest you use ActiveMQ — if you need queueing, you need it because of scalability, and StompServer simply isn’t built for scaling up to enterprise needs.

Queuing is available from release 0.25.0 onwards. It is NOT available in the 0.24.x releases.

Installing StompServer¶

First make sure you have the Ruby development package. On a Red Hat-based system, this would be:

yum install ruby-devel

Or on a Debian or Ubuntu-based system:

apt-get install ruby1.8-dev

StompServer is available as a gem, so you can install via:

gem install stompserver

Installing ActiveMQ¶

Download and install ActiveMQ: e.g.,:

$ wget http://www.apache.org/dyn/closer.cgi/activemq/apache-activemq/5.2.0/apache-activemq-5.2.0-bin.tar.gz
$ tar zxvf apache-activemq-5.2.0-bin.tar.gz

Configuring queuing storeconfigs¶

Configuring the queuing broker¶

Configuring StompServer¶

Configuration can either be done on the command line or via a config file. For example, the following command-line:

stompserver -q dbm

will start the stompserver on the default port of 61613 and use a DBM file as its backing store (i.e., while messages have not yet been picked up by puppetqd, they will be stored in a DBM file).

The configuration file:

$ cat /etc/puppet/puppet-queue.conf
:daemon: true
:working_dir: /tmp/stompserver
:storage: .queue
:queue: file
:auth: false
:debug: false
:group:
:user:
:host: 127.0.0.1
:port: 61613

configures the stompserver to use a file backing store.

Then run the stompserver:

stompserver -C /etc/puppet/puppet-queue.conf

Consult the stompserver’s README.txt for additional information on installation and configuration.

Configuring ActiveMQ¶

Create an activemq.xml file (based on the samples provided in the source), and enable stomp. The sample configuration uses the default backing store, AMQ, (which is a transactional journal in ActiveMQ 5) and only requires the one-line change below to enable stomp on port 61613:

<transportConnectors> <transportConnector name="stomp" uri="stomp://localhost:61613"/> </transportConnectors>

You can then start ActiveMQ:

activemq

You should consult the ActiveMQ documentation for additional information on configuring ActiveMQ.

Configuring Puppet¶

You need to configure both puppetmasterd and puppetqd to use queuing.

In this example, we’re keeping our stored configuration information in sqlite3 — if you’re using PostgreSQL, MySQL, or others, your dbadapter and dblocation information will differ).

Please refer to the basic stored configuration section on the wiki for details about creating the database for MySQL or PostgreSQL.

First we need to install the Stomp client. You can do this from a gem:

gem install stomp

Next, we need to configure Puppet itself. Note that the configuration of the Puppet master and the puppetqd is the same: this is because they both need to read information out of the same database (i.e., the database is a cache). Thus, a puppet.conf would contain:

[main]
  queue_type = stomp
  queue_source = stomp://localhost:61613
  dbadapter = sqlite3
[puppetmasterd]
  async_storeconfigs = true

Note that these can all be put onto a command line:

puppetmasterd --async_storeconfigs --queue_type=stomp --queue_source="stomp://localhost:61613" \
  --dbadapter sqlite3

You then need to run the puppetqd daemon.

puppetqd

Bootstrapping into stored configuration¶

If you have a lot of hosts, turning on stored configuration might crush your database server. The main problem is that data for every host needs to be INSERTed into the database. Once you get past that hurdle there’s much less SQL activity. Here are some notes about how we (usg.edu folks) got over the hurdle. The main bottleneck for us was CPU—IO wait wasn’t an issue.

First we tuned our MySQL server (all based on masterzen’s recommendations). We added these lines to the [mysqld] section of my.cnf:

innodb_buffer_pool_size=2G
innodb_log_file_size=256M
innodb_log_buffer_size=64M
innodb_additional_mem_pool_size=20M
innodb_flush_method = O_DIRECT

Next we stopped puppetd on half of our ~400 servers. This, plus the MySQL changes above, allowed our server to survive the initial stored configuration hit of our other 200 servers. After the first 200 were in stored configuration we ran puppetd —test —tags SMALLMODULE on the second half of our hosts. Later we restarted puppetd.

Our clients check in once an hour. About 15 hosts check in per minute. The puppetmasterd and mysqld run on the same server (along with a few other applications). Our server has 8 CPUs and 12G of RAM.

Here’s a fine blog post by masterzen about this issue: http://www.masterzen.fr/2009/03/18/omg-storedconfigs-killed-my-database/

Cleaning Stored Configs¶

On occasion, something about a system changes and you will need to clean out the stored configuration database for a given node. This is fairly harmless to do because the information will repopulate the next time the node checks in with the master.

Download the cleaner script contained with the source:

https://github.com/puppetlabs/puppet/raw/master/ext/puppetstoredconfigclean.rb

Once you have the script in your puppet master, you can clean the information out of your stored configuration for a given node, but running the script like so:

./puppetstoredconfigclean.rb host1.example.com

This will remove all of the information about the node ‘host1.example.com’ from your stored configs database.

For more information on usage, you can run the script with —help like so:

./puppetstoredconfigclean.rb --help