The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

« Previous - Version 31/49 (diff) - Next » - Current version
Spencer Krum, 07/29/2011 12:22 pm


Testing Modules in the Puppet Forge

We are evaluating two tools for testing modules. One is rspec-puppet, the other is cucumber-puppet. The hope is to get standards for testing into the module forge and hook it into a CI framework such as hudson or jenkins. This will enable module forge users to immediately evaluate a module in terms of its test failure rate, and enable developers to see the intended behavior of the module. The higher level goal is to open a discussion among module developers on the Puppet Forge about the intended behavior of modules, what they need to do, what they should and should not be doing.

Another huge goal of this project is to develop a way to test puppet without running puppet apply and looking to see what worked or didn’t. Both of the tools under consideration allow the developer/sysadmin to test the modules against puppet to see if the compile (equivalent to a puppet apply —noop) and then hold the catalog object so that we can see if the catalog is formed as intended.


Cucumber-puppet is a puppetification of the gherkin based Behavior Driven Development (BDD) framework.

Cucumber has the advantage that it is a more mature codebase, is faster to start with, and uses the human-readable gherkin language for its tests. One disadvantage for cucumber-puppet is that it seems hard to contain all of the tests within a cucumber feature. Many of the tests we have written in cucumber puppet involved creating dummy nodes in a site.pp to write cucumber tests against. Cucumber-puppet also doesn’t seem to be compatible with Puppet 2.7.x, at least not yet.


Rspec-puppet is a puppetification of the rspec BDD framework.

Rspec-puppet has the advantage that it is brand new and actively developed, puppet already uses a lot of rspec to do testing, and since the gherkin layer is removed, can be coded entirely in ruby. Also, in contrast to cucumber-puppet, rspec-puppet tests can be entirely contained within the spec test file. And for compatibility, rspec-puppet works with Puppet 2.7.x through 2.7.2rc2 (although 2.7.x seems to handle catalog errors slightly differently right now).

Quick Getting started with our examples

We have taken the puppetlabs-apt module and written pretty comprehensive tests for it in both cucumber-puppet and rspec-puppet. In this example we will set up the cucumber-puppet and rspec-puppet tools and run both suites on the puppet-apt module. Note that for this example we will be using the nibalizer: puppet-apt and mlitteken: cucumber-puppet, rspec-puppet forks. For this example I have begun with a stock Ubuntu VM using the wonderful tool vagrant.

Setup

Start with the following packages(and their dependencies) installed via apt: git-core, rubygems1.8, ruby, ruby1.8-dev, libopenssl-ruby1.8

Start with the following packages(and their dependencies) installed via rubygems: gherkin, cucumber, gem-man, templater, rspec, puppet (version 2.6.9)

Get the testing tools

$ mkdir src
$ cd src
$ git clone -b example https://github.com/mlitteken/rspec-puppet.git
$ git clone -b example https://github.com/mlitteken/cucumber-puppet.git
$ ls
cucumber-puppet  rspec-puppet
$ echo $RUBYLIB

$ echo "export RUBYLIB=~/src/cucumber-puppet/lib:~/src/rspec-puppet/lib:${RUBYLIB}" >> ~/.bashrc
$ echo "export RUBYPATH=~/src/cucumber-puppet/bin:${RUBYPATH}" >> ~/.bashrc
$ echo "export PATH=${PATH}:~/src/cucumber-puppet/bin" >> ~/.bashrc
$ source ~/.bashrc

Create the file structure:

$ cd
$ mkdir puppetforge/{,manifests,modules}

Clone the apt module:

$ cd puppetforge/modules
$ git clone -b example https://github.com/nibalizer/puppet-apt apt
$ cd apt

Run cucumber-puppet

$ cd ~/puppetforge/modules/apt
$ cucumber-puppet features/modules/apt/apt_*.feature

If all went well you should have seen a ton of green fly by and a summary at the end:

27 scenarios (27 passed)
258 steps (258 passed)
0m4.228s

Cucumber and cucumber-puppet use green to indicate success, red to indicate test failure, and other colors to report information about the test run.

Run rspec-puppet

$ cd ~/puppetforge/modules/apt
$ rspec --color --format documentation spec/{classes,defines}/apt*_spec.rb

If all went well you should have seen a ton of green fly by and a summary at the end:

Finished in 8.88 seconds
119 examples, 0 failures

rspec and rspec-puppet use green to indicate success, red to indicate test failure, and other colors to report information about the test run.

Cucumber-puppet mini tutorial on use

What are we testing?

The apt module has within it a manifests directory and puppet manifests under that. The point of this very quick example is to test the apt class to see if it will create a file resource “sources.list”.

Create a feature

Cucumber-puppet runs on files called testname.features, we have prepended the modulename to these tests to unify our naming scheme between cucumber-puppet and rspec-puppet. We have tried to name the test files after the name of the file in the manifests directory they are testing. Feature files go in features/modules/apt/.

Lets create a file, our file will be named apt_example_new.feature because we have already defined an apt_init.feature. Absolute path, for clarity, is: ~/puppetforge/apt/features/modules/apt/apt_example.new.feature

```gherkin Feature: example init.pp test In order to have an effective apt module The apt class needs to have certain files And must run some execs

Scenario: Basic class, sanity check
Given a node of class "apt"
When I compile the catalog
Then compilation should succeed
And all resource dependencies should resolve

```

Rspec-puppet mini tutorial on use