History

blog

Testing Modules in the Puppet Forge

We are evaluating two tools for testing modules. One is rspec-puppet, the other is cucumber-puppet. The hope is to get standards for testing into the module forge and hook it into a CI framework such as hudson or jenkins. This will enable module forge users to immediately evaluate a module in terms of its test failure rate, and enable developers to see the intended behavior of the module. The higher level goal is to open a discussion among module developers on the Puppet Forge about the intended behavior of modules, what they need to do, what they should and should not be doing.

Another huge goal of this project is to develop a way to test puppet without running puppet apply and looking to see what worked or didn’t. Both of the tools under consideration allow the developer/sysadmin to test the modules against puppet to see if they compile (equivalent to a puppet apply —noop) and then hold the catalog object so that we can see if the catalog is formed as intended.

Cucumber-puppet is a puppetification of the gherkin-based Cucumber Behavior Driven Development (BDD) framework.

Cucumber has the advantage that it is a more mature codebase, is faster to start with, and uses the human-readable gherkin language for its tests. One disadvantage for cucumber-puppet is that it seems hard to contain all of the tests within a cucumber feature. Many of the tests we have written in cucumber-puppet involved creating dummy nodes in a site.pp to write cucumber tests against. Cucumber-puppet also doesn’t seem to be compatible with Puppet 2.7.x, at least not yet (this is no longer true: it was addressed in commit https://github.com/nistude/cucumber-puppet/commit/41cfdb588b7bf16c235c01241df36aa77e0e5f1b).

Rspec-puppet is a puppetification of the rspec BDD framework.

Rspec-puppet has the advantage that it is brand new and actively developed, puppet already uses a lot of rspec to do testing, and, since the gherkin layer is removed, it can be coded entirely in ruby. Also, in contrast to cucumber-puppet, rspec-puppet tests can be entirely contained within the spec test file. And for compatibility, rspec-puppet works with Puppet 2.7.x through 2.7.2rc2 (although the head of the 2.7.x branch seems to handle catalog errors slightly differently right now).

Quick Getting Started With Our Examples

We have taken the puppetlabs-apt module and written pretty comprehensive tests for it in both cucumber-puppet and rspec-puppet. In this example we will set up the cucumber-puppet and rspec-puppet tools and run both suites on the puppet-apt module. Note that for this example we will be using the nibalizer fork of puppet-apt, the mlitteken fork of cucumber-puppet and the puppetlabs fork of rspec-puppet. For this example I have begun with a stock Ubuntu VM using the wonderful tool Vagrant.

Setup

Start with the following packages (and their dependencies):

Installed via apt:

  • git-core
  • rubygems1.8
  • ruby
  • ruby1.8-dev
  • libopenssl-ruby1.8

Installed via rubygems:

  • gherkin
  • cucumber
  • gem-man
  • templater
  • rspec
  • puppet (version 2.6.9)

Get the testing tools:

$ mkdir src
$ cd src
$ git clone https://github.com/puppetlabs/rspec-puppet.git
$ git clone -b example https://github.com/mlitteken/cucumber-puppet.git
$ ls
cucumber-puppet  rspec-puppet
$ echo $RUBYLIB

$ echo "export RUBYLIB=~/src/cucumber-puppet/lib:~/src/rspec-puppet/lib:${RUBYLIB}" >> ~/.bashrc
$ echo "export RUBYPATH=~/src/cucumber-puppet/bin:${RUBYPATH}" >> ~/.bashrc
$ echo "export PATH=${PATH}:~/src/cucumber-puppet/bin" >> ~/.bashrc
$ source ~/.bashrc

Create the file structure:

$ cd
$ mkdir puppetforge/{,manifests,modules}

Clone the apt module:

$ cd puppetforge/modules
$ git clone -b example https://github.com/nibalizer/puppet-apt apt
$ cd apt

Run cucumber-puppet

$ cd ~/puppetforge/modules/apt
$ cucumber-puppet features/modules/apt/apt_*.feature

If all went well you should have seen a ton of green fly by and a summary at the end:

27 scenarios (27 passed)
258 steps (258 passed)
0m4.228s

Cucumber and cucumber-puppet use green to indicate success, red to indicate test failure, and other colors to report information about the test run.

Run rspec-puppet

$ cd ~/puppetforge/modules/apt
$ rspec --color --format documentation spec/{classes,defines}/apt*_spec.rb

If all went well you should have seen a ton of green fly by and a summary at the end:

Finished in 8.88 seconds
119 examples, 0 failures

Rspec and rspec-puppet use green to indicate success, red to indicate test failure, and other colors to report information about the test run.

Cucumber-puppet mini tutorial on use

What are we testing?

The apt module has within it a manifests directory and puppet manifests under that. The point of this very quick example is to test the apt class to see if it will create a file resource “sources.list”.

Create a very basic feature

Cucumber-puppet runs on files called testname.features; we have prepended the modulename to these tests to unify our naming scheme between cucumber-puppet and rspec-puppet. We have tried to name the test files after the name of the file in the manifests directory they are testing. Feature files go in features/modules/apt/.

Lets create a file, our file will be named apt_example_init.feature because we have already defined an apt_init.feature. (The absolute path, for clarity, is: ~/puppetforge/apt/features/modules/apt/apt_example_init.feature)

Feature: example init.pp test
  In order to have an effective apt module
  The apt class needs to have certain files
  And must run some execs

    Scenario: Apt Class, basics, sanity check
    Given a node of class "apt"
    When I compile the catalog
    Then compilation should succeed
    And all resource dependencies should resolve

Run this command with:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

If all went well it should have read the Scenario back to you in green, indicating success. The lines under “Feature” are fluff and are not parsed by cucumber-puppet. The lines under Scenario are parsed and have precise meanings to cucumber-puppet. They do exactly what they look like they do. This is a very basic scenario and it serves mostly to demonstrate that the catalog compiles and resources dependencies resolve.

Add a test to the feature

You may find it advantageous to use vim tools for cucumber.

Add the following to your apt_example_init.feature, below the first scenario. To be clear: we are adding a second scenario below the first. These scenarios will compile the catalog independently, ideally they will test different things. Make sure to indent properly!

    Scenario: Basic class, sanity check
    Given a node of class "apt"
    When I compile the catalog
    Then compilation should succeed
    And all resource dependencies should resolve
    And the exec named "apt_update" should be run

And run the test:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

It erred out in yellow. “Undefined step”. That means we have to write a step definition for this step.

Add a step definition

Lets go back to the error from before. A couple things to see in this error:

And the exec named "apt_update" should be run # features/modules/apt/apt_example_init.feature:17
      Undefined step: "the exec named "apt_update" should be run" (Cucumber::Undefined)
      features/modules/apt/apt_example_init.feature:17:in `And the exec named "apt_update" should be run'

This means the step in question is undefined, and we’ll need to write a step for it.

2 scenarios (1 undefined, 1 passed)
9 steps (1 undefined, 8 passed)
0m0.781s

Notice that of 2 Scenarios, only 1 failed; the other passed quite elegantly.

You can implement step definitions for undefined steps with these snippets:

Then /^the exec named "([^"]*)" should be run$/ do |arg1|
  pending # express the regexp above with the code you wish you had
end

This gives you a prototype step definition to match against. Cucumber-puppet works by matching the steps defined in features to regular expressions defined in ruby files in features/steps. Since we want to test an exec type we’ll add this to features/steps/exec.rb

$ vim ~/puppetforge/features/steps/exec.rb

Add the following to the file: (Yes, this is mostly a copy and paste operation)

Then /^the exec named "([^"]*)" should be run$/ do |arg1|
  steps %Q{ 
    Then there should be a resource "Exec[#{arg1}]"
  }
end

The “Then there should be a resource "Exec[#{arg1}]”“ business is another step definition, you can find it in features/steps/puppet.rb . Cucumber-puppet allows us to chain them together so that we can specify high-level things in the feature, but still get good test coverage.

And run the test:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

All greens!

2 scenarios (2 passed)
9 steps (9 passed)
0m0.745s

Working with manifests

Puppet-cucumber can instantiate nodes of class, but it can also read a site.pp file.

The features/support/hooks.rb file is where most of this is configured.


before do
  # adjust local configuration like this
  @puppetcfg['confdir'] = File.join(File.dirname(__FILE__), '..')
  @puppetcfg['manifest'] = File.join(@puppetcfg['confdir'], 'manifests', 'site.pp')
  @puppetcfg['modulepath']  = File.join(@puppetcfg['confdir'],  '..', '..')

  # adjust facts like this
  @facts["architecture"] = "i386"
  @facts["lsbdistcodename"] = "natty"
  @facts["lsbdistdescription"] = "Ubuntu 11.04"
  @facts["lsbdistrelease"] = "11.04"
  @facts["lsbmajdistrelease"] = "11"
end

Here we have a lot of control over cucumber-puppet’s behavior. We specify a puppet-confdir a manifest, a modulepath, and facts. Since we are declaring a puppet-confdir, we can place a puppet.conf file there and configure almost any facet of the master or slave.

Since we are sourcing features/manifests/site.pp (non-standard — we made this up, and if you have a better idea please let us know), we can define nodes there. Add this node to the end of that file:

node "example" {
  include apt 
}

And add the following to your apt_example_init.feature

    Scenario: Basic class, sourced from site.pp, sanity check
    Given a node named "example"
    When I compile the catalog
    Then compilation should succeed
    And all resource dependencies should resolve
    And the exec named "apt_update" should be run

And run the test:

$ cucumber-puppet features/modules/apt/apt_example_init.feature

All greens!

3 scenarios (3 passed)
14 steps (14 passed)
0m0.921s

Rspec-puppet mini tutorial on use

What are we testing?

The apt module has within it a manifests directory and puppet manifests under that. The point of this very quick example is to test the apt class to see if it will create a file resource “sources.list”.

Create a very basic test

Rspec-puppet runs on ruby files. The standard rspec-puppet layout has two spec subdirectories, defines and classes. Tests that focus on testing defines go in the define directory while tests that focus on classes go in the classes directory. The naming scheme is to use the name of the resouce being tested, replacing :: with _ and ending with _spec.rb. So if we were testing the main apt class, we would call it apt_spec.rb and it would be placed in the spec/classes directory.

Lets create a file, our file will be named apt_init_spec.rb because we have already defined an apt_spec.rb. (The absolute path, for clarity, is: ~/puppetforge/apt/spec/classes/apt_init_spec.rb)

require 'spec_helper'

describe 'apt', :type => :class do
  let(:title) { 'init' }
  
  describe "Apt class with no parameters, basic test" do
    let(:params) { { } }
 
    it { should create_class("apt")\
      .with_disable_keys(false)\
      .with_always_apt_update(false) }
  end
end

Run this command with:

$ rspec --color --format documentation spec/classes/apt_init_spec.rb

If all went well it should have read the second describe back, followed by ‘should create Class[“apt”]’ to you in green, indicating success. This is a very basic rspec test and it serves mostly to demonstrate that the catalog compiles and that the desired class is created. It should look like the following:

apt
  Apt class with no parameters, basic test
    should create Class["apt"]

Finished in 0.29773 seconds
1 example, 0 failures

Now lets look at the test line by line. The first line, require ‘spec_helper’ loads a file that will hold our default module path. It is explained further a little later. The next line defines our test. We are testing the ‘apt’ class and it is of type :class. After each describe statement we can use let statements to define aspects of the class or define. For example we choose the title for our class to be init. The next describe is a block to hold a whole test on the class. The describe takes a parameter which is the description for the test, between ‘describe’ and ‘do’. In this example we have used a let for an empty set of params. If we wanted to pass params to the apt class we could also use ‘let(:params) { { :disable_keys => true } }’ After the let, we use an ‘it’ block to actually describe a test. In this case we said that it should create a class using the create_class matcher. The create_class, and other create matchers can also take chained qualifiers like ‘with_disable_keys’ or ‘with_notify’. We can use this to verify parameters or properties of resources. Rspec-puppet will look for any parameter with the name of the string following ‘with’. So if you looked for .with_garbage, it would look for a parameter or property called ‘garbage’.

Add another test

Add the following to your apt_init.spec, below the first ‘it’ block. To be clear: we are adding a second ‘it’ block below the first. This will add a further test to the apt class we started working on.

    it { should create_exec("apt_update")\
      .with_subscribe(['File[sources.list]','File[sources.list.d]'])\
      .with_refreshonly(true)}

And run the test:

$ rspec --color --format documentation spec/classes/apt_init_spec.rb

All greens!

apt
  Apt class with no parameters, basic test
    should create Class["apt"]
    should create Exec["apt_update"]

Finished in 0.36027 seconds
2 example, 0 failures

You can configure rspec-puppet, assigning a modulepath to work from. That is why we require ‘spec_helper’ in our test. ‘spec_helper’ should have contents similar to the following (this assumes the directory structure explained above):

require 'rspec-puppet'

RSpec.configure do |c|
    c.module_path = File.expand_path(File.join(File.dirname(__FILE__), '..', '..'))
end

Going Further

Big Picture

The endgame for this project is to have continuous integration in place for modules on the Puppet Forge. We hope we have provided a decent framework for module developers to provide unit tests for their modules, but that is not the end of the line. The project can only succeed with the involvement of the community. Puppet Labs wants to support the community in any way it can, such as by providing infrastructure, by being a community leader, and by leading the charge. Puppet Labs would like to see an open standard, openly developed, with community involvement. to test modules, and to see that standard implemented with a tool such as Jenkins, Hudson, or Buildbot. Puppet Labs could produce a draft standard for module testing, but at this point that would be premature. Testing modules in the Puppet Forge can work with support from the community, but can only work with community involvement and support.

Technical Details

Open Source

We have been working on two open source projects, rspec-puppet and cucumber-pupppet. Development on both projects is ongoing at our forks on github at github.com/mlitteken/cucumber-puppet and github.com/puppetlabs/rspec-puppet, with pull requests for our changes against the original repositories. There is also active communication on features and pull requests with the maintainer developers.

What we can do, but ugly

On the cucumber-puppet front:

  • We can’t test a define without invoking a manifest.
  • We can specify facts, but only by including a yaml file or modifying hooks file which will be global for all tests.

On the rspec-puppet front:

  • If a test requires both a class and a define: we have to pass a “pre_condition” which is a hand coded puppet manifest snippet, hard coded into the rspec test.
  • If we want to add facts: We pass a hash of facts for each test which is either global to all tests or hard coded for individual tests.

What we can’t do, period.

On the cucumber-puppet front:

On the rspec-puppet front:

  • We can’t access the scope object or any variables in scope.
  • We haven’t even begun to hook it into CI tools.
  • We can’t bulk test a ‘fact grid’ without hand-coding different facts sets.