The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com
Test isolation: test idempotency
|Status:||Code Insufficient||Start date:||11/28/2009|
|Affected Puppet version:||0.25.1||Branch:|
Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com
This is a grab bag of issues centered around the observation that a valid test should produce the same results if run twice in succession. It’s a rather low bar, but some of our tests still fail to meet it. It’s also something that you wouldn’t see unless you specifically went looking for it (so no one is going to be mislead by these test failures in the normal course of things).
Still, they’re worth cleaning up since each one is, at least potentially, a source of future spurious test failures. A test that changes it’s behavior when run twice in succession must 1) depend in some way on external state, 2) change some aspect of the external external state, and thus could be either the cause or the victim in a more normal test isolation problem.
To reproduce any of these just run the same test twice:
ruby spec/unit/myspec.rb spec/unit/myspec.rb -t 60 --format s
While I am lumping these all into one ticket there are actually several sub-categories.
h3. Setup/tear down problems.
First off we have specs that fail one or more test on the second pass
spec/unit/application/puppetmasterd.rb fails one test on the second pass:
should dispatch to parseonly if parseonly is set (FAILED - #)
spec/unit/util/autoload.rb fails four tests on the second pass:
should not die an if a RuntimeError exception is thrown (FAILED - #) should not die an if a LoadError exception is thrown (FAILED - #) should not die an if a SyntaxError exception is thrown (FAILED - #) should require the full path to the file (FAILED - #)
spec/unit/provider/ssh_authorized_key/parsed.rb fails three tests on the second pass:
should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys1 (FAILED - #) should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys2 (FAILED - #) should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys (FAILED - #)
Some parts of the system perform registrations; tests that use these parts of the system may unwittingly change persistent state:
spec/integration/indirector/rest.rb ./lib/puppet/indirector.rb:21:in `indirects': Already handling indirection for test_indirected_foo; cannot also handle test_indirected_foo (ArgumentError)
./lib/puppet/util/queue.rb:55:in `register_queue_type': Queue type default is already registered (Puppet::Error) from ./spec/unit/util/queue.rb:22
h3. Head scratchers
Several specs contain tests that fail if run once, but succeed (on both passes) if the spec is run twice. I have no ready explanation for this. The specs and the tests are:
should start the http client up on creation (FAILED - #)
Test isolation: spec/unit/provider/mount/parsed.rb
should write the mount to disk when :flush is called (FAILED - #)
Test isolation: spec/unit/indirector/indirection.rb
should not create a terminus instance until one is actually needed (FAILED - #)
h3. Constant redefinition
Finally, we have a number of tests which generate warnings because they define a constant in the global name space and do not subsequently remove it (thus leading to an “already initialized constant” warning the second time):
spec/unit/network/handler/fileserver.rb PLUGINS spec/unit/util/settings/file_setting.rb FileSetting spec/unit/transaction/event.rb Event spec/unit/transaction/change.rb Change
#4 Updated by Markus Roberts about 5 years ago
- Target version changed from 0.25.2 to 0.25.3
These issues aren’t visible to people who aren’t looking for them, and this ticket is probably too much of a grab-bag in any case. I’m bumping to 0.25.3 and we’ll probably want to split it up into more actionable items.