The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #2879

Test isolation: test idempotency

Added by Markus Roberts over 4 years ago. Updated over 2 years ago.

Status:Code InsufficientStart date:11/28/2009
Priority:LowDue date:
Assignee:-% Done:

0%

Category:testing
Target version:-
Affected Puppet version:0.25.1 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket may be automatically exported to the PUP project on JIRA using the button below:


Description

This is a grab bag of issues centered around the observation that a valid test should produce the same results if run twice in succession. It’s a rather low bar, but some of our tests still fail to meet it. It’s also something that you wouldn’t see unless you specifically went looking for it (so no one is going to be mislead by these test failures in the normal course of things).

Still, they’re worth cleaning up since each one is, at least potentially, a source of future spurious test failures. A test that changes it’s behavior when run twice in succession must 1) depend in some way on external state, 2) change some aspect of the external external state, and thus could be either the cause or the victim in a more normal test isolation problem.

To reproduce any of these just run the same test twice:

ruby spec/unit/myspec.rb spec/unit/myspec.rb -t 60 --format s

While I am lumping these all into one ticket there are actually several sub-categories.

h3. Setup/tear down problems.

First off we have specs that fail one or more test on the second pass

spec/unit/application/puppetmasterd.rb fails one test on the second pass:

should dispatch to parseonly if parseonly is set (FAILED - #)

spec/unit/util/autoload.rb fails four tests on the second pass:

should not die an if a RuntimeError exception is thrown (FAILED - #)
should not die an if a LoadError exception is thrown (FAILED - #)
should not die an if a SyntaxError exception is thrown (FAILED - #)
should require the full path to the file (FAILED - #)

spec/unit/provider/ssh_authorized_key/parsed.rb fails three tests on the second pass:

should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys1 (FAILED - #)
should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys2 (FAILED - #)
should be able to parse example data in ./test/data/providers/ssh_authorized_key/parsed/authorized_keys (FAILED - #)

h3. Registration

Some parts of the system perform registrations; tests that use these parts of the system may unwittingly change persistent state:

spec/integration/indirector/rest.rb ./lib/puppet/indirector.rb:21:in `indirects': Already handling indirection for test_indirected_foo; cannot also handle test_indirected_foo (ArgumentError)

    from ./spec/integration/indirector/rest.rb:11

spec/unit/util/queue.rb

./lib/puppet/util/queue.rb:55:in `register_queue_type': Queue type default is already registered (Puppet::Error)
    from ./spec/unit/util/queue.rb:22

h3. Head scratchers

Several specs contain tests that fail if run once, but succeed (on both passes) if the spec is run twice. I have no ready explanation for this. The specs and the tests are:

spec/unit/network/client.rb

should start the http client up on creation (FAILED - #)

Test isolation: spec/unit/provider/mount/parsed.rb

should write the mount to disk when :flush is called (FAILED - #)

Test isolation: spec/unit/indirector/indirection.rb

should not create a terminus instance until one is actually needed (FAILED - #)

h3. Constant redefinition

Finally, we have a number of tests which generate warnings because they define a constant in the global name space and do not subsequently remove it (thus leading to an “already initialized constant” warning the second time):

spec/unit/network/handler/fileserver.rb      PLUGINS
spec/unit/util/settings/file_setting.rb      FileSetting
spec/unit/transaction/event.rb               Event
spec/unit/transaction/change.rb              Change

History

#1 Updated by Jesse Wolfe over 4 years ago

  • Status changed from Accepted to In Topic Branch Pending Review
  • Branch set to http://github.com/jes5199/puppet/tree/ticket/0.25.x/2966

#2 Updated by Jesse Wolfe over 4 years ago

  • Branch deleted (http://github.com/jes5199/puppet/tree/ticket/0.25.x/2966)

#3 Updated by Jesse Wolfe over 4 years ago

  • Status changed from In Topic Branch Pending Review to Code Insufficient

#4 Updated by Markus Roberts over 4 years ago

  • Target version changed from 0.25.2 to 0.25.3

These issues aren’t visible to people who aren’t looking for them, and this ticket is probably too much of a grab-bag in any case. I’m bumping to 0.25.3 and we’ll probably want to split it up into more actionable items.

#5 Updated by Markus Roberts over 4 years ago

  • Target version changed from 0.25.3 to 0.25.4

#6 Updated by James Turnbull over 4 years ago

  • Target version changed from 0.25.4 to 0.25.5

#7 Updated by James Turnbull about 4 years ago

  • Target version changed from 0.25.5 to 49

#8 Updated by James Turnbull over 3 years ago

  • Target version deleted (49)

#9 Updated by Joshua Lifton over 2 years ago

  • Assignee deleted (Jesse Wolfe)

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

Also available in: Atom PDF