The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #5200

Many-to-many implementation of stages needlessly inefficient

Added by Markus Roberts over 5 years ago. Updated over 3 years ago.

Status:ClosedStart date:11/03/2010
Priority:NormalDue date:
Assignee:Nick Lewis% Done:

0%

Category:performance
Target version:-
Affected Puppet version:2.6.0 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Trevor Vaughan reports:

Stages appear, by default, to generate a many-to-many dependency graph. By adding 'pre' and 'post' stages that contain a single item, the graphs are vastly reduced in complexity and we're seeing something like a 3X speedup in graph execution in simple test cases. I've attached a couple of puppet files illustrating the issue with their associated graphs. If the graphs could be auto-reduced to a single point prior to execution automatically, it would be quite helpful.

I believe the savings could be even greater in complex situations. We had discussed addressing this internally by creating a whit (null resource) but the idea apparently did not make it into code. It should be an easy win.


Related issues

Related to Puppet - Bug #4590: SimpleGraph is slow Closed 08/23/2010
Related to Puppet - Bug #5012: File should autorequire Mounts Code Insufficient 10/15/2010
Related to Puppet - Feature #5015: mount should autorequire mountpoint Code Insufficient 10/15/2010
Related to Puppet - Bug #428: Deletion should invert sort order Rejected
Related to Puppet - Bug #5369: Autorequire loop resolution inadequate Accepted 11/19/2010
Related to Puppet - Feature #6911: Flexible, deterministic, unguessable application order Closed 03/30/2011
Related to Puppet - Bug #9671: transaction eval_generate slow on many files in recurse m... Closed 09/26/2011

History

#1 Updated by Markus Roberts over 5 years ago

  • Assignee set to Markus Roberts
  • Target version set to 2.7.x

Luke, Teyo & Nigel have all agreed that this would be very nice to get in Statler.

#2 Updated by Markus Roberts over 5 years ago

Just to clarify, I linked the two autorequire tickets due to implicit conceptual collision between the issues discussed on the respective threads on puppet-dev; specifically, those tickets propose expanding / extending the autorequire functionality while Luke’s refactor/master/3691-no_relationship_graph branch, which he suggested as an alternate resolution on the thread for this ticket removes autorequire functionality and it’s not immediately clear (to me at least) how it could be restored in that model.

I think eliminating the second graph as Luke’s branch does is the right idea, and I think representing containment with bracket node as Tervor, Jesse and other have advocated is better than adding edge types. But I’m not yet seeing how either idea can work in actual code. Just adding bracket nodes as an added complication in splice! looks like the easiest, though significantly less efficient. In short, I’m finding myself in agreement with pretty much everyone on the details and with almost no one on the big picture.

#3 Updated by Markus Roberts about 5 years ago

To update: Jesse & I have a working sketch of the bracket solution which keeps the “two graphs” structure for now. It’s turning out to be a needed step for #6911 and will probably be folded in with that.

#4 Updated by Markus Roberts about 5 years ago

The branch on #6911 now contains a sketch of the fix for this, with one known bug and a small amount of cleanup left to do.

#5 Updated by Anonymous over 4 years ago

  • Assignee deleted (Markus Roberts)

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

#6 Updated by Anonymous over 4 years ago

This issue was assigned to a former Puppet Labs employee. Adding back to the pool of unreviewed issues.

#7 Updated by Ben Hughes over 4 years ago

  • Description updated (diff)
  • Status changed from Accepted to Unreviewed

#8 Updated by Nigel Kersten over 4 years ago

  • Description updated (diff)
  • Status changed from Unreviewed to Needs More Information
  • Assignee set to Nick Lewis

Nick, didn’t this get resolved when we had the same issue with class –> class relationships and the Whit implementation was meant to solve it?

#9 Updated by Nick Lewis about 4 years ago

  • Status changed from Needs More Information to Closed

This has been resolved for a year or so.

#10 Updated by Anonymous over 3 years ago

  • Target version deleted (2.7.x)

Also available in: Atom PDF