The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #3156

batchable yum and RPM transactions should be batched

Added by Anonymous about 6 years ago. Updated over 2 years ago.

Status:Needs More InformationStart date:02/06/2010
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:package
Target version:-
Affected Puppet version:0.25.4 Branch:
Keywords:performance yum rpm redhat fedora centos customer

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1507


Description

For users mantaining systems with a lot of managed packages (such as desktop configurations), puppet runs can approach 30 minutes plus.

Most of this is due to yum being slow if run multiple times, and it is desirable to batch all yum invocations into single yum commands as much as possible.

It has been suggested that this also be done for rpmquery so as to not repeat it.

We should be able to walk the graph and move the yum/RPM opeartions together in a batchable way, and this would help tremendously with run time on RHEL/CentOS/Fedora.

(May offer similar savings WRT up2date, though IMHO yum on RHEL4 is a desirable practice if you are not a Satellite user.)


Related issues

Related to Puppet - Feature #2198: Install multiple package within a single call to the pack... Investigating 04/25/2009
Related to Puppet - Feature #4983: Remove packages at same time Duplicate 10/11/2010

History

#1 Updated by Peter Meier about 6 years ago

this is similar (or maybe even a dupe?) to #2198 .

There have been some discussion about that on: http://groups.google.com/group/puppet-dev/browse_thread/thread/424b7cbfe52ccfd0/f1065d31cab567ef and http://groups.google.com/group/puppet-dev/browse_thread/thread/424b7cbfe52ccfd0/f1065d31cab567ef

Unfortunately track on the work got lost, as nobody commented on http://groups.google.com/group/puppet-dev/msg/12f918125efd898a

I would really appreciate this feature. I think that other resource-types might be able to be batched together as well, however packages are the most obvious to be done.

However for me it is clear that with the underlying graph only resources which have either no dependencies attached or only other deps of the same resource-type can be batched.

So I imagine 2 things:

  1. batching resources of the same resource-type (which are batchable) with no other dependencies and which target the same state (such as absent or present) together and apply them in one shot.
  2. doing some optimizations on the graph, which batches resources of the same resource-type targeting the same state together and hence apply them in one shot as well.

For the second one imagine the following manifest, which might be quite common:

yumrepo{'somerepo':
  [...]
}

package{'rpmforge':
  [...],
  require => Yumrepo['somerepo'],
}

package{'some-service-package':
  [...],
  require => Package['rpmforge'],
}

file{'/etc/some-service.conf':
  [...],
  require => Package['some-service-package'],
  notify => Service['some-service'],
}

service{'some-service':
  [...],
}

As far as I understood the graph that puppet builds (please correct me if I’m wrong) would look like:

 Yumrepo{'somerepo'] -> Package['rpmforge'] -> Package['some-service-package'] -> File['/etc/some-service.conf'] -> Service['some-service']

So what I imagine that this work on the graph would do the following:

 Yumrepo{'somerepo'] -> BatchedPackage['rpmforge','some-service-package'] -> File['/etc/some-service.conf'] -> Service['some-service']

which would apply @yum-cron@ and @some-service-package@ in one shot, if both would need to be applied.

For sure for both use-cases puppet should sort out the resources that wouldn’t need to be applied and don’t take any action on them, however I imagine that could be done within this batched-resource.

I think that the second use-case is harder to implement and as discussed in the previous discussion the first 1 would already be a good shot and also a solid basis for the second one.

Disclaimer: This is all based on how I understand the resource-graph that puppet builds, it can be that what I proposed can’t be done, as I misunderstood something. Please, correct me if I’m wrong. Further I think that I might have missed some cases where this won’t work.

#2 Updated by James Turnbull about 6 years ago

  • Category set to package
  • Status changed from Unreviewed to Needs More Information
  • Priority changed from High to Normal

Thanks Peter – that was the ticket I was trying to find the other day.

Mike – that was the bit on the IRC where I was clumsily trying to say “more complex than yum/rpm/ blah blah transaction changes”.

#3 Updated by Marc Zampetti over 5 years ago

This is definitely needed. Another use case is when one wishes to install one package that conflicts with another. For example, I have mysql packages that I built that provide an updated version of Mysql. I want to install that package, but it conflicts with the base mysql package from the os provider. Using up2date or YUM, I can delete the old package and install the new one in a single transaction, and everyone is happy. But I cannot do that with Puppet.

#4 Updated by Curtis Ruck over 3 years ago

This is also needed where you have multilib versions (i586 & x86_64) versions installed and you need to update them both concurrently.

#5 Updated by Zachary Stern over 2 years ago

  • Keywords changed from performance yum rpm redhat fedora centos to performance yum rpm redhat fedora centos customer

#7 Updated by James Patterson over 2 years ago

Redmine Issue #3156 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1507

Also available in: Atom PDF