Feature #1519

Speed up checksumming

Added by Steve McIntosh almost 5 years ago. Updated about 2 years ago.

Status:AcceptedStart date:08/18/2008
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:fileserving
Target version:3.x
Affected Puppet version:0.24.4 Branch:
Keywords:

Description

According to dtrace, checksuming (MD5) takes a lot of time for each catalog run. Anything that can be done to speed this up would be valuable.

3371-If-lite-is-not-specified-we-hand-the-reading-of-the.patch Magnifier (1.5 KB) Steve McIntosh, 09/02/2008 09:49 pm

History

#1 Updated by James Turnbull almost 5 years ago

  • Category set to fileserving
  • Status changed from Unreviewed to Needs Decision
  • Assignee set to Luke Kanies
  • Target version set to 4
  • 3 changed from Unknown to Hard

Might be solved by some work Luke is undertaking now.

#2 Updated by Luke Kanies almost 5 years ago

  • Status changed from Needs Decision to Accepted
  • Assignee changed from Luke Kanies to Steve McIntosh

This is orthogonal to the work I’m doing.

Steve, don’t you have code to attach to the ticket?

#4 Updated by James Turnbull almost 5 years ago

  • Status changed from Accepted to Needs More Information

#5 Updated by James Turnbull almost 5 years ago

  • Status changed from Needs More Information to Needs Decision

#6 Updated by James Turnbull almost 5 years ago

  • Assignee changed from Steve McIntosh to Luke Kanies

#7 Updated by Luke Kanies almost 5 years ago

  • Status changed from Needs Decision to Ready For Checkin

#8 Updated by Luke Kanies almost 5 years ago

  • Status changed from Ready For Checkin to Tests Insufficient

The code is good, but I’m pretty sure this will break the existing tests for the Checksums module.

#9 Updated by Luke Kanies almost 5 years ago

I just ran the tests and the mocks have now changed:

1)
Mocha::ExpectationError in 'Puppet::Util::Checksums when using md5 should use incremental Digest::MD5 sums to calculate file checksums'
#.file('/path/to/my/file') - expected calls: 0, actual calls: 1
/Users/luke/git/puppet/lib/puppet/util/checksums.rb:76:in `checksum_file'
/Users/luke/git/puppet/lib/puppet/util/checksums.rb:20:in `md5_file'
./spec/unit/util/checksums.rb:55:in `send'
./spec/unit/util/checksums.rb:55:
/Users/luke/git/puppet/spec/monkey_patches/add_confine_and_runnable_to_rspec_dsl.rb:19:in `run'
/Users/luke/git/puppet/spec/monkey_patches/add_confine_and_runnable_to_rspec_dsl.rb:17:in `each'
/Users/luke/git/puppet/spec/monkey_patches/add_confine_and_runnable_to_rspec_dsl.rb:17:in `run'

2)
Mocha::ExpectationError in 'Puppet::Util::Checksums when using sha1 should use incremental Digest::SHA1 sums to calculate file checksums'
#.file('/path/to/my/file') - expected calls: 0, actual calls: 1
/Users/luke/git/puppet/lib/puppet/util/checksums.rb:76:in `checksum_file'
/Users/luke/git/puppet/lib/puppet/util/checksums.rb:49:in `sha1_file'
./spec/unit/util/checksums.rb:55:in `send'
./spec/unit/util/checksums.rb:55:
/Users/luke/git/puppet/spec/monkey_patches/add_confine_and_runnable_to_rspec_dsl.rb:19:in `run'
/Users/luke/git/puppet/spec/monkey_patches/add_confine_and_runnable_to_rspec_dsl.rb:17:in `each'
/Users/luke/git/puppet/spec/monkey_patches/add_confine_and_runnable_to_rspec_dsl.rb:17:in `run'

Finished in 0.082169 seconds

20 examples, 2 failures

#10 Updated by Luke Kanies about 3 years ago

  • Status changed from Tests Insufficient to Accepted
  • Assignee deleted (Luke Kanies)
  • Target version changed from 4 to 2.7.x

This is at least somewhat better now because we’re using 4096 blocks.

#11 Updated by Nigel Kersten about 2 years ago

  • Target version changed from 2.7.x to 3.x

Aren’t the tests still insufficient?

Also available in: Atom PDF