The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #13199

md5lite, mtime not honoured for file type/provider

Added by Ken Barber over 2 years ago. Updated 9 months ago.

Status:AcceptedStart date:03/16/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:file
Target version:3.5.0
Affected Puppet version:2.7.20 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1208


Description

It seems I can’t get the puppetmaster to honour the checksum => mtime setting, or the md5lite setting. So the following example has no performance improvement over just using md5:

file { "/testtransfer":
  ensure => directory,
  recurse => remote,
  purge => true,
  checksum => mtime,
  source => "puppet:///modules/${module_name}/bigfileshere",
}

The problem seems to be in multiple places. The first place, I can’t get the file_server/metadata to respond with mtime, md5lite checksums:

# curl -k --cert /etc/puppetlabs/pupem --key /etc/puppetlabs/puppet/ssl/private_keys/puppetclient2.vm.pem --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem -H 'Accept: yaml' 'https://puppet:8140/production/file_metadatas/modules/filetransfer/sles/SLES-11-SP1-DVD-x86_64-GM-DVD1.iso checksum_type=md5lite'
--- 
  - !ruby/object:Puppet::FileServing::Metadata
    checksum: "{md5}d2e10420f3689faa49a004b60fb396b7"
    checksum_type: md5
    destination: 
    expiration: 2012-03-16 17:49:23.600743 -07:00
    ftype: file
    group: 0
    links: !ruby/sym manage
    mode: 420
    owner: 0
    path: /etc/puppetlabs/puppet/modules/filetransfer/files/sles/SLES-11-SP1-DVD-x86_64-GM-DVD1.iso
    relative_path: .
    stat_method: !ruby/sym lstat

This is fixed with this patch:

diff --git a/lib/puppet/indirector/file_server.rb b/lib/puppet/indirector/file_server.rb
index 9516a40..9fbd57e 100644
--- a/lib/puppet/indirector/file_server.rb
+++ b/lib/puppet/indirector/file_server.rb
@@ -51,6 +51,7 @@ class Puppet::Indirector::FileServer < Puppet::Indirector::Terminus
     Puppet::FileServing::Fileset.merge(*filesets).collect do |file, base_path|
       inst = model.new(base_path, :relative_path => file)
       inst.links = request.options[:links] if request.options[:links]
+      inst.checksum_type = request.options[:checksum_type] if request.options[:checksum_type]
       inst.collect
       inst
     end

Which does the correct thing:

# curl -k --cert /etc/puppetlabs/puppet/ssl/certs/puppetclient2.vm.pem --key /etc/puppetlabs/puppet/ssl/private_keys/puppetclient2.vm.pem --cacert /etc/puppetlabs/puppet/ssl/certs/ca.pem -H 'Accept: yaml' 'https://puppet:8140/production/file_metadatas/modules/filetransfer/sles/SLES-11-SP1-DVD-x86_64-GM-DVD1.iso?checksum_type=md5lite'
--- 
  - !ruby/object:Puppet::FileServing::Metadata
    checksum: "{md5lite}bf619eac0cdf3f68d496ea9344137e8b"
    checksum_type: md5lite
    destination: 
    expiration: 2012-03-16 17:51:43.033651 -07:00
    ftype: file
    group: 0
    links: !ruby/sym manage
    mode: 420
    owner: 0
    path: /etc/puppetlabs/puppet/modules/filetransfer/files/sles/SLES-11-SP1-DVD-x86_64-GM-DVD1.iso
    relative_path: .
    stat_method: !ruby/sym lstat

The second problem seems to be in the provider itself, it seems to me as if its still trying to do an md5 scan on the local filesystem – but I’ll have to dig deeper to find the root cause for this.


Related issues

Related to Puppet - Bug #18333: checksum => 'mtime' fails Investigating
Duplicated by Puppet - Bug #6004: checksum => mtime still reads the source and target files Duplicate 01/25/2011

History

#1 Updated by Ken Barber over 2 years ago

  • Description updated (diff)

#2 Updated by Ken Barber over 2 years ago

  • Description updated (diff)

#3 Updated by Ken Barber over 2 years ago

  • Description updated (diff)

#4 Updated by Oliver Hookins over 2 years ago

By any chance is this the same bug as reported in this thread? http://www.mail-archive.com/puppet-users@googlegroups.com/msg22693.html

#5 Updated by Ken Barber over 2 years ago

I don’t know Oliver – I certainly didn’t get that kind of error message. Puppet ignores the mtime/md5lite setting and just falls back to md5 in code (which is a hard-coded default).

#6 Updated by Kelsey Hightower over 2 years ago

  • Status changed from Unreviewed to Investigating
  • Assignee set to Ken Barber

#7 Updated by Nigel Kersten over 2 years ago

  • Affected Puppet version deleted (2.7.12rc2)

Ken, did you find anything else out here? Does it really not work at all?

#8 Updated by Juan Moyano over 2 years ago

I’m having the same issue here ( using puppet master 2.6.2-5 on Debian Squeeze )

Trying to improve file serving perfomance tweaking checksum options with no luck.

#9 Updated by Ken Barber over 2 years ago

  • Status changed from Investigating to Accepted
  • Assignee deleted (Ken Barber)
  • Affected Puppet version set to 2.7.14

I think I’ve pretty much established this is true, but alas – its not a quick fix.

#10 Updated by Dustin Mitchell over 1 year ago

  • Affected Puppet version changed from 2.7.14 to 2.7.20

Still the case in 2.7.20

#11 Updated by Stefan Wiederoder over 1 year ago

and still with puppet 3.x?

any chance to get this bug on the roadmap? otherwise it´d be better to fix the documentation.

#12 Updated by Benoît Allard about 1 year ago

This is not fixed in 3.x either (at least 3.1.1)

I get the following error:

change from {md5lite}365b7b8a76573d68f6749e61b30f73ea to {md5}251743dfd3fda414570524bac9e55381 failed: Could not rename temporary file c:/dotNetFx40_Full_x86_x64.exe.puppettmp_9346 to c:/dotNetFx40_Full_x86_x64.exe: File written to disk did not match checksum; discarding changes ({md5lite}251743dfd3fda414570524bac9e55381 vs {md5lite}365b7b8a76573d68f6749e61b30f73ea)

As you can see, he is comparing the md5lite he calculated (365b…) with the md5 he got from the server he is handling as a md5lite (2517…) …

#13 Updated by Michelle Johansen 10 months ago

  • Target version set to 3.5.0

#14 Updated by Anonymous 9 months ago

Redmine Issue #13199 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1208

Also available in: Atom PDF