The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #9145

error message is not clear when puppet agent runs out of disk space during cert generation

Added by Dan Bode over 4 years ago. Updated over 4 years ago.

Status:AcceptedStart date:08/22/2011
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:SSL
Target version:-
Affected Puppet version: Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

When certs generated by puppet agent fail b/c of disk space, the error message says that something is wrong with the ca headers:

This was observed against puppet 2.6.9

root@ubuntu-1004-32-2:/etc/puppetlabs/puppet# puppet agent --test --debug --trace
debug: Failed to load library 'selinux' for feature 'selinux'
debug: Puppet::Type::User::ProviderPw: file pw does not exist
debug: Puppet::Type::User::ProviderDirectoryservice: file /usr/bin/dscl does not exist
debug: Puppet::Type::User::ProviderUser_role_add: file roledel does not exist
debug: Puppet::Type::User::ProviderLdap: true value when expecting false
debug: Puppet::Type::File::ProviderMicrosoft_windows: feature microsoft_windows is missing
debug: /File[/etc/puppetlabs/puppet/puppet.conf]: Autorequiring File[/etc/puppetlabs/puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/certificate_requests]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/etc/puppetlabs/puppet/ssl/private_keys]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/var/opt/lib/pe-puppet/client_data]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/var/opt/lib/pe-puppet/client_yaml]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/var/opt/lib/pe-puppet/state/graphs]: Autorequiring File[/var/opt/lib/pe-puppet/state]
debug: /File[/var/opt/lib/pe-puppet/lib]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/var/opt/lib/pe-puppet/state]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/var/opt/lib/pe-puppet/clientbucket]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/private]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/var/run/pe-puppet/agent.pid]: Autorequiring File[/var/run/pe-puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/certs]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/etc/puppetlabs/puppet/ssl/public_keys]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/etc/puppetlabs/puppet/ssl]: Autorequiring File[/etc/puppetlabs/puppet]
debug: /File[/var/opt/lib/pe-puppet/facts]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/public_keys]/ensure: created
debug: /File[/etc/puppetlabs/puppet/ssl/certs]/ensure: created
debug: /File[/etc/puppetlabs/puppet/ssl/private]/ensure: created
debug: /File[/etc/puppetlabs/puppet/ssl/certificate_requests]/ensure: created
debug: /File[/etc/puppetlabs/puppet/ssl/private_keys]/ensure: created
debug: Finishing transaction -610347968
debug: /File[/etc/puppetlabs/puppet/ssl/public_keys]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/etc/puppetlabs/puppet/ssl]: Autorequiring File[/etc/puppetlabs/puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/private_keys]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/var/opt/lib/pe-puppet/facts]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/certificate_requests]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/var/opt/lib/pe-puppet/state]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/private]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: /File[/var/opt/lib/pe-puppet/lib]: Autorequiring File[/var/opt/lib/pe-puppet]
debug: /File[/etc/puppetlabs/puppet/ssl/certs]: Autorequiring File[/etc/puppetlabs/puppet/ssl]
debug: Finishing transaction -611172438
info: Creating a new SSL key for ubuntu-1004-32-2
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for ca
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
info: Caching certificate_request for ubuntu-1004-32-2
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/base.rb:42:in `initialize'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/base.rb:42:in `new'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/base.rb:42:in `read'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector/ssl_file.rb:86:in `find'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector/indirection.rb:214:in `find_in_cache'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector/indirection.rb:183:in `find'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector.rb:50:in `find'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/host.rb:162:in `certificate'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/host.rb:187:in `generate'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/host.rb:228:in `wait_for_cert'
/opt/puppet/lib/site_ruby/1.8/puppet/application/agent.rb:194:in `setup_host'
/opt/puppet/lib/site_ruby/1.8/puppet/application/agent.rb:259:in `setup'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:304:in `run'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:420:in `hook'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:304:in `run'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:411:in `exit_on_fail'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:304:in `run'
/opt/puppet/lib/site_ruby/1.8/puppet/util/command_line.rb:62:in `execute'
/usr/local/bin/puppet:4
err: Cached certificate for ca failed: header too long
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for ca
warning: peer certificate won't be verified in this SSL session
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/base.rb:42:in `initialize'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/base.rb:42:in `new'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/base.rb:42:in `read'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector/ssl_file.rb:86:in `find'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector/indirection.rb:214:in `find_in_cache'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector/indirection.rb:183:in `find'
/opt/puppet/lib/site_ruby/1.8/puppet/indirector.rb:50:in `find'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/host.rb:162:in `certificate'
/opt/puppet/lib/site_ruby/1.8/puppet/ssl/host.rb:229:in `wait_for_cert'
/opt/puppet/lib/site_ruby/1.8/puppet/application/agent.rb:194:in `setup_host'
/opt/puppet/lib/site_ruby/1.8/puppet/application/agent.rb:259:in `setup'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:304:in `run'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:420:in `hook'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:304:in `run'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:411:in `exit_on_fail'
/opt/puppet/lib/site_ruby/1.8/puppet/application.rb:304:in `run'
/opt/puppet/lib/site_ruby/1.8/puppet/util/command_line.rb:62:in `execute'
/usr/local/bin/puppet:4
err: Cached certificate for ca failed: header too long
warning: peer certificate won't be verified in this SSL session
info: Caching certificate for ca
warning: peer certificate won't be verified in this SSL session
Exiting; no certificate found and waitforcert is disabled

This results in empty public and private key files.


Related issues

Duplicated by Puppet - Bug #4237: If a 0 byte file exists in the /var/lib/puppet/ssl/ca/req... Rejected 07/14/2010

History

#1 Updated by Ryan Conway over 4 years ago

Hey there,

I came across this problem today when our Puppet Master ran out of disk space – incoming certificate requests weren’t being written to the disk properly, resulting in zero byte files at /var/lib/puppet/ssl/ca/requests/.

This caused attempts to sign certificates to fail with a vague ‘header too long’ error message, and any attempt to interact with the ‘puppetca’ command failed with a similar ‘err: Could not call list: header too long’ error.

Removing the zero byte file restore functionality.

There is another ticket #4237 which describes this exact behaviour but has been rejected. I can reproduce this on Puppet Master 2.7.6, but haven’t been able to upgrade to 2.7.9 yet.

#2 Updated by Anonymous over 4 years ago

  • Category set to SSL

Ryan Conway wrote:

Hey there,

I came across this problem today when our Puppet Master ran out of disk space – incoming certificate requests weren’t being written to the disk properly, resulting in zero byte files at /var/lib/puppet/ssl/ca/requests/.

This caused attempts to sign certificates to fail with a vague ‘header too long’ error message, and any attempt to interact with the ‘puppetca’ command failed with a similar ‘err: Could not call list: header too long’ error.

Removing the zero byte file restore functionality.

There is another ticket #4237 which describes this exact behaviour but has been rejected. I can reproduce this on Puppet Master 2.7.6, but haven’t been able to upgrade to 2.7.9 yet.

The reason the other ticket was rejected was that we can’t generally defend against corruption caused by external failures like running out of disk space.

While true, we can certainly try and do a better job of providing helpful, clear failure messages rather than the nasty ones we give; that helps everyone, including us, make sure things work sanely.

We would absolutely accept a patch improving the error handling behaviour here, and it isn’t likely we will spend much internal time on it in the next few months otherwise. Not that it isn’t important, just not as important as some of the other targets we have.

#3 Updated by Ryan Conway over 4 years ago

Sure, I understand!

My only suggestion then would be to just make the file handling more robust so it skips files it has trouble reading or processing, so at least it won’t block the certificate authority from signing new requests completely, and of course your suggestion for more helpful error messages too.

#4 Updated by Anonymous over 4 years ago

Ryan Conway wrote:

Sure, I understand!

My only suggestion then would be to just make the file handling more robust so it skips files it has trouble reading or processing, so at least it won’t block the certificate authority from signing new requests completely, and of course your suggestion for more helpful error messages too.

That seems totally reasonable, also, in that corrupt data should generally not break the entire system. Robustness is, to a degree, reasonable.

When it comes to implementation we are likely to dig into the security implications of that in the concrete area of the CA and SSL, but I suspect that is an acceptable security choice.

Also available in: Atom PDF