The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #12879

undefined method `has_openstack_mac?' for Facter::Util::EC2:Module

Added by Chip Schweiss about 4 years ago. Updated about 3 years ago.

Status:InvestigatingStart date:02/29/2012
Priority:HighDue date:
Assignee:-% Done:

0%

Category:library
Target version:-
Keywords: Affected Facter version:1.6.6
Branch:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

This is on CentOS 6.2 w/ Puppet 2.6.14-1.el6 both from the Puppet yum repository.

The problem is only occurring in daemon mode of puppet.

This is in the syslog repeatedly:

Feb 29 08:14:55 hostname puppet-agent[21815]: Could not run Puppet configuration client: Could not retrieve local facts: undefined method `has_openstack_mac?' for Facter::Util::EC2:Module

I’ve checked on several systems and it appears to be a bug introduced in 1.6.6-1. The last successful run in the logs contains:

Feb 28 14:55:18 hostname puppet-agent[21815]: (/Stage[puppet]/Facter/Package[facter]/ensure) ensure changed '1.6.5-1.el6' to '1.6.6-1.el6'

History

#1 Updated by zach armstrong about 4 years ago

I am seeing this too in RHEL5 with puppet 2.7.11.

It doesn’t seem to occur every time puppet makes a run. I have not been able to figure out how to reliably repro it.

This error seems to be due to facter 1.6.6. Problem started from this commit: https://github.com/puppetlabs/facter/commit/cb598aab3e36767abd6ce5827e47099ecb49ef67#diff-0

If you want to revert these two files, you can grab them from here: https://github.com/puppetlabs/facter/blob/1df5b46e0ec0bd8a42254b571ca6eb1acd814ed6/lib/facter/util/ec2.rb https://github.com/puppetlabs/facter/blob/1df5b46e0ec0bd8a42254b571ca6eb1acd814ed6/lib/facter/ec2.rb

#2 Updated by Ken Barber about 4 years ago

  • Category set to library
  • Status changed from Unreviewed to Investigating
  • Assignee set to Ken Barber

So I’m trying to understand this, as it doesn’t appear that the code is doing anything abnormal here, it feels like either a library loading issue or perhaps resident files (either from an old install or an alternative install). But alas, I need more information.

You say here:

The problem is only occurring in daemon mode of puppet.

Now I found this pastie with a log of the issue:

http://pastie.org/3451152

In that particular case the problem seems obvious, the agent has upgraded facter – the util class & the fact has changed, but because the util class doesn’t automatically reload on its own, we are seeing a missing method. A reload of the puppet agent should solve this.

Really in that kind of scenario – the agent needs to be reloaded. I would expect this to be the normal process for a facter or puppet upgrade – ie. always restart the agent. Otherwise, libraries that have changed (like the EC2 library) will still reflect the old revision in memory.

So my question is – have you tried restarting the agent? And does that alleviate the problem?

#3 Updated by Chip Schweiss about 4 years ago

I tried restarting numerous times. It did not help, the same errors occurred.

#4 Updated by Ken Barber about 4 years ago

So I can’t replicate it yet, I’m putting it on a 30 second run cycle to see if I can replicate it in time.

Questions:

  • If you run just ‘facter’ or ‘facter -p’ on their own, do you have a problem? Does the EC2 fact work as expected? (ie. do you get ec2_* entries in facter). Of course if you are not running this on Amazon, then you probably won’t.
  • Can you run ‘puppet agent -t’ and does it work/fail?
  • How easy is this to replicate for yourself? Is running puppet in agent mode now completely broken?

Also – can you show me the outcome of the following:

  • rpm -qa | grep facter
  • grep ‘has_openstack_mac?’ ‘/usr/lib/ruby/site_ruby/1.8/facter/util/ec2.rb’
  • find / -name ‘ec2.rb’

(the third item will take quite some time I presume)

Of course my objective right now is to replicate the problem, to help me understand the fault … so anything else you can think of would be great.

#5 Updated by Chris Möser about 4 years ago

I have the same issue running Ubuntu 10.04 LTS server. On top I have a duplicate server running the same version(s)/packages not showing this error. On that one rig the command puppet agent --test --verbose returns the error. The client puppet.conf looks like this:

[main]
logdir=/var/log/puppet
vardir=/var/lib/puppet
ssldir=/var/lib/puppet/ssl
rundir=/var/run/puppet
factpath=$vardir/lib/facter
templatedir=$confdir/templates
[master]
# These are needed when the puppetmaster is run by passenger
# and can safely be removed if webrick is used.
ssl_client_header = SSL_CLIENT_S_DN
ssl_client_verify_header = SSL_CLIENT_VERIFY
[agent]
server=server.domain.tld

which is pretty much unmodified. The same configuration was running pre-update to the current version. If there is any further information you need, just ask. If there is any command to run, just drop a line.

UPDATE:

facter -p just drops the same error on this erroneous machine. It doesn’t generate any further output like the other machine when I run that command.

The service on this machine can be established with service puppet start after which it is present in the process list. The error occurs when I run it manually to verify correct processing of new manifests.

Chris

#6 Updated by Chris Möser about 4 years ago

I was able to reconstruct and solve this. I had a facter gem installed (1.6.5) and at some later point the deb from puppetlabs. Deleting the gem fixed it for me.

#7 Updated by Steve Traylen about 4 years ago

Also observed this on RHEL6 from EPEL pkgs to puppet lab packages.

Plan was:

  • Update facter-1.6.5-1.el6 to facter-1.6.6-1.el6
  • Update Puppet 2.6.13-2.el6 to 2.7.11-2.el6

The following happened:

  • The puppet deamon was running with 2.6.13-2.el6 and first puppetrun updated puppet (before facter)
  • This SIGTERMS puppet ‘Caught TERM; calling stop’ so that puppet run stops.
  • 15 minutes later puppet runs again and does the facter update.
  • 15 minutes later puppet runs again and spits out the EC2 error.
  • 15 later same as last step and repeat.

For me anyway restarting the puppet daemon was enough for everything to settle down, all seems well now… Guessing there is confusion between the facter at puppet startup and facter on disk?

#8 Updated by Orion Poplawski about 4 years ago

For me this is triggered when running puppet in daemon mode and facter was upgraded from 1.6.5 to 1.6.6. I filed https://bugzilla.redhat.com/show_bug.cgi?id=806370 against Fedora for it suggesting that the factor package restart puppet agent daemon on upgrade. That fixes the problem, but we’re wondering if that is the right thing to do in general or if this was an unexpected change introduced in 1.6.6 and unlikely to re-occur.

Perhaps the puppet agent daemon needs to be changed to reload the facter modules when it starts a run rather on startup?

#9 Updated by Todd Zullinger about 4 years ago

In particular, I’m unsure if adding ‘service puppet condrestart’ is going to cause us pain. If we restart puppet and it is mid-run, classes could be left partially updated. I’m unclear how resilient puppet is to stopping the agent while it is in the middle of a run. I know there were some tickets on what it should do in this case, but I’m not sure if those are resolved. It’s a tricky issue.

#10 Updated by Ken Barber almost 4 years ago

  • Assignee deleted (Ken Barber)

#11 Updated by Orion Poplawski over 3 years ago

Similar issue with update to 1.6.14:

puppet-agent[1154]: Could not run Puppet configuration client: Could not retrieve local facts: undefined method `kernel_fact_value' for Facter::Util::Processor:Module

#12 Updated by Orion Poplawski about 3 years ago

Any more thoughts on this? I seem to get this with every facter update.

#13 Updated by Charlie Sharpsteen about 3 years ago

May not be the full story, but this is reproducible when running facter on a machine with multiple versions installed—-similar to what Chris Moser reported. For example, on Ubuntu with Facter installed from the PuppetLabs APT repository alongside a local git clone running under envpuppet:

root@ubuntuagent:~# facter
Error: undefined method `kernel_fact_value' for Facter::Util::Processor:Module

Then if I remove the version installed through APT:

root@ubuntuagent:~# apt-get remove facter
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages were automatically installed and are no longer required:
  libshadow-ruby1.8 libaugeas0 ruby-json libruby ruby libaugeas-ruby1.8 hiera libjson-ruby augeas-lenses
Use 'apt-get autoremove' to remove them.
The following packages will be REMOVED:
  facter puppet puppet-common
...and so on

root@ubuntuagent:~# facter
No LSB modules are available.
architecture => i386
augeasversion => 0.10.0
...many more facts

And if I re-install Facter, the problem returns:

root@ubuntuagent:~# apt-get install facter
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  facter
...blah blah
Setting up facter (1.6.17-1puppetlabs1) ...

root@ubuntuagent:~# facter
Error: undefined method `kernel_fact_value' for Facter::Util::Processor:Module

Similar behavior occurs on CentOS with yum. So, this definitely looks like some sort of library loading issue.

Also available in: Atom PDF