The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #16770

"uninitialized constant ActiveRecord" on puppet 3 w/ storedconfigs

Added by eric sorenson over 3 years ago. Updated about 3 years ago.

Status:ClosedStart date:10/03/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Affected Puppet version:3.0.0 Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Two users on the mailing list reported the same error loading active record on Telly: https://groups.google.com/d/topic/puppet-users/3tC0gYg9XA0/discussion

One is on EL6, the other Ubuntu 10.4

Yesterday my puppetmaster and nodes got upgraded to puppet-3.0.0. 

Since then, all puppet runs have been failing with this error: 

Error: Could not retrieve catalog from remote server: Error 400 on 
SERVER: Could not autoload puppet/indirector/node/active_record: 
uninitialized constant ActiveRecord 


My colleague and I have put a few hours into trying to work out what's 
wrong. rubygem-activerecord-2.1.1-2.el6.noarch is installed from the 
puppetlabs RPM repo. We've reinstalled all components but made no progress. 

Related issues

Related to Puppet Community Package Repository - Refactor #14080: Debian Squeeze package puppetmaster: Could not autoload a... Closed 04/19/2012
Related to Puppet - Bug #4063: unguarded ActiveRecord dependency causes rake spec to fail Closed 06/23/2010
Related to Puppet - Bug #4763: ActiveRecord detection fails Closed 09/14/2010
Related to Puppet - Bug #2508: misleading error about ActiveRecord versions Closed 08/06/2009
Related to Puppet - Bug #2324: Puppetd fails to start without rails Closed 06/04/2009

History

#1 Updated by eric sorenson over 3 years ago

  • Target version set to 3.0.1

#2 Updated by Joe Hillenbrand over 3 years ago

The Ubuntu 10.04.1 guy here.

Here is the result of puppet agent -t --trace:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER: Could not autoload puppet/indirector/node/active_record: uninitialized constant ActiveRecord
/usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:65:in `deserialize'
/usr/lib/ruby/vendor_ruby/puppet/indirector/rest.rb:121:in `find'
/usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:191:in `find'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:243:in `retrieve_new_catalog'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:348:in `thinmark'
/usr/lib/ruby/1.8/benchmark.rb:308:in `realtime'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:347:in `thinmark'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:242:in `retrieve_new_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:67:in `retrieve_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:107:in `prepare_and_retrieve_catalog'
/usr/lib/ruby/vendor_ruby/puppet/configurer.rb:159:in `run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45
/usr/lib/ruby/vendor_ruby/puppet/agent/locker.rb:20:in `lock'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45
/usr/lib/ruby/1.8/sync.rb:230:in `synchronize'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:45
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:119:in `with_client'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:42
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:84:in `run_in_fork'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:41
/usr/lib/ruby/vendor_ruby/puppet/application.rb:175:in `call'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:175:in `controlled_run'
/usr/lib/ruby/vendor_ruby/puppet/agent.rb:39:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:338:in `onetime'
/usr/lib/ruby/vendor_ruby/puppet/application/agent.rb:311:in `run_command'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:438:in `plugin_hook'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util.rb:500:in `exit_on_fail'
/usr/lib/ruby/vendor_ruby/puppet/application.rb:346:in `run'
/usr/lib/ruby/vendor_ruby/puppet/util/command_line.rb:76:in `execute'
/usr/bin/puppet:10
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Here are the gems I have installed:

~$ sudo gem list
*** LOCAL GEMS ***
activemodel (3.0.11)
activerecord (3.0.11)
activesupport (3.0.11)
arel (2.0.10)
bigdecimal (1.1.0)
builder (2.1.2)
bundler (1.1.5)
charlock_holmes (0.6.8)
i18n (0.5.0)
io-console (0.3)
json (1.5.4)
minitest (2.5.1)
multi_json (1.3.6)
puppet-lint (0.2.1)
rake (0.9.2.2)
rdoc (3.9.4)
tzinfo (0.3.33)

#3 Updated by Anonymous over 3 years ago

Joe, are those gems installed on the agent or on the master?

#4 Updated by Joe Hillenbrand over 3 years ago

Both. I’m testing the agent and master on the same server.

#5 Updated by Josh Cooper over 3 years ago

OK, so some progress on this:

From the mailing list https://groups.google.com/d/topic/puppet-users/3tC0gYg9XA0/discussion Jonathan Gazeley wrote:

Both puppetmasters are running identical puppet configs. The old puppetmaster has these gems:

activemodel (3.2.8, 3.2.3, 3.0.9) 
activerecord (2.1.1) 
activesupport (3.2.8, 3.2.3, 3.0.9, 2.1.1) 
arel (3.0.2, 2.0.10) 
builder (3.0.0, 2.1.2) 
hiera (1.0.0) 
i18n (0.6.0, 0.5.0) 
multi_json (1.3.2) 
rake (0.8.7) 
tzinfo (0.3.33, 0.3.29) 

The new puppetmaster has these:

activerecord (2.1.1) 
activesupport (2.1.1) 

Using inconsistent activesupport/activerecord gems is definitely a problem. I can reproduce the error on ubuntu 10.10:

# cat /etc/puppet/puppet.conf
[rails]
rails_loglevel=debug

[master]
storeconfigs=true
dbadapter=sqlite3

# puppet --version
3.0.0
# ruby --version
ruby 1.8.7 (2010-06-23 patchlevel 299) [x86_64-linux]
# gem list

*** LOCAL GEMS ***

activemodel (3.2.8)
activerecord (2.1.1)
activesupport (3.2.8, 2.1.1)
builder (3.0.3)
i18n (0.6.1)
multi_json (1.3.6)

Then initiate an agent run, and I see this in the master’s syslog:

# tail -f /var/log/syslog
...
Oct  9 21:36:14 ubuntu1010 puppet-master[8914]: Could not autoload puppet/indirector/node/active_record: uninitialized constant ActiveRecord

Removing all gems and installing libactiverecord-ruby solves the problem:

# gem uninstall activemodel
# gem uninstall activerecord
# gem uninstall activesupport
# gem uninstall builder
# gem uninstall i18n
# gem uninstall multi_json
# apt-get install libactiverecord-ruby
...
Errors were encountered while processing:
 puppetmaster

The install didn’t restart puppetmaster correctly. So stopped puppetmaster, re-did install, and it started puppetmaster:

# /etc/init.d/puppetmaster stop
 * Stopping puppet master                                                [ OK ] 
# apt-get install libactiverecord-ruby
Setting up puppetmaster (3.0.0-1puppetlabs1) ...
 * Starting puppet master                                                [ OK ] 

And now storeconfigs are working as expected:

Oct  9 21:39:39 ubuntu1010 puppet-master[9616]: Connecting to sqlite3 database: /var/lib/puppet/state/clientconfigs.sqlite3
Oct  9 21:39:39 ubuntu1010 puppet-master[9616]: Using cached facts for localagent
Oct  9 21:39:39 ubuntu1010 puppet-master[9616]: Using cached node for localagent
Oct  9 21:39:39 ubuntu1010 puppet-master[9616]: Warning: calling Plugins.search with empty module path.
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Using cached facts for localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Expiring the node cache of localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Using cached facts for localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Using cached node for localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Compiled catalog for localagent in environment production in 0.02 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Caching catalog for localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Searched for resources in 0.01 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Searched for resource params and tags in 0.00 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Resource removal in 0.00 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Resource merger in 0.03 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Resource addition in 0.00 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Performed resource comparison in 0.03 seconds
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Using cached facts for localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Using cached node for localagent
Oct  9 21:39:40 ubuntu1010 puppet-master[9616]: Saved catalog to database in 0.09 seconds

Separately, Andy and I saw the same error when running:

ruby 1.9.3p125 
activerecord 2.2.3
activesupport 2.2.3

However, when using ruby 1.8.7 with the libactiverecord-ruby package, the activerecord, activesupport, etc code is placed directly in the ruby source, e.g. /usr/lib/ruby/1.8/activerecord.rb, and so ruby will ignore whatever gems you may have installed, even if those gems are inconsistent.

Joe/Jonathan, can you guys remove all of your gems and install the package containing the ruby activerecord bindings (for me it was libactiverecord-ruby), and see if the problem is resolved. Also check which version of ruby puppet is using.

#6 Updated by Anonymous over 3 years ago

  • Target version changed from 3.0.1 to 3.0.x

This didn’t make the cut for 3.0.1. I’m retargeting it at 3.0.x.

Note: at the moment it isn’t clear that there is an issue here that we can fix.

#7 Updated by Anonymous over 3 years ago

Joe, Jonathan,

Josh updated this issue with a bunch of information on the 9th, but it was down at the bottom of the email and might have been easily overlooked.

Could you please try the following and let us know if it resolved this issue?

Joe/Jonathan, can you guys remove all of your gems and install the package containing the ruby activerecord bindings (for me it was libactiverecord-ruby), and see if the problem is resolved. Also check which version of ruby puppet is using.

Thanks, -Jeff

#8 Updated by Joe Hillenbrand over 3 years ago

I can’t test this because I’m really happy with PuppetDB and I don’t want to mess with it.

Sorry.

#9 Updated by Jonathan Gazeley over 3 years ago

I’ll try that as soon as I can. Unfortunately we are running it in a production environment and I currently don’t have authorisation to “break” it again. As soon as I can get clearance, I will test the instructions above.

Cheers, Jonathan

#10 Updated by Josh Cooper over 3 years ago

  • Status changed from Needs More Information to Closed

Joe, Jonathan, we haven’t received any more reports on this issue, so we’re pretty confident it’s a rubygems configuration issue. I’m going to close this ticket for now, but if you run into the problem, feel free to reopen.

#11 Updated by aric gardner over 3 years ago

  • Status changed from Closed to Re-opened

Same error here, fresh install of puppet master on a virtual machine (for the purpose of testing 3.0.x) Just running puppetmaster from webrick as there is no need for scale while testing

Note: Puppet runs as expected if I disable storeconfigs

  root@puppet:/etc/puppet# puppet master --debug --no-daemonize --verbose
  <snip>
  Error: Cached catalog for pgmaster failed: Could not autoload puppet/indirector/catalog/active_record: uninitialized constant ActiveRecord
  Error: Could not autoload puppet/indirector/facts/active_record: uninitialized constant ActiveRecord
  <snip>

OS

  DISTRIB_DESCRIPTION="Ubuntu 12.04.1 LTS"  

Sources.list

# Puppetlabs products 
deb http://apt.puppetlabs.com precise main 
deb-src http://apt.puppetlabs.com precise main

Installed packages

ii  facter                           1.6.13-1puppetlabs1          Ruby module for collecting simple facts about a host operating system
ii  hiera                            1.1.0-1puppetlabs1           A simple pluggable Hierarchical Database.
ii  puppet                           3.0.1-1puppetlabs1           Centralized configuration management - agent startup and compatibility scripts
ii  puppet-common                    3.0.1-1puppetlabs1           Centralized configuration management
ii  puppetlabs-release               1.0-5                        "Package to install Puppet Labs gpg key and apt repo"
ii  puppetmaster                     3.0.1-1puppetlabs1           Centralized configuration management - master startup and compatibility scripts
ii  puppetmaster-common              3.0.1-1puppetlabs1           Puppet master common scripts
ii  libaugeas-ruby1.8                0.3.0-1.1ubuntu4             Augeas bindings for the Ruby language
ii  libjson-ruby                     1.6.3-1                      Transitional package for ruby-json
ii  libruby                          4.8                          Transitional package for libruby1.8
ii  libruby1.8                       1.8.7.352-2ubuntu1.1         Libraries necessary to run Ruby 1.8
ii  libshadow-ruby1.8                1.4.1-8build1                Interface of shadow password for Ruby 1.8
ii  ruby                             4.8                          Transitional package for ruby1.8
ii  ruby-json                        1.6.3-1                      JSON library for Ruby
ii  ruby1.8                          1.8.7.352-2ubuntu1.1         Interpreter of object-oriented scripting language Ruby 1.8

Installing libactiverecord-ruby

root@puppet:/# apt-get install libactiverecord-ruby
root@puppet:/etc/puppet# puppet master --debug --no-daemonize --verbose
<snip>
Error: no such file to load -- mysql
Debug: No plugins found in subpath '/etc/puppet/modules/rabbitmq/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/etc/puppet/modules/auto_update_security/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/etc/puppet/modules/redis/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/etc/puppet/modules/standardize_admin_environment/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/etc/puppet/modules/iptables/lib' (file / directory does not exist)
Debug: No plugins found in subpath '/etc/puppet/modules/keepalived/lib' (file / directory does not exist)
Info: Connecting to mysql database: puppet
!!! The bundled mysql.rb driver has been removed from Rails 2.2. Please install the mysql gem and try again: gem install mysql.
<snip>

So…

apt-get install ruby gems

88MB later

root@puppet:/etc/puppet# gem install mysql
Fetching: mysql-2.8.1.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing mysql:
    ERROR: Failed to build gem native extension.

    /usr/bin/ruby1.8 extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of
necessary libraries and/or headers.  Check the mkmf.log file for more
details.  You may need configuration options.

     Provided configuration options:
    --with-opt-dir
    --without-opt-dir
    --with-opt-include
    --without-opt-include=${opt-dir}/include
    --with-opt-lib
    --without-opt-lib=${opt-dir}/lib
    --with-make-prog
    --without-make-prog
    --srcdir=.
    --curdir
    --ruby=/usr/bin/ruby1.8
    --with-mysql-config
    --without-mysql-config
    --with-mysql-dir
    --without-mysql-dir
    --with-mysql-include
    --without-mysql-include=${mysql-dir}/include
    --with-mysql-lib
    --without-mysql-lib=${mysql-dir}/lib
    --with-mysqlclientlib
    --without-mysqlclientlib
    --with-mlib
    --without-mlib
    --with-mysqlclientlib
    --without-mysqlclientlib
    --with-zlib
    --without-zlib
    --with-mysqlclientlib
    --without-mysqlclientlib
    --with-socketlib
    --without-socketlib
    --with-mysqlclientlib
    --without-mysqlclientlib
    --with-nsllib
    --without-nsllib
    --with-mysqlclientlib
    --without-mysqlclientlib
    --with-mygcclib
    --without-mygcclib
    --with-mysqlclientlib
    --without-mysqlclientlib


  Gem files will remain installed in /var/lib/gems/1.8/gems/mysql-2.8.1 for inspection.
  Results logged to /var/lib/gems/1.8/gems/mysql-2.8.1/ext/mysql_api/gem_make.out

Being friday afternoon, I’m going to commit this report and pick it up again on monday.

#12 Updated by Josh Cooper over 3 years ago

Hi Aric, what’s the output of gem list?

#13 Updated by aric gardner over 3 years ago

Josh Cooper wrote:

Hi Aric, what’s the output of gem list?

None, gem install mysql failed.

 root@puppet:/etc/puppet# gem list
 *** lOCAL GEMS ***

#14 Updated by aric gardner over 3 years ago

aric gardner wrote:

Josh Cooper wrote:

Hi Aric, what’s the output of gem list?

None, gem install mysql failed.

 root@puppet:/etc/puppet# gem list
 *** LOCAL GEMS ***
 apt-get install libmysql-ruby libmysqlclient-dev

Fixed my issue.

Also I see that

  Warning: ActiveRecord-based storeconfigs and inventory are deprecated. See http://links.puppetlabs.com/activerecord-deprecation
    (at /usr/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `new')          

I guess I’ll be moving to puppetdb in any case.

#15 Updated by Josh Cooper over 3 years ago

  • Status changed from Re-opened to Closed

Closing this again as note-11 was resolved in note-14.

#16 Updated by nabeel shahzad over 3 years ago

Is this something that can be resolved by installing/using PuppetDB?

#17 Updated by Anonymous over 3 years ago

nabeel shahzad wrote:

Is this something that can be resolved by installing/using PuppetDB?

Yes, we have reports that PuppetDB is a viable workaround and alternative approach that avoids this issue.

Hope this helps, -Jeff

#18 Updated by Patrick Hemmer over 3 years ago

Jeff McCune wrote:

Yes, we have reports that PuppetDB is a viable workaround and alternative approach that avoids this issue.

I’m using PuppetDB in 2.7 and any attempts to upgrade puppet to 3.0 results in this error…

#19 Updated by Anonymous over 3 years ago

Patrick Hemmer wrote:

Jeff McCune wrote:

Yes, we have reports that PuppetDB is a viable workaround and alternative approach that avoids this issue.

I’m using PuppetDB in 2.7 and any attempts to upgrade puppet to 3.0 results in this error…

Could you please paste the output of the puppet run that produces the error with the --trace and --evaltrace flags enabled? (Protip: <pre> </pre> tags in Redmine are a good way to wrap terminal output).

-Jeff

#20 Updated by Josh Cooper over 3 years ago

Patrick Hemmer wrote:

Jeff McCune wrote:

Yes, we have reports that PuppetDB is a viable workaround and alternative approach that avoids this issue.

I’m using PuppetDB in 2.7 and any attempts to upgrade puppet to 3.0 results in this error…

Please also see my comments in note-5. This issue is likely due to mismatched activerecord/activesupport gems. Also if you’re using storeconfigs with mysql, see the comment in note-14 to make sure you have the right ruby-mysql bindings.

#21 Updated by Anonymous over 3 years ago

  • Target version deleted (3.0.x)

#22 Updated by Patrick Hemmer about 3 years ago

Ok, so I finally took the time to upgrade from 2.7 to 3.1, and I did find the cause of this issue.

The cause is a bug in regards to the agent storing the catalog and fact cache. It looks like the default caching type changed between 2.7 and 3.0, and both catalogs and facts are cached by the agent in a local database. Since my system does not have the ActiveRecord or ActiveSupport gems, puppet was failing when trying to access the cache. After adding the following entries to the routes.yaml file, everything started working.

agent:
  facts:
    cache: yaml
  catalog:
    cache: json

Additionally the man page for puppet.conf states that catalog_cache_terminus controls how the catalog is stored. This is incorrect. Changing this parameter has absolutely no effect. puppet agent --configprint catalog_cache_terminus always shows store_configs, even if I set it to something else (like json). The only way to update the cache type was using routes.yaml.

Also available in: Atom PDF