The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #9290

Puppet master fails with 'stack level too deep' error when storeconfigs = true with rails stack 3.1.0

Added by Mark Stanislav over 2 years ago. Updated 4 months ago.

Status:AcceptedStart date:08/31/2011
Priority:HighDue date:
Assignee:eric sorenson% Done:

0%

Category:Rails
Target version:3.1.x
Affected Puppet version:2.7.3 Branch:
Keywords:storeconfigs rails activerecord 3.1.0 customer

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1064


Description

Out of nowhere, a known-working Puppet stack build script started to fail. The tell-tale error is ‘stack level too deep’ which from other historical bug reports always seems related to Ruby directly. After many hours of digging around, I checked for updated gem versions on the system. There was an update to rails 3.1.0 (and activerecord, etc.) which apparently is breaking the Puppet master when working with storeconfigs = true. Disabling storeconfigs immediately works as expected again.

When reverting from rails and friends 3.1.0 to 3.0.10, Puppet again works as expected.

I don’t have any insight into where exactly this is all failing, only the version which is causing the issue and the condition to emulate it.

This was only tested again Puppet 2.7.3 using the gem install with both webrick and Apache+Passenger.

Attached is a debug output.

error-trace (11.4 KB) Mark Stanislav, 08/31/2011 04:33 pm

master-trace.txt Magnifier (285 KB) Markus Strauss, 03/04/2012 10:30 pm

resource-fix.diff Magnifier - don't look up parameter anywhere special if it is 'id'. (356 Bytes) jared jennings, 07/24/2012 11:15 am


Related issues

Related to Puppet - Bug #9304: Problem with activerecord 3.1.0, puppet 2.6.2, and storec... Accepted 09/01/2011
Related to Puppet - Bug #11883: Spec tests are failing when you have Rails 3.1.3 installed Accepted 01/10/2012

History

#1 Updated by James Turnbull over 2 years ago

  • Category set to Rails
  • Keywords changed from storeconfigs rails activerecord to storeconfigs rails activerecord 3.1.0

#2 Updated by James Turnbull over 2 years ago

  • Status changed from Unreviewed to Accepted
  • Priority changed from Normal to High

#3 Updated by Carl Lantz over 2 years ago

I am having the same error using the git repository as of today. Downgrading to activerecord 3.0.10 fixed the issue. It errored with both webrick and apache+passenger. The following was repeated several times with webrick:

/usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:177:in parameter' /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:87:in[]‘ /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:81:in add_constraints' /usr/lib/ruby/gems/1.8/gems/activemodel-3.1.0/lib/active_model/attribute_methods.rb:102:ineach_with_index’ /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:55:in each' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:55:ineach_with_index' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:55:in add_constraints' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association_scope.rb:33:inscope' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association.rb:99:in association_scope' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/association.rb:88:inscoped' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/collection_proxy.rb:51:in __send__' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/collection_proxy.rb:51:inscoped' /usr/lib/ruby/gems/1.8/gems/activerecord-3.1.0/lib/active_record/associations/collection_proxy.rb:102:in method_missing' /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:177:inparameter' /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:87:in `[]'

#4 Updated by Tray Torrance over 2 years ago

Confirmed for me – Puppet 2.7.3, Unicorn 4.1.1

Downgrading from activerecord 3.1.0 to 3.0.10 resolved it here, as well. Will try to test on 2.7.5 this weekend.

#5 Updated by Jan Hebler over 2 years ago

Same Error with 2.7.5

#6 Updated by Joshua Hoblitt over 2 years ago

Same error with 2.7.6

err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep

#7 Updated by Justin Honold over 2 years ago

Also ran into this one, thx for the downgrade tip.

#8 Updated by Keiran S over 2 years ago

I hit this also, downgrading to activerecord 3.0.10 and its related dependencies resolved it for me.

K

#9 Updated by Anchor Systems over 2 years ago

Another “it’s broken” observation here — Puppet 2.7.6, infinite loop with ARec 3.1.1, works fine with 3.0.11. Might be worth at least putting a version constraint on ARec at load time, so people don’t get bitten, until the problem can be found and fixed properly.

#10 Updated by Jordan Sissel about 2 years ago

Confirmed still in ActiveRecord 3.2.0

#11 Updated by Jonas Genannt about 2 years ago

If you hvae more than one Rails version installed, you can force an older activerecord version to puppet:

gem ‘activerecord’, ‘=2.2.2’ require ‘activerecord’

Add this into your config.ru file.

#12 Updated by Anthony Caiafa about 2 years ago

looks like the same is happening with

puppet-server-2.7.1-1

activerecord (3.2.1)

#13 Updated by Cody Robertson about 2 years ago

I’m also experiencing this after upgrading to activerecord 3.2.1. I’ve found the newest branch that works is 3.0.x (3.0.11 as of this comment). The 3.1.x and 3.2.x seem to produce this error.

gem install activerecord -v 3.0.11

#14 Updated by Daniel Pittman about 2 years ago

If anyone experiencing this problem could test out the code in https://github.com/puppetlabs/puppet/pull/497 and see if it makes any difference, that would be awesome.

#15 Updated by Markus Strauss about 2 years ago

I have applied the patch for lib/puppet/rails/inventory_node.rb from pull req. 497, but still, same problem on my old Debian Lenny. See the attached trace.

#16 Updated by Daniel Pittman about 2 years ago

Well, it was a hope. Thanks for confirming this is still a problem.

#17 Updated by Lluís Gili almost 2 years ago

the workaround proposed at note 11 made my rack crash, instead I’ve modified lib/puppet/feature/rails.rb:

 Puppet.features.add(:rails) do
   begin
+    # http://projects.puppetlabs.com/issues/9290
+    gem 'activerecord', '=3.0.6'
     require 'active_record'
     require 'active_record/version'

#18 Updated by Patrick Buckley almost 2 years ago

With the sql injection vuln in older versions of active record 3.x this has become an issue for us.

activerecord 3.2.5 puppet 2.7.14 puppet dashboard 1.2.8

puppetmaster

notice: Compiled catalog for in environment production in 4.45 seconds info: Caching catalog for debug: Searched for resources in 0.00 seconds debug: Searched for resource params and tags in 0.00 seconds debug: Resource removal in 0.00 seconds debug: Resource merger in 0.00 seconds err: stack level too deep

puppet agent

err: Could not retrieve catalog from remote server: Error 400 on SERVER: stack level too deep warning: Not using cache on failed catalog err: Could not retrieve catalog; skipping run

Another relevant note this doesn’t happen on all of our nodes. Only on nodes created after the active record upgrade so far.

#19 Updated by Patrick Buckley almost 2 years ago

Also editing /usr/lib/ruby/1.8/puppet/feature/rails.rb and making the change above except point it at active record 3.0.13 fixed the problem for me.

#20 Updated by Mike Shuey almost 2 years ago

FYI, I ran into this problem today on a FreeBSD host, using puppet 2.7.16 and activerecord 3.2.5 and 3.2.6. Installing activerecord 3.0.15, and adding that 1-line patch to force the older-version gem, fixed things. Without this workaround, any use of @@syntax in a config would cause a thread on the server to consume an entire core, recurse infinitely, consume scads of memory, and eventually bomb out with a “stack too deep” message. With the workaround, things behave as you’d expect.

#21 Updated by Owen Mehegan almost 2 years ago

Just got bitten by this again with an accidental ActiveRecord gem update. Is anyone looking into this?

#22 Updated by Jeremy Cole over 1 year ago

Using Puppet 2.7.18 and ruby 1.8.7 on the puppetmaster and 2.6.16 on the puppet client. Hit this bug immediately after enabling stored configs. Installed activerecord version 3.0.11 and now have 3 versions intalled (I have other rails apps which need 3.2.1):

$gem list | grep activerecord
activerecord (3.2.1, 3.0.11, 2.3.14)

According to the fix in msg 11 above I need to add “gem ‘activerecord’, ‘=3.0.11’ require ‘activerecord’” to my config.ru file. I have no idea which config.ru file to add this to nor where in the file to place this line. Which file exactly do I need to edit and it would be much appreciated if anyone could give me an indication of which line to replace. Msg 19 seems to indicate that I need to edit /usr/lib/ruby/1.8/puppet/feature/rails.rb however I only have /usr/lib/ruby/site_ruby/1.8/puppet/feature/rails.rb which I assume is the same. Again not sure what to replace.

Any help would be greatly appreciated as we’re stuck until we can resolve this.

#23 Updated by Jeffery Mires over 1 year ago

Jeremy Cole wrote:

Using Puppet 2.7.18 and ruby 1.8.7 on the puppetmaster and 2.6.16 on the puppet client. Hit this bug immediately after enabling stored configs. Installed activerecord version 3.0.11 and now have 3 versions intalled (I have other rails apps which need 3.2.1):

$gem list | grep activerecord
activerecord (3.2.1, 3.0.11, 2.3.14)

According to the fix in msg 11 above I need to add “gem ‘activerecord’, ‘=3.0.11’ require ‘activerecord’” to my config.ru file. I have no idea which config.ru file to add this to nor where in the file to place this line. Which file exactly do I need to edit and it would be much appreciated if anyone could give me an indication of which line to replace. Msg 19 seems to indicate that I need to edit /usr/lib/ruby/1.8/puppet/feature/rails.rb however I only have /usr/lib/ruby/site_ruby/1.8/puppet/feature/rails.rb which I assume is the same. Again not sure what to replace.

Any help would be greatly appreciated as we’re stuck until we can resolve this.

Hey Jeremy,

There should already be everything besides these two lines,


   +    # http://projects.puppetlabs.com/issues/9290
   +    gem 'activerecord', '=3.0.11'

be sure to add them between existing lines like so


Puppet.features.add(:rails) do
   begin
   +    # http://projects.puppetlabs.com/issues/9290
   +    gem 'activerecord', '=3.0.11'
   require 'active_record'
   require 'active_record/version'

#24 Updated by jared jennings over 1 year ago

I had this same problem, with Puppet 2.7.6 and Rails 3.2.6. I tried downgrading Rails, but it didn’t initially work, and it’s been a long time since Rails 3.0.x, hasn’t it? So I decided to find out what the problem was.

By adding a print statement to the Puppet::Rails::Resource.parameter method, I found that the parameter causing the infinite loop was id. After staring at the backtrace for a long time, I thought it was odd that after fifteen calls inside ActiveRecord it would suddenly call back into Puppet:

  ...
  /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:186:in `parameter'
  /usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:95:in `[]'
  /usr/lib/ruby/gems/1.8/gems/activerecord-3.2.6/lib/active_record/associations/association_scope.rb:70:in `add_constraints'
  /usr/lib/ruby/gems/1.8/gems/activesupport-3.2.6/lib/active_support/dependencies.rb:202:in `each_with_index'
  ...

So I changed the [] method as in the attached diff. Everything appeared to suddenly work, and my node began configuring itself. The node having succeeded once, it kept working even after I put the code back to the original, until I nuked the storedconfig database. Perhaps this is a clue as to the nature of the bug.

I don’t fully understand the change I’ve made. I haven’t run any of the tests. I haven’t made a pull request, because I can’t push to GitHub from the network I’m on. Maybe later.

The code I changed originated in commit:6314745c three years ago, and still exists in the master branch today, so this patch would probably apply to any recent Puppet version.

#25 Updated by Tim Bishop over 1 year ago

I can confirm the issue on FreeBSD with puppet 2.7.19 and rails 3.2.8.

I can also confirm that Jared’s workaround in the previous update fixes it. It’d be great if somebody familiar with that code could check it and commit it if appropriate (or feedback if it’s not).

#26 Updated by Daniel Friesen over 1 year ago

I’d really like to see this fixed. I’m using the absolute latest stable software; puppet 3.0.0 and activerecord 3.2.8 and a perfectly reasonable configuration is completely broken…

#27 Updated by Tim Bishop over 1 year ago

  • Affected Puppet version changed from 2.7.3 to 3.0.1

Still seeing this with Puppet 3.0.1. Fix in comment 24 still works.

#28 Updated by Josh Cooper over 1 year ago

  • Assignee set to eric sorenson
  • Affected Puppet version changed from 3.0.1 to 2.7.3

Hi Tim, we try to use the affected field to keep track of when the bug was first encountered, but it’s “good” to hear that this is still affecting you in 3.0.1.

#29 Updated by Tim Bishop over 1 year ago

Sorry, my mistake. I assumed the opposite; that it was the most recent version. But upon reflection, it makes a lot more sense for it to be the first version :–)

#30 Updated by Spenser Gilliland over 1 year ago

Running into this issue with 2.6.17 from EPEL. Switching to activerecord 3.0.11 from gem allowed me to work around the issue.

#31 Updated by Ken Johnson over 1 year ago

We’ve received a ticket indicating that a commercial customer has run into this as well, and that the patch attached successfully resolved the issue for them. Version information they provided is Puppet 2.7.18, Fedora ~18, using mod_passenger and Rails 3.2.8.

#32 Updated by Ingo Rohlfs about 1 year ago

  • Target version set to 3.1.x
  • Support Urls deleted (https://support.puppetlabs.com/tickets/1819)

Iam on Puppet Version 3.1.1 and still have the problem with activerecord 3.2.13

#33 Updated by Charlie Sharpsteen about 1 year ago

  • Keywords changed from storeconfigs rails activerecord 3.1.0 to storeconfigs rails activerecord 3.1.0 customer

#35 Updated by Jonathan Furrer 11 months ago

  • Support Urls deleted (https://support.puppetlabs.com/tickets/1819)

I can confirm that the attached patch worked for me with puppet 3.2.1 and rails 3.2.13 with same versioned active* gems

#36 Updated by Nacho Barrientos 10 months ago

  • Support Urls deleted (https://support.puppetlabs.com/tickets/1819)

Hi,

We’re also seeing this problem with Puppet 3.2.2 (EL6) and Activerecord 3.0.11 provided by a gem :(

Could not retrieve catalog from remote server: Error 400 on SERVER: Puppet::Parser::AST::Variable failed with error SystemStackError: stack level too deep at >etc/puppet/environments/devel/modules/foo/manifests/bar.pp:11 on node node.example.com

The attached patch didn’t help either.

#37 Updated by Nick Bausch 10 months ago

  • Support Urls deleted (https://support.puppetlabs.com/tickets/1819)

Hello,

Running puppet 2.7.6-1/mod_passenger-3.0.12 I was able to correct error this by freezing activerecord in rails.rb as suggested in comment 23

   +    # http://projects.puppetlabs.com/issues/9290
   +    gem 'activerecord', '=3.0.11'

After upgrading to puppet 2.7.22/mod_passenger-3.0.12 the error returned. Freezing activerecord using above lines in config.ru as suggested in comment 11 seems to have limped me along again.

Thanks,

Nick

#38 Updated by Nacho Barrientos 10 months ago

  • Support Urls deleted (https://support.puppetlabs.com/tickets/1819)

In our particular case, this problem was found when migrating from Puppet 2.7.18 to Puppet 3.2.2.

Apparently the situation improves for a short period of time when we restart the masters. We noticed it when we tried to apply the patch attached above. Basically we applied it, restarted all our masters backend and we didn’t see any “stack level too deep” for a while (~1h hour) so we thought it was fixed. However, later on we began to experience the problem again.

In short:

  • Patch in comment#1 didn’t work.
  • Forcing activerecord version didn’t work either (we use puppetdb, though) — tried with 3.0.11 and 3.0.15. We’re now at 2.3.16.
  • Reducing the number of deprecation warnings like [1] alleviated a bit the number of 400s we were getting.

Finally, we were forced to roll back to Puppet 2 :(

[1]

Variable access via ‘serveradmin’ is deprecated. Use ‘@serveradmin’ instead. template[/etc/puppet/environments/devel/modules/apache/templates/httpd.conf.erb]:181

#39 Updated by Charlie Sharpsteen 10 months ago

Nacho,

I believe what you are seeing after upgrading to 3.2.2 is #21376. This issue has been patched and a fix will be released in 3.2.3.

#40 Updated by Charlie Sharpsteen 10 months ago

  • Support Urls deleted (https://support.puppetlabs.com/tickets/1819)

#41 Updated by Nacho Barrientos 10 months ago

Thanks for the heads-up. Eager to give 3.2.3 a spin! :)

#42 Updated by Darin Perusich 7 months ago

I’m seeing this on OpenSUSE 12.3 with the disto provided packages.

puppet-3.0.2 rubygem-activerecord-3_2-3.2.12 rubygem-mysql2-0.3.11

#43 Updated by Darin Perusich 7 months ago

Darin Perusich wrote:

I’m seeing this on OpenSUSE 12.3 with the disto provided packages.

puppet-3.0.2 rubygem-activerecord-3_2-3.2.12 rubygem-mysql2-0.3.11

I just upgrade to puppet 3.2.4, from systemsmanagement:puppet repo, still seeing getting “stack level too deep”. Same version of activerecord though.

http://download.opensuse.org/repositories/systemsmanagement:/puppet/openSUSE_12.3/

#44 Updated by Tim Bishop 5 months ago

Still seeing this in 3.3.1. The resource-fix.diff patch still solves the problem. Please could it be applied?

#45 Updated by Jason Antman 4 months ago

Redmine Issue #9290 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1064

Also available in: Atom PDF