The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #21376

Stack level too deep after updating from 3.1.1 to 3.2.2

Added by Peter Meier almost 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:
Priority:HighDue date:
Assignee:Patrick Carlisle% Done:

0%

Category:-
Target version:3.2.3
Affected Puppet version:3.2.1 Branch:https://github.com/puppetlabs/puppet/pull/1717
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

After updating my master from 3.1.1 to 3.2.2 I started getting failing clients with stack level too deep. These failures happen rather randomly and only after a while. So usually after the master is running for a while it will start failing for certain nodes.

The error I get is

    stack level too deep
    /usr/lib/ruby/site_ruby/1.8/puppet/util/errors.rb:23:in `adderrorcontext'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast.rb:64:in `safeevaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:15:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `each'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast.rb:62:in `safeevaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:15:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `each'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `evaluate'
     
    [...]
     
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:15:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `each'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast.rb:62:in `safeevaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:15:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `each'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast.rb:62:in `safeevaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:15:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `each'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast/block_expression.rb:10:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/ast.rb:62:in `safeevaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/resource/type.rb:137:in `evaluate_code'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/resource.rb:81:in `evaluate'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:293:in `evaluate_main'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:97:in `compile'
    /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler/none.rb:6:in `profile'
    /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:26:in `profile'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:97:in `compile'
    /usr/lib/ruby/site_ruby/1.8/puppet/parser/compiler.rb:29:in `compile'
    /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:86:in `compile'
    /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler/none.rb:6:in `profile'
    /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:26:in `profile'
    /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:84:in `compile'
    /usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/compiler.rb:44:in `find'
    /usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in `find'
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:128:in `do_find'
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:88:in `send'
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:88:in `process'
    /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler/none.rb:6:in `profile'
    /usr/lib/ruby/site_ruby/1.8/puppet/util/profiler.rb:26:in `profile'
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:82:in `process'
    /usr/lib/ruby/site_ruby/1.8/puppet/network/http/rack.rb:21:in `call'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/rack/request_handler.rb:96:in `process_request'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:516:in `accept_and_process_next_request'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_request_handler.rb:274:in `main_loop'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:206:in `start_request_handler'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:171:in `send'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:171:in `handle_spawn_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/utils.rb:470:in `safe_fork'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:166:in `handle_spawn_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `__send__'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:180:in `start'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/rack/application_spawner.rb:129:in `start'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:253:in `spawn_rack_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server_collection.rb:132:in `lookup_or_add'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:246:in `spawn_rack_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server_collection.rb:82:in `synchronize'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server_collection.rb:79:in `synchronize'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:244:in `spawn_rack_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:137:in `spawn_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/spawn_manager.rb:275:in `handle_spawn_application'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `__send__'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:357:in `server_main_loop'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/lib/phusion_passenger/abstract_server.rb:206:in `start_synchronously'
    /usr/lib/ruby/gems/1.8/gems/passenger-3.0.17/helper-scripts/passenger-spawn-server:99

I tried to catch the resource that is being evaluated in /usr/lib/ruby/site_ruby/1.8/puppet/resource/type.rb before it fails and it has always been Class[main].

It looks like it starts failing after various nodes have asked for their catalog and hence many different modules have been loaded.

With the help from #puppet-dev I started debugging this issue. People from there will add more information.

I give it a high priority as this is something that worked on 3.1.1 and fails heavily on 3.2.2 so 3.2.2 is currently not useable for me, but it’s the only version with the latest security fix for 3.x.


Related issues

Related to Puppet - Bug #21869: another "Error: Could not request certificate: stack leve... Merged - Pending Release
Duplicated by Puppet - Bug #20632: Puppet 3.2.0rc2 "stack level too deep" Duplicate
Duplicated by Puppet - Bug #21442: Agent runs fail after 3.2.2 upgrade - a problem with impo... Duplicate

History

#1 Updated by Anonymous almost 3 years ago

After looking into this, it looks like the recursion is occuring from repeated calls into BlockExpression, which has been setup by calls to Puppet::Resource::Type#merge (https://github.com/puppetlabs/puppet/blob/53241a91c687336a273291aa6e71f90fc7752581/lib/puppet/resource/type.rb#L194). Repeated calls to #merge will result in larger and larger chains of recursive calls in BlockExpression. It appears that #merge is called in Puppet::Resource::TypeCollection#add (https://github.com/puppetlabs/puppet/blob/2624eee7b3a6443700a4984e72782735e08192c4/lib/puppet/resource/type_collection.rb#L46).

I think that what is occurring is that whenever a manifest is loaded a new Class[main] is created and then merged into the existing Class[main]. When Class[main] gets evaluated, then we end up with a stack overflow if there have been enough manifests loaded. This has not yet been confirmed.

#2 Updated by Patrick Carlisle almost 3 years ago

  • Status changed from Unreviewed to In Topic Branch Pending Review
  • Assignee set to Peter Meier
  • Branch set to https://github.com/puppetlabs/puppet/pull/1717

Peter, can you give this patch a try?

#3 Updated by Anonymous almost 3 years ago

As an update on this. The way that we reproduced the problem was to create a module that had 10000 pp files in it, where each contained:

$b = "hello"
class b::b {}

Using an include of all of those classes didn’t trigger the problem. However, using an import "b/*" did. The difference turns out to be when the Class[main] is merged and evaluated. With import all of the merging happens early one during parsing and then when the compiler calls evaluate_main it blows up. However include is lazy about it and so Class[main] is small when first evaluated and then grows, but isn’t evaluated the same way again.

#4 Updated by Peter Meier almost 3 years ago

  • Assignee changed from Peter Meier to Patrick Carlisle

I tested the patch and so far it works great.

#5 Updated by Anonymous almost 3 years ago

  • Status changed from In Topic Branch Pending Review to Merged - Pending Release
  • Target version set to 3.2.3
  • Affected Puppet version set to 3.2.1

#6 Updated by Matthaus Owens almost 3 years ago

  • Status changed from Merged - Pending Release to Closed

Released in Puppet 3.2.3-rc1.

#7 Updated by Ilkka Tengvall almost 3 years ago

I hit the problem still using puppet-server-3.2.3-1.el6.noarch .

#8 Updated by Erik Dalén almost 3 years ago

I hit this bug using the resource_type API in 3.2.2, but it works now with 3.2.3.

So just a note that this also affected that and the fix was the same.

http://docs.puppetlabs.com/guides/rest_api.html#resource-types

Also available in: Atom PDF