The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #4602

TypeError with storeconfigs

Added by Ricky Zhou about 4 years ago. Updated almost 4 years ago.

Status:ClosedStart date:08/24/2010
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-
Affected Puppet version: Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.


Description

I get the following traceback on a puppet server with storeconfigs enabled (using puppet-2.6.1-0.2.rc2.fc13.noarch.rpm from tmz’s repo):

/usr/lib/ruby/1.8/yaml.rb:391:in `emit'
/usr/lib/ruby/1.8/yaml.rb:391:in `quick_emit'
/usr/lib/ruby/1.8/yaml/types.rb:48:in `to_yaml'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/quoting.rb:31:in `quote'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/mysql_adapter.rb: 236:in `quote'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2992:in `attributes_with_quotes'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2983:in `each'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2983:in `attributes_with_quotes'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2898:in `create_without_timestamps'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/timestamp.rb:53:in `create_without_callbacks'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:266:in `create'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2874:in `create_or_update_without_callbacks'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:250:in `create_or_update'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2538:in `save_without_validation'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/validations.rb:1078:in `save_without_dirty'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/dirty.rb:79:in `save_without_transactions'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `send'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `with_transaction_returning_status'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `transaction'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:182:in `transaction'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:228:in `with_transaction_returning_status'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:208:in `rollback_active_record_state!'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/has_many_association.rb:61:in `insert_record'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:145:in `send'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/associations/association_proxy.rb:145:in `send'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/autosave_association.rb:298:in `save_collection_association'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/autosave_association.rb:289:in `each'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/autosave_association.rb:289:in `save_collection_association'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/autosave_association.rb:168:in `autosave_associated_records_for_param_values'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:180:in `evaluate_method'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:180:in `evaluate_method'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:180:in `instance_eval'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:180:in `evaluate_method'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:166:in `call'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:93:in `run'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `each'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `send'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:92:in `run'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/callbacks.rb:276:in `run_callbacks'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:344:in `callback'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:283:in `update'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2874:in `create_or_update_without_callbacks'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/callbacks.rb:250:in `create_or_update'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/base.rb:2538:in `save_without_validation'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/validations.rb:1078:in `save_without_dirty'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/dirty.rb:79:in `save_without_transactions'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `send'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:229:in `with_transaction_returning_status'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/connection_adapters/abstract/database_statements.rb:136:in `transaction'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:182:in `transaction'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:228:in `with_transaction_returning_status'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:208:in `rollback_active_record_state!'
/usr/lib/ruby/gems/1.8/gems/activerecord-2.3.5/lib/active_record/transactions.rb:196:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/resource.rb:95:in `merge_parser_resource'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:253:in `perform_resource_merger'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:251:in `each'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:251:in `perform_resource_merger'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:197:in `compare_to_catalog'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:21:in `debug_benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:11:in `railsmark'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/core_ext/benchmark.rb:10:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:11:in `railsmark'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:21:in `debug_benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:196:in `compare_to_catalog'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:165:in `merge_resources'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:21:in `debug_benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:11:in `railsmark'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/core_ext/benchmark.rb:10:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:11:in `railsmark'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:21:in `debug_benchmark'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/host.rb:164:in `merge_resources'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/active_record.rb:30:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:11:in `railsmark'
/usr/lib/ruby/gems/1.8/gems/activesupport-2.3.5/lib/active_support/core_ext/benchmark.rb:10:in `realtime'
/usr/lib/ruby/site_ruby/1.8/puppet/rails/benchmark.rb:11:in `railsmark'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/catalog/active_record.rb:29:in `save'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector/indirection.rb:197:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/indirector.rb:50:in `find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:101:in `do_find'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `send'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/handler.rb:68:in `process'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick/rest.rb:23:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:104:in `service'
/usr/lib/ruby/1.8/webrick/httpserver.rb:65:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:45:in `listen'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `call'
/usr/lib/ruby/1.8/webrick/server.rb:173:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:162:in `start_thread'
/usr/lib/ruby/1.8/webrick/server.rb:95:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `each'
/usr/lib/ruby/1.8/webrick/server.rb:92:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:23:in `start'
/usr/lib/ruby/1.8/webrick/server.rb:82:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:42:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `initialize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `new'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:41:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `synchronize'
/usr/lib/ruby/site_ruby/1.8/puppet/network/http/webrick.rb:38:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:127:in `listen'
/usr/lib/ruby/site_ruby/1.8/puppet/network/server.rb:142:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/daemon.rb:125:in `start'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:114:in `main'
/usr/lib/ruby/site_ruby/1.8/puppet/application/master.rb:46:in `run_command'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:300:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:397:in `exit_on_fail'
/usr/lib/ruby/site_ruby/1.8/puppet/application.rb:300:in `run'
/usr/lib/ruby/site_ruby/1.8/puppet/util/command_line.rb:55:in `execute'
/usr/bin/puppet:4
err: wrong argument type String (expected Data)

For what it’s worth, I’ve seen this type of error on all versions starting from the 2.6.0 release candidates.


Related issues

Related to Puppet - Bug #4506: storing catalog in stored config produces: err: allocator... Closed 08/10/2010
Related to Puppet - Bug #4882: 2.6.1 can not deserialize Puppet::Resource::References Duplicate 09/29/2010

History

#1 Updated by Markus Roberts about 4 years ago

  • Status changed from Unreviewed to Investigating

I suspect that this is a duplicate of #4506, though that may not be obvious.

Could you check to see if your table has enormous values in some rows, remove those rows, and try again with the code in the testing branch (which contains the fix for #4506)?

#2 Updated by Markus Roberts about 4 years ago

Note that the fix for #4506 did not make it into the testing branch until this morning.

#3 Updated by Ricky Zhou about 4 years ago

  • Affected Puppet version deleted (2.6.1rc2)

Sorry for letting this bug sit, I have not seen this error since recreating the DB and updating to 2.6.1.

#4 Updated by James Turnbull about 4 years ago

  • Status changed from Investigating to Closed

Fixed in 2.6.1

#5 Updated by Andrew Forgue about 4 years ago

  • Status changed from Closed to Re-opened

I still get this error on 2.6.1 server even with a cleaned out database on both 2.6.1 clients and 0.25.5 clients.

It works for a while (maybe another host corrupts the storeconfig database?) and then dies on the client with:

err: Could not run Puppet configuration client: Parameter require failed: No title provided and "#<YAML::Object:0x2b28e6ed0578>" is not a valid resource reference

and on the server looks OK:

debug: Searched for resources in 0.00 seconds
debug: Searched for resource params and tags in 0.00 seconds
debug: Resource removal in 0.00 seconds
debug: Resource merger in 0.00 seconds
debug: Added resources(initialization) in 2.21 seconds
debug: Added resources(tags) in 1.04 seconds
debug: Added resources(parameters) in 4.68 seconds
debug: Resource addition in 25.93 seconds
debug: Performed resource comparison in 25.93 seconds
debug: Using cached node for lab-auxlin01.ecs.apple.com
debug: Saved catalog to database in 25.94 seconds

And then subsequent runs produce:

err: Could not retrieve catalog from remote server: Error 400 on SERVER: wrong argument type String (expected Data)

and on the server I get the same stack trace as above. If I drop the database and re-create it I can do this whole process again.

I see above you say “enormous values” on rows. Are you talking about the param_values table? How big is enormous?

#6 Updated by Andrew Forgue about 4 years ago

  • Status changed from Re-opened to Closed

Re-closing ticket — It appears that there was a 0.25.x puppet master connected to the same storeconfig database. Since upgrading that to 2.6.1 and re-creating the DB I have yet to see this problem in 2.6.1. Documenting my idiocy so others don’t have to figure it out. You can look in the YAML file on the client for YAML::Object to see which resource is broken.

#7 Updated by Peter Meier about 4 years ago

I just encounter that as well on a migrated 2.6.1 environment with a 2.6.1 client.

#8 Updated by Peter Meier about 4 years ago

  • Status changed from Closed to Re-opened

Peter Meier wrote:

I just encounter that as well on a migrated 2.6.1 environment with a 2.6.1 client.

So I found out that it fails to serialize the following column of an exported resource:

#< ActiveRecord::ConnectionAdapters::MysqlColumn:0xb73fd420 @scale=nil, @sql_type="text", @type=:text, @precision=nil, @primary=false, @limit=nil, @default="", @name="value", @null=false>
#< YAML::Object:0xb77fcabc @class="Puppet::Resource::Reference", @ivars={"builtin_type"=>nil, "title"=>"concat_/etc/amanda/DailySet1/disklist", "type"=>"Exec"}>

Puppet::Resource::Reference have been removed while switching from 0.25.x to 2.6.1 from the codebase.

I have plenty of stored values containing such ruby objects serialzed as yaml in the current database in the param_values table.

Deleting these entries fix the problem.

I’m reopening the ticket as it have been closed with a wrong assumption and I think that one of the devs should look at the issues I found out. My 2c are that we have references as yaml serialized with ruby class references that don’t exist anymore in 2.6.1

#9 Updated by Markus Roberts about 4 years ago

Peter —

This isn’t the same error (though it has the same symptom); could you open a new ticket for it (don’t forget to set the affected version to 2.6.1), mark it as related to this one, and close this ticket? If possible, include the information that led you to conclude that it was the Puppet::Resource::References causing the problem.

I’m not sure if we’ll have a reasonable fix for it (beyond telling people to clear out their stored configs when upgrading) but that discussion belongs on its own ticket.

— Markus

#10 Updated by Peter Meier about 4 years ago

Markus —

This isn’t the same error (though it has the same symptom); could you open a new ticket for it (don’t forget to set the affected version to 2.6.1), mark it as related to this one, and close this ticket? If possible, include the information that led you to conclude that it was the Puppet::Resource::References causing the problem.

I’m pretty sure that this is the same error, as I got the same stacktrace. I don’t really understand why this should be a different error.

I would describe the problem as follow:

  • 0.25.x stores relations of exported resources as serialized yaml that include a reference to the ruby class Puppet::Resource::References
  • 2.6 removed this class
  • 2.6 master gets problem dealing with such stored resources
  • cleaning the database from such resources “fixes” the problem

I could track things down that it happens while storing an association of an exported resource which is a serialized yaml object of a class that isn’t anymore existent in 2.6.

As imho the problem isn’t related to #4506 (it also has a different symptom) it was closed under wrong assumptions. The first re-opening was also under a wrong assumption, but which actually makes sense, as there was still a 0.25.x master writing to the database that would still create such serialized references from time to time in the database. It is therefore clear, that cleaning the database and running only 2.6 masters “fixes” the problem. Hence, the second closing was also under a wrong assumption that the problem is fixed.

To summarize I would say:

  • The error isn’t related to #4506 hence also the fix doesn’t apply
  • The problem occurs with a certain kind of exported resources created on 0.25.x masters
  • A possible workaround is to clean the database from such resources

Sorry, to insist on staying here, but I would simply open another bug report with the same stacktrace and add the things I found as I have done here in the comments. So in my opinion we would end up with another identical bug report. But maybe I just missed something.

#11 Updated by Markus Roberts about 4 years ago

  • Status changed from Re-opened to Closed

Peter —

It’s a different error because it has a different cause even though the symptoms are identical. While the symptoms are what are observed directly we are really only interested in the causes (which we can’t observe directly) since that’s what we need to change to fix the problem.

In the present case, anything that causes data to be serialized by one version that can not be deserialized by another would cause this stack trace, so the fact that they are the same doesn’t really mean much. A neighborhood constructed with faulty wiring and a neighborhood being targeted by an arsonist might have well nigh identical symptoms, but the causes (and thus the steps necessary to prevent future recurrences) would be very different.

I’ll open the new ticket.

— Markus

#12 Updated by Peter Meier about 4 years ago

It’s a different error because it has a different cause even though the symptoms are identical. While the symptoms are what are observed directly we are really only interested in the causes (which we can’t observe directly) since that’s what we need to change to fix the problem.

Ok, I didn’t realize these are different causes. Sorry.

Also available in: Atom PDF