Puppet

Markus, I’d love to test your patch. Is this in your own clone or in the puppet repository?

Also, this looks like the root cause might impact more than just the environment column. My environment argument isn’t working when I have storedconfigs enabled.

Good too know, maybe the shorthand should be added to http://docs.puppetlabs.com/guides/from_source.html?

Re: second paragraph. I noticed that puppetd —environment jkoppe completely stopped working. As soon as I disabled storedconfigs, puppetd —environment jkoppe started working again.

That doesn’t seem to make a difference.

I’m not sure Markus' patch fixes anything since I can’t find a code path that touches the code that was changed. But I’m also unable to reproduce the problem, so I could be off.

Markus, did you ever hear back from Jason Koppe on if your change helped him? I wanted to add him as a watcher but he’s not in the dropdown. I forgot how to add people there. For now I’ll just email him separately to ask if your patch worked for him.

I’ll keep looking, but when I run storeconfigs and get the hosts to update, it happens via this code path:

“/Users/matthewrobinson/work/puppet/lib/puppet/indirector/facts/active_record.rb:34:in save'", "/Users/matthewrobinson/work/puppet/lib/puppet/node/facts.rb:15:insave'”, “/Users/matthewrobinson/work/puppet/lib/puppet/indirector.rb:64:in save'", "/Users/matthewrobinson/work/puppet/lib/puppet/indirector/catalog/compiler.rb:27:inextract_facts_from_request'”,

I started writing a test, but then thought I’d reproduce the problem since I got to wondering if the stubbed out node object we were describing in tests wasn’t what it actually looked like when we got to the code in question in lib/puppet/rails/host.rb.

Here’s the spec I started for host_spec.rb

+    describe "when storing the host in the database" do
+      before :each do
+        @resource1 = stub_everything 'res1'
+        @resource2 = stub_everything 'res2'
+        @resources = [ @resource1, @resource2 ]
+        Puppet::Rails::Host.stubs(:transaction).yields
+        Puppet::Rails::Host.expects(:find_by_name).with("foo").returns @host
+      end
+      it "should set parameters and then call save" do
+        @host.expects(:merge_facts)
+        @host.expects(:merge_resources)
+#       @host.expects(:environment=)
+#       @host.expects(:ip=)
+        @host.expects(:last_compile=)
+        @host.expects(:save)
+        Puppet::Rails::Host.store(@node, @resources).should == @host
+      end
+    end

I had tested MarkusQ/tickets/2.6.x/4487. I just started puppet master on tickets/2.6.x/4487 and tested it again. Here’s what the host environment column looks like now:

mysql for root@localhost on localhost> select * from hosts where name like '%name%'\G
*************************** 1. row ***************************
environment: --- &id006 !ruby/object:Puppet::Node::Environment
cache_timestamp: 2010-09-20 19:31:14.984732 -05:00
known_resource_types: &id001 !ruby/object:Puppet::Resource::TypeCollection
definitions:
luceneslave::profile_cron_indexsync: !ruby/object:P
1 row in set (0.00 sec)

I just started puppet master on 2.6.1rc4 and I’m no longer seeing the problem I initially reported.

I looks like a lot of the code in lib/puppet/rails/host.rb became obsolete with Luke’s commit b9c95ebf81eeb78297003de2d0ed4ca048412393, and code that actually called the .store method was removed in 863c50b1273a7fa48d74fb74948938214b45967c when Puppet was version 0.24.8.

I haven’t proven this yet, but I’ll pick this back up tomorrow to see if that’s true and we can remove a bunch of that code.

Markus, were you able to reproduce the initial bug in this ticket? I never could. Also, did you see your change make a difference in code execution, or was your change just based on reading through the code for suspicious lines?

It was based on code reading; I could not reproduce the problem, so that’s all I had to go on.

It appears from the note above that Jason’s no longer seeing it either as of 2.6.1.

Should we then close this as resolved and open a new one aimed at 2.7.x for dead code removal?

I can reproduce this problem.

mysql> select count(*) from hosts where length(environment) > 2000;
+----------+
| count(*) |
+----------+
|        5 |
+----------+
1 row in set (0.00 sec)

mysql> select environment from hosts where length(environment) > 2000 group by environment \G
*************************** 1. row ***************************
environment: --- "--- \"--- \\\"--- \\\\\\\"--- \\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\"--- \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\ 

Seems to only affect some servers – I’ll see if I can find differences.

mysql> select substring_index(name, ".", 2), length(environment) from hosts order by length(environment) ;
...

| solr45-1.staging1             |                  10 |
| membase1.staging1             |                  10 |
| ip-10-196-39-192.ec2          |                  10 |
| ip-10-196-39-192.staging1     |                  10 |
| solr00-4.prod                 |                  14 |
| solr00-2.prod                 |                  96 |
| solr00-3.prod                 |                 556 |
| solr45-2.prod                 |               65535 |
| solr45-1.prod                 |               65535 |
| solr00-1.prod                 |               65535 |
| solr13-1.prod                 |               65535 |
| solr13-2.prod                 |               65535 |
+-------------------------------+---------------------+
37 rows in set (0.00 sec)

Hi –

For what its worth, I was seeing this exact same problem. I ended up updating to ruby-mysql 2.8.2 which seems to have fixed this. I’m testing this out a little more, wanting the puppet client to restart a few dozen more times before calling this resolved (in my case anyway), but that may help you out in your situation. It certainly fixed my instance of the problem.

I’m using CentOS 5.5 with a bunch of rebuilt gems and things like that (we rebuild everything, and re-tag/sign them), so most of them are all fairly new. I suspect that you followed the setup instructions listed in Redmine, as I did, that called for specific versions, which may not apply any more in the 2.6 branch, which I’m using (2.6.2, to be exact).

Hope that helps.

Thanks for the update Dan, and thanks even more for letting us know about your workaround.

Hi –

My pleasure. I’m not a great developer, so I’m hoping that someone can eventually confirm from a programmatic standpoint that this is indeed the proper solution to the given problem.

mysql> select substring_index(name;, ".", 2), length(environment), substring(environment,1,20) from hosts where environment like "%---%" order by length(environment);
+-------------------------------+---------------------+-----------------------------+
| substring_index(name, ".", 2) | length(environment) | substring(environment,1,20) |
+-------------------------------+---------------------+-----------------------------+
| solr45-2.prod                 |               65535 | --- "--- \"--- \\\"-        |
| solr13-1.prod                 |               65535 | --- "--- \"--- \\\"-        |
| solr13-2.prod                 |               65535 | --- "--- \"--- \\\"-        |
| solr45-1.prod                 |               65535 | --- "--- \"--- \\\"-        |
| solr00-1.prod                 |               65535 | --- "--- \"--- \\\"-        |
+-------------------------------+---------------------+-----------------------------+

Seems like there’s some bad yaml encapsulation going on. I don’t think this is a mysql rubygem bug.

Hi,

Experiencing a similar issue where 0.24.8 set the storeconfigs puppet db (MySQL) environment column value to “production”, 2.6.2 sets the value to “—– production”.

“/var/lib/gems/1.8/bin/puppetmasterd —configprint environment” returns “production”

Cheers Geoff

Hi –

I’ll be honest, I’ve been kind of lazy in regards to updating classes of configuration directives in puppet.conf; puppetd opposed to the new “agent”, puppetmasterd instead of the new “master”, etc etc.

I tested and confirmed that this only happens when using the new class name of “master” instead of my previously used “puppetmasterd” configuration class. This is the only time that I can absolutely reproduce this.

If someone else can attempt the same test to further confirm, I suspect this will help a great deal. Simply change [master] to [puppetmasterd]. It will still work, albeit deprecated so you’ll want to keep an eye on it for future releases, but still.

Thanks -dant

I should note now that I am using puppet in a masterless deployment.

This isn’t specific to mysql. I found this while debugging puppet with minstrel:

% RUBYLIB=lib RUBY_INSTRUMENT=ActiveRecord::ConnectionAdapters::SQLite3Adapter FACTER_funfact="new fact" bin/minstrel puppet apply --storeconfigs --dblocation ./storeconfigs.sqlite -e 'notice($funfact)'

...

enter ActiveRecord::ConnectionAdapters::SQLite3Adapter#update_without_query_dirty(["UPDATE \"hosts\" SET \"environment\" = '--- \"--- \\\"--- \\\\\\\"--- \\\\\\\\\\\\\\\"--- production\\\\\\\\\\\\\\\"\\\\\\\"\\\"\"', \"last_compile\" = '2010-11-15 15:55:48.458498', \"updated_at\" = '2010-11-15 15:55:48.623126'  WHERE (\"hosts\".\"id\" = 1)"])

As above, this is with sqlite3, not mysql.

I can confirm that. In my case it occurs on both backends: mysql and sqlite. In case of using sqlite, it looks like there is no limit for length of this field, so puppet agent which tries to use stored variables eats all memory after some time. I’m using puppet from debian lenny-backports (puppet 2.6.2-1~bpo50+1)

mysql> select name, length(environment) from hosts;
+----------------------+---------------------+
| name                 | length(environment) |
+----------------------+---------------------+
| debian-1.example.com |               65535 |
| debian-3.example.com |               65535 |
| debian-4.example.com |               65535 |
| debian-2.example.com |               65535 |
+----------------------+---------------------+
4 rows in set (0.00 sec)

This looks like an incompatibility of Puppet 2.6.x and older rails/activerecord/mysql gem versions. Reproducible with Centos 5.5 and Debian Lenny running rails 2.2.1 and 2.2.0. Installed Ruby EE 1.8.7 on Centos 5.5, installed rails through gems instead of through yum and the errors subside completely.

Cody,

My systems use newer versions than 2.2.x:

• activerecord (3.0.3)
• ruby-mysql (2.9.3)

Also observed this problem with the non-pure-ruby mysql gem ‘mysql’ latest version.

Cody Herriges wrote:

I have found the issue and a fix but don’t 100% understand why the current code is not functional. The patch I have attached fixes the issue.

node.parameter[“environment”] will return a string. node.environment is an object that contains yaml. If you do a .to_s on the object you will get the environment name prepended with —– and someplace in the active_record terminus and rails code for puppet it gets put in the database as is. Just using the parameter which is always a properly formatted string works. I am interested more in where the database call happens and tracking down why the previous code doesn’t work.

I would change the branch in Redmine if I could get the branch to push…I will change it when I figure out why it won’t push.

Just noting that I’ve applied this patch to a 2.6.2 (gentoo) and it works.

Just poked at this for a while. I thought it might be due to us specially handling environment when parsing the config file, but that doesn’t appear to be the case.

Getting review eyes onto the code now.

I can reproduce at-will on 2.6.x and 2.6.next

ii  libmysql-ruby1.8                   2.8.2-1
ii  rails-ruby1.8                      2.3.5-1.2