Puppet

Discussed this with Josh. There is a configuration option that you can use to turn this off; when I get a minute, I’m going to re-break my DNS, confirm that I see the slowdown again, and then try running with that option turned off just to confirm.

Pending the outcome of that experiment, we may want to reconsider whether or not this is enabled by default.

One other bit of info: I now have a suspicion that this affects EL flavors of linux out-of-the-box much more readily than Debian flavors. I do not recall having this problem with my ubuntu 10.04 VM, but did have it with Cent5 and Cent6. My theory is that “hostname” returns (for all intents and purposes) the contents of /etc/hostname on debian systems, and debian doesn’t seem to try to put any domain name in there. Meanwhile, on the EL flavors, “hostname” seems to return what’s in /etc/sysconfig/networking… which, for me, on out-of-the-box Cent installs appears to be “hostname.localdomain”. Facter then sees this as the FQDN and things get weird from there… all of this could use further investigation, though.

Confirmed that the use_srv_records setting also alleviates this problem.

In my acceptance testing environment, I logged into my CentOS6 node and changed the /etc/sysconfig/network line back to its original value:

HOSTNAME=pe-centos6.localdomain

Rebooted, and ‘hostname’ now returns a pseudo-fqdn again:

[root@pe-centos6 ~]# hostname
pe-centos6.localdomain
[root@pe-centos6 ~]# 

Facter interprets this as the fqdn again:

[root@pe-centos6 ~]# facter fqdn
pe-centos6.localdomain
[root@pe-centos6 ~]# 

And running this command:

time puppet agent --test --debug

regularly takes over three minutes to execute. It very visibly hangs on this line:

debug: Searching for SRV records for domain: localdomain

at least twice during each execution of the agent. If I then edit the /etc/puppet/puppet.conf and adding the line:

use_srv_records=false

and then run the agent, the agent execution time has dropped from over 3 minutes to less than 5 seconds.

Another option to consider might be to disable this option for acceptance testing? i.e., when the framework generates the config file for agents, it could add this setting?

Would lessen the likelihood of developers encountering this problem, but would also introduce a slight variation between our acceptance testing environment and our customers' likely default environments, which might not be desirable.

FWIW I spent a bit of time looking at the code in question, and the Ruby Resolv::DNS library that we are using did not have any obvious hooks for setting a timeout… so I didn’t see any easy way of dealing with that, short of launching a new thread that we could kill after our timeout window had passed. And it sounds like we’ve not been having much fun with code that spawns new threads recently.

Perhaps I overlooked the timeout setting, or perhaps there is another library that would provide this functionality…

Chris Price wrote:

FWIW I spent a bit of time looking at the code in question, and the Ruby Resolv::DNS library that we are using did not have any obvious hooks for setting a timeout… so I didn’t see any easy way of dealing with that, short of launching a new thread that we could kill after our timeout window had passed. And it sounds like we’ve not been having much fun with code that spawns new threads recently.

Perhaps I overlooked the timeout setting, or perhaps there is another library that would provide this functionality…

You probably want the timeout module from the standard library, which should implement that all correctly, I believe.