Puppet

Hello all,

I seem to have stumbled onto a possible bug where the puppet master seems to be discarding any and all facts that a client sends to it. I’ve been trying a large number of configuration changes, and nothing seems to work.

First of all, the setup:

puppet master:

Oracle enterprise Linux 5.6 (A RHEL derivative)
puppet 2.7.3
mongrel 1.1.5
apache 2.2.3
ruby 1.8.5

puppet client:

Oracle enterprise Linux 5.6
puppet 2.7.3
ruby 1.8.5

Next, the symptom:

Whenever I try to access a fact in a puppet configuration or a template it comes back as an empty string. This holds for any fact, and seems to be true for every client that I have tried. I do know that the puppet master is receiving the facts since I have done packet captures using wireshark on the puppet master machine, and one can clearly see the facts being sent to the puppet master. And then the puppet master sends back a configuration that one would expect if you had sent no facts at all. I see the same result when I connect directly to the server using wget or telnet. Even if this is some configuration error on my part, I can’t think of any reason why this should ever happen. Hence, I am filing this as a bug.

I also have a second puppet server running puppet 2.7.0 that I had set up a while ago for testing purposes, and I am seeing the same behaviour on that machine as well.

What I have tried to fix this: At first, I thought this might have something to do with my storeconfigs. So I turned that off. It didn’t make any difference. In addition, I have tried all of the following configuration changes. None of them have helped:

• disabled and removed all custom facts
• disabled pluginsync
• twiddled inventory service settings
• Turned off external node configuration and switched to using AST
• Tried to explicitly set the environment for the client
• Switched to using a simple SSL PKI for puppet instead of a certificate heirarchy
• Opened up auth.conf to allow any REST API request from anywhere (And yes, I am fully aware that this is a bad idea in general. This is a temporary measure to eliminate any ACLs as culprits.)
• Tried using the global forms of the fact variables, i.e. $::ipaddress instead of $ipaddress
• Tried using the —ignorecache flag
• Upgraded from 2.7.1 to 2.7.3

I know some of them might not make sense, but I’ve been in a position of not knowing what else to do.

I have also placed some debugging messages in the following files in an attempt to catch puppet in the act of discarding the facts:

• puppet/indirector/indirection.rb
• puppet/indirector/catalog/compiler.rb
• puppet/indirector/catalog/static_compiler.rb

Alas, the facts had already been discarded by the time the execution reached this code. (This might be obvious to some of the core developers, but this was my first deep look at the puppet ruby code.) If someone could tell me where in puppet code it first picks up the request, that would be great. Then I could actually trace the data path from its start.

I have briefly looked at the ruby gem ruby-debug, but I have not had much to start using it.

Tests that I have been using:

I have been using two pieces of puppet configuration to test this. The first is the following notify resource:

notify {
    "Echo-IP":
        message  => "IP = $::ipaddress",
        withpath => true;
}

I would expect the following:

notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: IP = xxx.xxx.xxx.xxx
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: defined 'message' as 'IP = xxx.xxx.xxx.xxx'
info: /Stage[main]/Subversion::Server/Notify[Echo-IP]: Scheduling refresh of (completed_Subversion::Server)

Instead, I see the following:

notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: IP = 
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: defined 'message' as 'IP = '
info: /Stage[main]/Subversion::Server/Notify[Echo-IP]: Scheduling refresh of (completed_Subversion::Server)

I have also been using the following suggestion from http://www.puppetcookbook.com/posts/see-all-client-variables.html:

file {
    "/tmp/facts.yaml":
         content => inline_template("<%= scope.to_hash.reject { |k,v| ! (k.is_a?(String) && v.is_a?(String) ) }.to_yaml %>");
}

This places all of the variables that are available to the client in file called /tmp/facts.yaml. I would expect to see my facts in there. Instead, I only see the following:

--- 
  name: subversion::server
  title: subversion::server
  servername: puppet.server.in.my.domain
  serverip: xxx.yyy.zzz.www
  caller_module_name: &id001 subversion
  domain: my.domain
  puppetserver: puppet.server
  ntpserver_prefer: ntp.my.domain
  environment: production
  module_name: *id001
  serverversion: 2.7.3

(As you can see some variables have been redacted.) Of these, domain, ntpserver_prefer, and puppetserver, are all set manually by me in the configuration. The rest I assume are generated by the puppet master. There is nothing that is generated specifically by the client. ipaddress, ipaddress*, netmask, netmask, hostname, fqdn, kernel, uptime, etc. are all notably absent.

Thank you very much in advance for any help.