The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #9109

fact and node caches not written to disk with Mongrel and Puppet > 2.7.0

Added by John Guthrie over 3 years ago. Updated almost 3 years ago.

Status:ClosedStart date:08/18/2011
Priority:HighDue date:
Assignee:Jacob Helwig% Done:

0%

Category:server
Target version:2.7.8
Affected Puppet version:2.7.3 Branch:https://github.com/jhelwig/puppet/tree/tickets/2.7.x/9109-retrieve-facts-from-post-in-mongrel
Keywords:puppet master discard facts

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.


Description

Hello all,

I seem to have stumbled onto a possible bug where the puppet master seems to be discarding any and all facts that a client sends to it. I’ve been trying a large number of configuration changes, and nothing seems to work.

First of all, the setup:

puppet master:

Oracle enterprise Linux 5.6 (A RHEL derivative)
puppet 2.7.3
mongrel 1.1.5
apache 2.2.3
ruby 1.8.5

puppet client:

Oracle enterprise Linux 5.6
puppet 2.7.3
ruby 1.8.5

Next, the symptom:

Whenever I try to access a fact in a puppet configuration or a template it comes back as an empty string. This holds for any fact, and seems to be true for every client that I have tried. I do know that the puppet master is receiving the facts since I have done packet captures using wireshark on the puppet master machine, and one can clearly see the facts being sent to the puppet master. And then the puppet master sends back a configuration that one would expect if you had sent no facts at all. I see the same result when I connect directly to the server using wget or telnet. Even if this is some configuration error on my part, I can’t think of any reason why this should ever happen. Hence, I am filing this as a bug.

I also have a second puppet server running puppet 2.7.0 that I had set up a while ago for testing purposes, and I am seeing the same behaviour on that machine as well.

What I have tried to fix this: At first, I thought this might have something to do with my storeconfigs. So I turned that off. It didn’t make any difference. In addition, I have tried all of the following configuration changes. None of them have helped:

  • disabled and removed all custom facts
  • disabled pluginsync
  • twiddled inventory service settings
  • Turned off external node configuration and switched to using AST
  • Tried to explicitly set the environment for the client
  • Switched to using a simple SSL PKI for puppet instead of a certificate heirarchy
  • Opened up auth.conf to allow any REST API request from anywhere (And yes, I am fully aware that this is a bad idea in general. This is a temporary measure to eliminate any ACLs as culprits.)
  • Tried using the global forms of the fact variables, i.e. $::ipaddress instead of $ipaddress
  • Tried using the —ignorecache flag
  • Upgraded from 2.7.1 to 2.7.3

I know some of them might not make sense, but I’ve been in a position of not knowing what else to do.

I have also placed some debugging messages in the following files in an attempt to catch puppet in the act of discarding the facts:

  • puppet/indirector/indirection.rb
  • puppet/indirector/catalog/compiler.rb
  • puppet/indirector/catalog/static_compiler.rb

Alas, the facts had already been discarded by the time the execution reached this code. (This might be obvious to some of the core developers, but this was my first deep look at the puppet ruby code.) If someone could tell me where in puppet code it first picks up the request, that would be great. Then I could actually trace the data path from its start.

I have briefly looked at the ruby gem ruby-debug, but I have not had much to start using it.

Tests that I have been using:

I have been using two pieces of puppet configuration to test this. The first is the following notify resource:

notify {
    "Echo-IP":
        message  => "IP = $::ipaddress",
        withpath => true;
}

I would expect the following:

notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: IP = xxx.xxx.xxx.xxx
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: defined 'message' as 'IP = xxx.xxx.xxx.xxx'
info: /Stage[main]/Subversion::Server/Notify[Echo-IP]: Scheduling refresh of (completed_Subversion::Server)

Instead, I see the following:

notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: IP = 
notice: /Stage[main]/Subversion::Server/Notify[Echo-IP]/message: defined 'message' as 'IP = '
info: /Stage[main]/Subversion::Server/Notify[Echo-IP]: Scheduling refresh of (completed_Subversion::Server)

I have also been using the following suggestion from http://www.puppetcookbook.com/posts/see-all-client-variables.html:

file {
    "/tmp/facts.yaml":
         content => inline_template("<%= scope.to_hash.reject { |k,v| ! (k.is_a?(String) && v.is_a?(String) ) }.to_yaml %>");
}

This places all of the variables that are available to the client in file called /tmp/facts.yaml. I would expect to see my facts in there. Instead, I only see the following:

--- 
  name: subversion::server
  title: subversion::server
  servername: puppet.server.in.my.domain
  serverip: xxx.yyy.zzz.www
  caller_module_name: &id001 subversion
  domain: my.domain
  puppetserver: puppet.server
  ntpserver_prefer: ntp.my.domain
  environment: production
  module_name: *id001
  serverversion: 2.7.3

(As you can see some variables have been redacted.) Of these, domain, ntpserver_prefer, and puppetserver, are all set manually by me in the configuration. The rest I assume are generated by the puppet master. There is nothing that is generated specifically by the client. ipaddress, ipaddress*, netmask, netmask, hostname, fqdn, kernel, uptime, etc. are all notably absent.

Thank you very much in advance for any help.


Related issues

Related to Puppet - Bug #9388: Agent not respecting environment Closed 09/08/2011
Duplicated by Puppet - Bug #8342: facts not available to mongrels behind nginx Duplicate 07/10/2011
Duplicated by Puppet - Bug #8420: Puppet 2.7.x using mongrel and apache doesn't work with n... Duplicate 07/14/2011

History

#1 Updated by James Turnbull over 3 years ago

  • Status changed from Unreviewed to Needs More Information

John – does this similar to #8342?

#2 Updated by John Guthrie over 3 years ago

Heh. That looks exactly like this bug, except I am using apache instead of nginx. In fact, I just switched from mongrel to webrick to see if that would affect anything and sure enough, facts don’t get discarded when I use webrick. I guess the other difference is that bug #8342 only refers to 2.7.1, and I am seeing this on 2.7.3. (Which I suppose is a largely superficial difference, but I thought that I would note it.)

#3 Updated by John Guthrie over 3 years ago

I should also note that I am seeing this behaviour with puppet 2.7.3 under ruby 1.8.5. I am also seeing this with puppet 2.7.0 under ruby 1.8.7.

#4 Updated by John Guthrie over 3 years ago

By the way, should I be changing the status of this issue from “Needs More Information” to something else. My only other two choices seem to be “In Topic Branch Pending Merge” and “Closed”.

#5 Updated by James Turnbull over 3 years ago

  • Status changed from Needs More Information to Accepted
  • Priority changed from Normal to High

#6 Updated by Radosław Antoniuk about 3 years ago

same problem with 2.7.1 and mongrel + apache

#7 Updated by jon de la vega about 3 years ago

Same here, we are using nginx+unicorn with about 250 servers. Using puppetmaster 2.7 all facts get discarded but it works fine with 2.6

#8 Updated by Jonathan Kinred about 3 years ago

I’m also seeing this issue with a 2.7.3 server using Mongrel behind Apache and a 2.7.3 client.

I am running 200+ clients which were upgraded from 0.25.2 –> 2.7.3 which don’t suffer from this issue. The client which has this issue was the first newly provisioned 2.7.3 client I’d tried to run Puppet on.

I upgraded my master to use mod_passenger which resolves the issue.

#9 Updated by Tim Yim about 3 years ago

Same problem here:

Clean install of: CentOS 6
puppet 2.7.5
facter 1.6.2
ruby 1.8.7
nginx 0.8.54

NOTE: Puppet and Facter were installed directly from PuppetLabs yum repo.

Using the File /tmp/facts.yaml trick form above, the nodes would only print the same small subset of facts as pasted above.

Installing 2.6.9 and then yum upgrading to 2.7.5 would fix the issue for me:

$ yum install puppet-2.6.9
$ yum upgrade puppet

Oh, also, apparently 2.6.9 is the magic number. 2.6.10,11,12 is broken as well!

I hope this helps guys.

Keep up the great work!

#10 Updated by Nigel Kersten about 3 years ago

I just reproduced this running Puppet out of the 2.7rc branch, using my own mongrel startup script.

#11 Updated by Nigel Kersten about 3 years ago

ok, so some troubleshooting should make it easier for someone to actually fix this.

When running against mongrel, the following info (with $vardir = /var/lib/puppet) doesn’t ever get populated.

/var/lib/puppet/yaml/node/pe-centos6.localdomain.yaml
/var/lib/puppet/yaml/facts/pe-centos6.localdomain.yaml

If I start up a webrick master on another port, and puppet against it, then the subsequent mongrel runs pick up the data out of the disk cache and it’s populated correctly.

#12 Updated by Nigel Kersten about 3 years ago

  • Subject changed from facts being discarded to fact and node caches not written to disk with Mongrel and Puppet > 2.7.0

#13 Updated by Jacob Helwig about 3 years ago

  • Status changed from Accepted to Investigating
  • Assignee set to Jacob Helwig

I’m able to reproduce this with a fresh CentOS 6 x86_64 VM, with the following setup (and their dependencies):

  • Puppet 2.7.5-1.el6 (From yum.puppetlabs.com)
  • Facter 1.6.2-1.el6 (From yum.puppetlabs.com)
  • epel-release 6-5
  • mongrel 1.1.5
  • nginx 0.8.54-1.el6
  • ruby 1.8.7.299-5.el6_0.1

nginx.conf:

user  daemon daemon;
worker_processes  4;

error_log       /var/log/nginx-puppet.log notice;
pid             /var/run/nginx-puppet.pid;

events {
    worker_connections  1024;
}

http {
    default_type  application/octet-stream;

    sendfile        on;
    tcp_nopush      on;

    large_client_header_buffers     16      4k;
    proxy_buffers                   128     4k;

    keepalive_timeout  65;
    tcp_nodelay        on;

    ssl                     on;
    ssl_certificate         /var/lib/puppet/ssl/certs/centos-6-x86-64-master.testing-vm.perlninja.com.pem;
    ssl_certificate_key     /var/lib/puppet/ssl/private_keys/centos-6-x86-64-master.testing-vm.perlninja.com.pem;
    ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;
    ssl_ciphers             SSLv2:-LOW:-EXPORT:RC4+RSA;
    ssl_session_cache       shared:SSL:8m;
    ssl_session_timeout     5m;
    ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;

    upstream puppet-production {
        server 127.0.0.1:18140;
    }

    server {
        listen                  8140;
        ssl_verify_client       optional;
        root                    /var/empty;
        access_log              /var/log/nginx/access-8140.log;

        location / {
            proxy_pass          http://puppet-production;
            proxy_redirect      off;
            proxy_set_header    Host             $host;
            proxy_set_header    X-Real-IP        $remote_addr;
            proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
            proxy_set_header    X-Client-Verify  SUCCESS;
            proxy_set_header    X-Client-DN      $ssl_client_s_dn;
            proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
            proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
            proxy_read_timeout  65;
        }
    }
}

Invoking a single worker on the command line via: puppet master --servertype mongrel --masterport 18140 --no-daemonize --verbose

With this setup, I get the following for /tmp/facts.yaml, when running puppet agent --test:

--- 
  name: main
  title: main
  servername: localhost.localdomain
  serverip: 10.0.2.15
  module_name: ""
  environment: production
  serverversion: 2.7.5

#14 Updated by Marek Kroemeke about 3 years ago

I can also confirm that this happens on freshly installed RHEL 5.7, with puppet 2.7.6 (from puppetlabs repo) and facter 1.6.2-1.el5 . In addition – rolling back to 2.6.9 and then installing 2.7.6 again fixes the problem of facter’s facts, but custom facts within modules are still not visible. Our setup uses apache and mongrel.

#15 Updated by Jacob Helwig about 3 years ago

It looks like the problem comes about when using >= 2.7.0 clients with a master running using mongrel. 2.7.0 switched to using POST requests for sending the facts to the master, and the REST handler we have for mongrel isn’t setup to retrieve the facts from the POST body.

I need to write up some tests for this, but Nick L. and I believe that the following will fix the problem:

diff --git i/lib/puppet/network/http/mongrel/rest.rb w/lib/puppet/network/http/mongrel/rest.rb
index 7ef13f0..2c17974 100644
--- i/lib/puppet/network/http/mongrel/rest.rb
+++ w/lib/puppet/network/http/mongrel/rest.rb
@@ -28,6 +28,8 @@ class Puppet::Network::HTTP::MongrelREST < Mongrel::HttpHandler
   # testing purposes.
   def params(request)
     params = Mongrel::HttpRequest.query_parse(request.params["QUERY_STRING"])
+    params.merge!(Mongrel::HttpRequest.query_parse(body(request))) if http_method(request).upcase == 'POST'
+
     params = decode_params(params)
     params.merge(client_info(request))
   end
@@ -41,7 +43,12 @@ class Puppet::Network::HTTP::MongrelREST < Mongrel::HttpHandler

   # return the request body
   def body(request)
-    request.body.read
+    body = request.body.read
+    # We rewind the body, since read on a StringIO is destructive, and
+    # subsequent reads will return an empty string.
+    request.body.rewind
+
+    body
   end

   def set_content_type(response, format)

#16 Updated by Jacob Helwig about 3 years ago

  • Status changed from Investigating to In Topic Branch Pending Review
  • Branch set to https://github.com/jhelwig/puppet/tree/tickets/2.7.x/9109-retrieve-facts-from-post-in-mongrel

I’ve submitted pull request 182 to resolve this in 2.7.x.

(#9109) Retrieve request parameters from the request body for POSTs

When using Puppet with Mongrel, the facts were being lost with 2.7.0+
clients, since they were switched over to using POST requests for fact
submission.  This was happening because the request parameters were
only being retrieved from the query parameters of the URL, which will
not include the POSTed data.

We now merge the body of the post together with the query parameters
when dealing with POST requests.

We also rewind the request body after reading it, since retrieving the
request body multiple times would return an empty string after the
first time.

Paired-with: Nick Lewis <nick@puppetlabs.com>

#17 Updated by Jacob Helwig about 3 years ago

  • Target version set to 2.7.x

#18 Updated by Josh Cooper about 3 years ago

  • Status changed from In Topic Branch Pending Review to Merged - Pending Release

Merged into 2.7.x in merge commit https://github.com/puppetlabs/puppet/commit/08547719ab0d45cc3dae7ff1fbd3d3ffc877630b Note if the request body is larger than Const::MAX_BODY (which for 1.1.5 is 1024 * (80 + 32) = 114688), then mongrel will store the body in a tempfile instead of a StringIO. But in either case, the rewind method will do the right thing.

#19 Updated by Matthaus Owens almost 3 years ago

  • Status changed from Merged - Pending Release to Closed
  • Target version changed from 2.7.x to 2.7.8

Released in 2.7.8rc1

Also available in: Atom PDF