The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #16637

Puppet confdir and vardir are wrong when running non-root

Added by Anonymous over 3 years ago. Updated about 3 years ago.

Status:ClosedStart date:09/29/2012
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:settings
Target version:3.0.0
Affected Puppet version:3.0.0 Branch:https://github.com/puppetlabs/puppet/pull/1194
Keywords:telly settings defaults confdir vardir runmode run_mode master system

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

Overview

Puppet master should default to confdir of ~/.puppet and vardir of ~/.puppet/var when running as non-root, instead defaults to /etc/puppet and /var/lib/puppet respectively.

In Puppet 3.0.0, the semantics of the term, “configuration directory” (confdir) are as follows:

  1. If confdir is explicitly configured, this value wins.
  2. If Puppet is running as root (or the OS equivalent) then use the system configuration directory. (e.g. /etc/puppet for FOSS or /etc/puppetlabs/puppet for PE)
  3. In all other situations use ~/.puppet

These semantics are no longer affected by the specific username when running non-root, or the application being run (master, agent, etc…).

This is not actually the case in 3.0.0 though:

Actual Behavior

$ puppet master --verbose --no-daemonize
Error: Could not set 'directory' on ensure: Permission denied - /etc/puppet
Error: Could not set 'directory' on ensure: Permission denied - /etc/puppetWrapped exception:
Permission denied - /etc/puppet
Error: /File[/etc/puppet]/ensure: change from absent to directory failed: Could not set 'directory' on ensure: Permission denied - /etc/puppet
/File[/etc/puppet/var.master]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/bucket]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/bucket]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/log]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/log]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/log/masterhttp.log]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/log/masterhttp.log]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/yaml]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/yaml]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/ssl]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/ssl]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/ssl/public_keys]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/ssl/public_keys]: Skipping because of failed dependencies/File[/etc/puppet/var.master/lib]: Dependency File[/etc/puppet] has failures: trueWarning: /File[/etc/puppet/var.master/lib]: Skipping because of failed dependencies/File[/etc/puppet/var.master/ssl/certificate_requests]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/ssl/certificate_requests]: Skipping because of failed dependencies/File[/etc/puppet/var.master/run]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/run]: Skipping because of failed dependencies/File[/etc/puppet/manifests]: Dependency File[/etc/puppet] has failures: trueWarning: /File[/etc/puppet/manifests]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/ssl/private]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/ssl/private]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/ssl/private_keys]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/ssl/private_keys]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/rrd]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/rrd]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/ssl/certs]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/ssl/certs]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/reports]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/reports]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/server_data]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/server_data]: Skipping because of failed dependencies
/File[/etc/puppet/var.master/state]: Dependency File[/etc/puppet] has failures: true
Warning: /File[/etc/puppet/var.master/state]: Skipping because of failed dependencies
Error: Could not prepare for execution: Got 3 failure(s) while initializing: Could not set 'directory' on ensure: Permission denied - /etc/puppet; Could not set 'directory' on ensure: Permission denied - /etc/puppet
Wrapped exception:
Permission denied - /etc/puppet; change from absent to directory failed: Could not set 'directory' on ensure: Permission denied - /etc/puppet

Expected behavior

confdir and vardir should default to my home directory when run as non-root user “jeff”

$ puppet master --verbose --no-daemonize
Info: Creating a new SSL key for ca
Info: Creating a new SSL certificate request for ca
Info: Certificate Request fingerprint (SHA256): E4:95:B1:A5:01:A5:07:80:0B:B7:C6:5E:C1:4F:58:EF:CD:FF:D3:DE:EC:30:EF:10:3C:92:53:91:7A:33:26:BC
Signed certificate request for ca
Rebuilding inventory file
Info: Creating a new certificate revocation list
Info: Creating a new SSL key for mccune.local
Info: Creating a new SSL certificate request for mccune.local
Info: Certificate Request fingerprint (SHA256): A8:77:22:5A:D0:C8:89:69:8E:3B:38:7A:0B:43:E3:D7:AA:E8:7F:73:F3:DC:E6:E2:0C:E1:BA:23:41:ED:4B:CF
mccune.local has a waiting certificate request
Signed certificate request for mccune.local
Removing file Puppet::SSL::CertificateRequest mccune.local at '/Users/jeff/.puppet/ssl/ca/requests/mccune.local.pem'
Removing file Puppet::SSL::CertificateRequest mccune.local at '/Users/jeff/.puppet/ssl/certificate_requests/mccune.local.pem'
Starting Puppet master version 3.0.0

Related issues

Related to Puppet - Bug #15337: Do not merge configuration files in Telly Closed 07/02/2012
Related to Puppet - Bug #16137: Cannot start Puppet service on Ubuntu Closed 08/27/2012
Related to Puppet - Bug #15185: Settings do not expand ~ in user configuration directory Closed 06/23/2012
Related to Puppet - Bug #6734: root:root ownership on /var/lib/puppet breaks puppetmasterd Closed 03/16/2011
Related to Puppet - Bug #5303: Puppet master fails when run as root due to inability to ... Closed 11/15/2010
Related to Puppet - Bug #14609: Wrong ssldir used when using 3.0rc1 packages under passenger Closed 05/21/2012
Related to Puppet - Bug #13588: log dir is not permissioned properly Closed 04/02/2012
Related to Puppet - Bug #7070: Rack puppet master fails to start with Permission denied ... Needs More Information 04/12/2011
Related to Puppet - Bug #7052: Cert generation fails using "--ssldir" Needs Decision 04/11/2011
Related to Puppet - Bug #4964: wrong mode for directory /etc/puppet Accepted 10/08/2010
Related to Puppet - Bug #3922: Backing up files by md5 and saving them "as user" Duplicate 06/01/2010
Related to Puppet - Bug #3236: err: Could not retrieve catalog from remote server: No su... Investigating 02/23/2010
Related to Puppet - Bug #3121: Issues running puppetmasterd as a genetic user Rejected 01/28/2010
Related to Puppet - Bug #2705: Fails to create reports for new nodes Duplicate 10/07/2009
Related to Puppet - Bug #2626: Unhelpful error message - err: Report store failed: Inval... Closed 09/11/2009
Related to Puppet - Bug #2095: Changing the permissions of /etc/puppet/puppet.conf via p... Accepted 03/19/2009
Related to Puppet - Bug #1138: /var/puppet/yaml not created until it's too late Closed
Related to Puppet - Bug #985: /usr/bin/puppetmasterd --mkusers fails Closed
Related to Puppet - Bug #977: Directory ownership of /var/puppet/run Closed
Related to Puppet - Bug #816: puppetmaster doesn't create "$vardir/facts" on startup Closed
Related to Puppet - Bug #68: puppetmasterd fails to create required directories Closed
Related to Puppet - Bug #17156: Regression: Can't restrict the plugins loaded by puppet d... Rejected
Related to Puppet - Bug #17709: Puppet failing to run - Error: Could not intialize global... Closed
Related to Puppet - Bug #18047: Puppet 3 doesn't run in non-elevated context on Windows Closed
Related to Puppet - Bug #17613: puppet master ignores config and commandline parameters o... Needs More Information 11/14/2012
Related to Puppet - Bug #12540: puppet config print generates invalid log settings if pup... Closed 02/09/2012
Duplicates Puppet - Bug #4224: vardir and confdir should be in ~/.puppet if not run as root Closed 07/13/2010
Duplicated by Puppet - Bug #15214: Get rid of coupling between run_mode and confdir/vardir Duplicate 06/25/2012

History

#2 Updated by Anonymous over 3 years ago

  • Status changed from Unreviewed to Accepted
  • Assignee deleted (Anonymous)

History

Historically, we’ve gotten this “wrong” quite a few times when we’ve done major and minor releases: (“this” being the selection of default path values relative to the application mode and the effective UID of the process with nasty errors rising up due to invalid permission.)

  1. (#14609) – Wrong ssldir used when using 3.0rc1 packages under passenger
  2. (#13588) – log dir is not permissioned properly
  3. (#12494) – When cwd is invalid, puppet prints a stack trace
  4. (#10914) – Fail to generate a fresh CA with 2.6.12 (if ssldir not in std. location)
  5. (#9862) – puppet 2.7 cannot run without puppet group on the system
  6. (#7070) – Rack puppet master fails to start with Permission denied if $HOME is not writable
  7. (#7052) – Cert generation fails using “—ssldir”
  8. (#6734) – root:root ownership on /var/lib/puppet breaks puppetmasterd
  9. (#6256) – /var/lib/puppet/rrd not created
  10. (#5303) – Puppet master fails when run as root due to inability to create rrd directory
  11. (#5530) – puppet master fails to create “$vardir/rrd”
  12. (#4964) – wrong mode for directory /etc/puppet
  13. (#4385) – vardir and confdir are set to ‘~’ when running puppet master as non-root user through Passenger
  14. (#4253) – puppetmaster started in a non accessible directory for the puppet user causing problems
  15. (#4224) – vardir and confdir should be in ~/.puppet if not run as root
  16. (#3922) – Backing up files by md5 and saving them “as user”
  17. (#3236) – err: Could not retrieve catalog from remote server: No such file or directory – /var/puppet/client_yaml/catalog
  18. (#3121) – Issues running puppetmasterd as a genetic user
  19. (#2705) – Fails to create reports for new nodes
  20. (#2639) – Fail to store reports in simple default config
  21. (#2626) – Unhelpful error message (unhelpful subject too.)
  22. (#2519) – getcwd error when puppetmasterd is started from a directory where puppet user has no rights
  23. (#2500) – puppetmaster should not load certs when not running under webrick
  24. (#2460) – ssl/private_keys ownership + Passenger
  25. (#2095) – Changing the permissions of /etc/puppet/puppet.conf via puppet crashes puppetmaster
  26. (#1138) – /var/puppet/yaml not created until it’s too late
  27. (#985) – /usr/bin/puppetmasterd —mkusers fails
  28. (#977) – Directory ownership of /var/puppet/run
  29. (#816) – puppetmaster doesn’t create “$vardir/facts” on startup
  30. (#68) – puppetmasterd fails to create required directories

Yes, that’s a two digit bug directly related to this issue. :/

#3 Updated by Anonymous over 3 years ago

  • Affected Puppet version set to 3.0.0

#4 Updated by Anonymous over 3 years ago

  • Status changed from Accepted to In Topic Branch Pending Review
  • Branch set to https://github.com/puppetlabs/puppet/pull/1194

#5 Updated by Anonymous over 3 years ago

  • Status changed from In Topic Branch Pending Review to Merged - Pending Release

#6 Updated by Anonymous over 3 years ago

  • Target version changed from 3.0.x to 3.0.1

#7 Updated by Matthaus Owens over 3 years ago

  • Status changed from Merged - Pending Release to Closed

Released in Puppet 3.0.1-rc1

#8 Updated by Curtis Ruck over 3 years ago

The sheer lack of thought for existing puppetmaster users on this commit is frightening. This completely trashed my test environment.

So, historically, puppet master runs as the user ‘puppet’ since it doesn’t need root access. But Puppet master should keep its certs in a location outside of his $HOME so things like httpd/mod_passenger can access the certificates. By defaulting $confdir to ~/.puppet your hiding the certs by default and making getting mod_passenger/httpd working for puppetmaster that more difficult.

#9 Updated by Anonymous over 3 years ago

  • Status changed from Closed to Needs More Information
  • Target version changed from 3.0.1 to 3.0.x

Curtis Ruck wrote:

The sheer lack of thought for existing puppetmaster users on this commit is frightening. This completely trashed my test environment.

Curtis, I’m really sorry this trashed your test environment. I worked quite a long time on this issue tried my best to resolve the issue once and for all. If you have the time, I’d like to take a moment and explain our thinking about this issue. First, please understand that we put a lot of thought into this. My goal was to resolve the issue once and for all, simplify the problem, simplify the explanation of the way Puppet behaves, and minimize the impact on you and all of our end users.

As you can see by the list of related tickets and the History section above, this issue has been a “looping.” In some releases we fix the issue, in other releases we regress. I found 30 tickets related to this issue going all the way back to the very beginning of the Puppet project. Using this data and history, I’ve come to the conclusion that if I simply try and fix the issue using the current semantics then there’s a very high likelihood another issue will come up after the 3.0.0 release. Basically, we’re unable to correctly implement the previous description of the behavior of the puppetmaster:

  1. Use explicitly set --confdir and --vardir if they’re set on the command line or in the configuration file. (Parsing puppet.conf poses a challenge because where is the default configuration file located given the rest of these rules?)
  2. If the process is a puppet master and is running as the system puppet user (which may not necessarily be “puppet”), then use the system directories.
  3. If the process is the master and is running as a user who is EUID != 0 and who is now the system puppet user, then use the home directories.
  4. If the process is anything other than the master and is running as EUID != 0, then use the home directories.
  5. If the process EUID == 0 then use the system directories.

So this set of rules pose a challenge to “get right.” We also have a lot of evidence that we’re unable to get it right. When we don’t get it right we negatively affect all of our uses and also cause failures on the puppet master. I went through this myself a number of times when I managed Puppet so I totally understand your pain. My thinking on this issue was that if I can cause just a little bit of pain in a major, backwards-incompatible release but it means that the issue will be resolved once and for all, then it’s worth it for all of us.

I asked myself, what to do? What are the options I have? What if I can change the rules? Puppet 3.0.0 is a major release and presents me with the opportunity to do just that. I can change the rules of the game and call out the change as backwards incompatible. My intent with this patch was to do just that, here’s what I came up with that I hope is more simple for everyone to understand from the perspective of a Puppet user and from my perspective of trying to make this code work well now and through the future.

  1. If the user explicitly sets --confdir and --vardir, then use those values.
  2. If the process us EUID == 0 then use the system directories.
  3. For all other cases, use the personal directories.

As you can see, this description of the behavior is much more simple. There are far fewer edge cases. The downside to this change is that I’ve eliminated the special case behavior for detecting if the process is running as the Puppet system user or not. My intent was to call out that this change requires a change to the config.ru to explicitly set --confdir and --vardir.

Now, I overlooked something for the 3.0.0 release. I thought the only change required would be to set --confdir, so I updated our example config.ru to contain this argument. However, both --confdir and --vardir must be specified in config.ru.

I’ve since fixed the config.ru example in this commit: 623fa02 Update config.ru to fix issue with vardir.

So this brings us up to the 3.0.1rc1 version of Puppet.

So, historically, puppet master runs as the user ‘puppet’ since it doesn’t need root access. But Puppet master should keep its certs in a location outside of his $HOME so things like httpd/mod_passenger can access the certificates. By defaulting $confdir to ~/.puppet your hiding the certs by default and making getting mod_passenger/httpd working for puppetmaster that more difficult.

httpd usually starts as EUID root and sheds permissions after reading the configuration files (and certificates). So it doesn’t really matter where you keep the certificates, Apache will be able to read them when it starts because it’s root.

There are actually two modes of operation for puppet masters. If you start the master from the command line using puppet master as root, then it will run as root for a time and drop privs down to the user you’ve configured as the system user. If you use passenger, however, the Puppet process never starts as root because Passenger will shed permissions and change to the EUID of the account who owns the config.ru file.

So this is all of the thinking that went into this relatively small, but knowingly-backwards-incompatible, change.

How did this issue affect you? What caused you to upgrade to 3.0.0 in your test environment? What is the behavior you expected? What was the behavior you received?

I hope this information and these questions help.

I’m going to re-open this issue because as I mentioned, I hoped to resolve this issue in a manner that will stay resolved. Based on this update it seems to still be an issue.

Cheers, -Jeff

#10 Updated by James Turnbull over 3 years ago

  • Assignee set to Curtis Ruck

#11 Updated by Anonymous over 3 years ago

  • Assignee changed from Curtis Ruck to Anonymous

I’m just re-assigning this to test if I can change the assignee.

-Jeff

#12 Updated by Anonymous over 3 years ago

  • Assignee changed from Anonymous to Curtis Ruck

#13 Updated by Anonymous over 3 years ago

  • Status changed from Needs More Information to Closed
  • Assignee deleted (Curtis Ruck)
  • Target version changed from 3.0.x to 3.0.0

Curtis,

I’m going to close this ticket again. If you have the opportunity to address the questions I posted in the 16 October update please update this ticket and I’ll re-open the issue.

Cheers, -Jeff

#14 Updated by Curtis Ruck over 3 years ago

Ok, so i’m trying to let puppet master (running under httpd) use the new location.

So when the puppet agent runs on the puppet master itself, it can’t trust the ‘self signed certificate in certificate chain for “/CN=Puppet CA: …”’ When I find /var/lib/puppet -name \*.pem i end up with two sets of ca certificates because the puppet agent generated its own set.

#15 Updated by Curtis Ruck over 3 years ago

So looking at my issues deeper…

I am using a puppet manifest to build my puppet master (for production). So my steps currently are creating the two sets of certificates (including CAs).

1) su – puppet -s /bin/bash -c “puppet master —no-daemonize -v”
2) puppet apply /etc/puppet/manifests/build.pp

In Step 1, it generates the /var/lib/puppet/.puppet/ssl directory. In Step 2, it generates the /var/lib/puppet/ssl directory.

I can’t seem to get #2 to not generate its own CA and certificates.

#16 Updated by Anonymous over 3 years ago

Curtis Ruck wrote:

Ok, so i’m trying to let puppet master (running under httpd) use the new location.

So when the puppet agent runs on the puppet master itself, it can’t trust the ‘self signed certificate in certificate chain for “/CN=Puppet CA: …”’ When I find /var/lib/puppet -name \*.pem i end up with two sets of ca certificates because the puppet agent generated its own set.

If you remove the CA certificate the agent uses, then the agent will automatically re-downloaded it from the master the next time it connects.

The CA certificate the agent uses is located at the path returned by the puppet agent --configprint localcacert command.

I’m going to leave this closed because this doesn’t seem like it’s a problem with confdir and vardir having a bug, but rather the CA certificates being out of sync.

On this subject, could we take the discussion of this issue to the puppet-users mailing list? I think there will be a higher chance of resolving the issue you’re facing if the discussion happens there. The issue you’re facing seems like a closely related, but separate, issue to the original bug this ticket refers to.

Hope this helps, -Jeff

#17 Updated by Anonymous over 3 years ago

Curtis Ruck wrote:

So looking at my issues deeper…

I am using a puppet manifest to build my puppet master (for production). So my steps currently are creating the two sets of certificates (including CAs).

1) su – puppet -s /bin/bash -c “puppet master —no-daemonize -v”

Ah, I think I see your problem. When you start Puppet as non-root, it’s going to use the $HOME directory of the puppet user instead of the system wide $vardir.

I think if you change your script to be something like this, you should be OK. But remember, you’re going to have to make sure the puppet user is able to write into the system directories.

# These run as root, so they'll return the system directories:
confdir="$(puppet master --configprint confdir)"
vardir="$(puppet master --configprint vardir)"
# Have the non-root process use the system directories
# instead of the default personal directories:
su - puppet -s /bin/bash -c "puppet master --confdir='${confdir}' --vardir='${vardir}' --no-daemonize -v"

2) puppet apply /etc/puppet/manifests/build.pp

In Step 1, it generates the /var/lib/puppet/.puppet/ssl directory. In Step 2, it generates the /var/lib/puppet/ssl directory.

I can’t seem to get #2 to not generate its own CA and certificates.

Step 1 is running as non-root so it has different default directories than Step 2, which is running as root. I think you just need to make these two steps match up. You could either run them both as the same effective UID, or explicitly set confdir and vardir on one or both to match up with the other.

Hope this helps, -Jeff

#18 Updated by Roman Chyla about 3 years ago

Jeff, may I suggest sb creates a unittest for this behaviour? As you say in some releases regressions happen and I am right now on a fresh Puppet installation, 3.0.1., I used puppetlabs repo to install puppetmaster. And I get the same error (but as root)

my puppet.conf

[main]
    ssldir = $vardir/ssl
    report=true
    server=adswhy
    pluginsync=true
    factpath = $vardir/lib/facter:$vardir/facts:/etc/puppet/facts

output

adswhy-56: sudo puppet config print factpath
[sudo] password for rchyla: 
Error: Could not intialize global default settings: Error converting value for param 'factpath': Could not find value for $vardir
adswhy-57: puppet config print factpath
/home/rchyla/.puppet/var/lib/facter:/home/rchyla/.puppet/var/facts

and if i comment factpath, i get:

adswhy-59: sudo puppet config print factpath
/var/lib/puppet/lib/facter:/var/lib/puppet/facts
adswhy-60: puppet config print factpath
/home/rchyla/.puppet/var/lib/facter:/home/rchyla/.puppet/var/facts

Please note that $vardir is used by ssldir in the [main] section, and it is fine (am I doing st wrong?)

To me it is scary to think that the same variation of error appears when it was fixed already

#19 Updated by Anonymous about 3 years ago

Roman Chyla wrote:

Jeff, may I suggest sb creates a unittest for this behaviour? As you say in some releases regressions happen and I am right now on a fresh Puppet installation, 3.0.1., I used puppetlabs repo to install puppetmaster. And I get the same error (but as root)

my puppet.conf

[…]

output […]

and if i comment factpath, i get:

[…]

Please note that $vardir is used by ssldir in the [main] section, and it is fine (am I doing st wrong?)

To me it is scary to think that the same variation of error appears when it was fixed already

This actually looks like a different issue in our settings subsystem. It still exists in 3.1.0 and our master development branch. Roman, would you mind filing this problem as a new issue and assign it to me? I’ll update the new issue with my investigation, but the use case you’re trying to achieve will be important to understand the context of the issue.

Thanks, -Jeff

Also available in: Atom PDF