Puppet

What’s the progress on this bug? Are we still committing to pushing this out in 0.25.x?

I care much more about it being fixed in 2.6.x than I do about 0.25.x assuming the latter is a more difficult patch to backport.

Nigel —

It’s part of a nest of related issues that are stalled on a design question: what is the authoritative source of the environment? I’ll take a look & see if there’s a better answer, but off the top of my head the problem is that the puppetmaster isn’t required to abide by the client’s environment selection but never informs the client of this. The client thus requests files in the environment it thinks it’s in (the environment is part of the request) and the puppetmaster (when fileserving) does abide by the client’s selection so…

— Markus

If the client began by asking the server “what environment do you think I should be in?” (issue #2748) then I think it would solve this problem.

I don’t actually use envirements. I’m just throwing out an idea.

What if you can set on the client what the environment is and one of the options is “auto” or “let_server_choose” or something similar. Then you make the the default on the client.

Would that be helpful?

I think that if you want the server to be authoritative over the client environment, you just allow the client to not define an environment, and if it does and the server overrides it, then a warning message is printed.

If you want the client to be authoritative over the environment, then you just don’t set it in your external node classifier.

How much compatibility is needed here? My gut says “a lot”.

I propose an option for puppet.conf that can configure the puppetmaster to behave as authoritative, non-authoritative, or legacy (a.k.a. broken), which should be the default.

However, when set to authoritative mode, old clients must be entertained, and I don’t suppose there is a way for the puppetmaster to tell them the environment they are to use. So when set to authoritative, the puppetmaster should detect old clients and switch their sessions to non-authoritative instead.

This way, existing setups aren’t broken, and users that choose to set the option get a coherent (although potentially confusing) result.

I don’t understand what about the current behavior needs to be preserved. Why do you think we need a legacy mode Felix?

Let me flesh out the proposal.

We have a setting for whether the master is authoritative over the client environment.

Server authoritative:

• Server sets environment and client sets environment, produce warning client/server side, use server env.
• Server does not set environment, client does, use client env.
• Server sets environment, client does not, use server env.

Server not authoritative:

• Server sets environment and client sets environment, produce warning server-side, use client env.
• Server does not set environment, client does, use client env.
• Server sets environment, client does not, use server env.

This does all assume that the idea of a “default” environment goes away client-side. I don’t see that being a huge problem, as users can simply set their default environment on the server, and they can still choose whether or not the server is authoritative.

Am I missing some use cases ?

Case “Server not authoritative” is what I am looking for. We use an external node classifier which always sets the environment. I’d like the ability to occasionally to override the server from the client for testing/debugging

Nigel Kersten wrote:

Let me flesh out the proposal.

We have a setting for whether the master is authoritative over the client environment.

Server authoritative:

• Server sets environment and client sets environment, produce warning client/server side, use server env.
• Server does not set environment, client does, use client env.
• Server sets environment, client does not, use server env.

Why do you say the 2nd here? We shouldn’t trust the client ever if the master is set authoritive even if there’s been human error – forgot to set it on the master – it should default to production if master authoritive that seems to be the sane combo and means the 2nd point above can go

Server not authoritative:

• Server sets environment and client sets environment, produce warning server-side, use client env.
• Server does not set environment, client does, use client env.
• Server sets environment, client does not, use server env.

This does all assume that the idea of a “default” environment goes away client-side. I don’t see that being a huge problem, as users can simply set their default environment on the server, and they can still choose whether or not the server is authoritative.

Why would the client when set authoritive not default to production? If we ship with this as default there’s no surprises for users when they upgrade but the people who wants new behavior can enable it, no surprises to anyone.

To bring up another point that is probably out of scope for this bug, but others may think differently.

The config file option “environment=foo” means completely different things in [master] and [agent].

In the latter case it means “set my environment to this”. In the former case it means “set the default node environment to this”.

The problem is when users set the value in [main], it flows down to the other blocks and means different things in both.

Nigel Kersten wrote:

We have a setting for whether the master is authoritative over the client environment.

Why do you need such a setting? It seems to add needless complexity.

Here’s my proposal.

From the client’s perspective:

1. The client/server interaction always begins with the client asking the server “I think my environment should be , and here are my facts and certificates and stuff; what do you think my environment should be?”

2. In response to (1), the server informs the client of what environment to use. The server is free to believe the client’s environment, or not, according to server configuration, which the client doesn’t need to know about.

3. The environment reported by the server to the client in (2) above is always authoritative. If the client doesn’t like it, then its only recourse is to fail.

From the server’s perspective:

1. No new configuration settings are needed.

2. If there’s an external node classifier, and if it specifies an environment, then that wins.

3. The external node classifier should have access to the client’s idea of the environment, and all the facts (see issue #2834), so the external node classifier is free to let the client’s idea of the environment win, or not, according to arbitrarily complex conditions.

4. If there’s no external node classifier, or if the external node classifier is silent about the environment, then the client’s environment is used, but if the client didn’t specify an environment, then the server’s default environment is used.

From the administrator’s perspective:

1. If you want the client’s environment to win, then don’t use an external node classifier, or use an external node classifier that doesn’t override the client’s idea of the environment.

2. If you want the server’s environment to win, then use an external node classifier that overrides the client’s environment.

3. If you have more complex requirements, then use a more complex external node classifier that sometimes overrides the client’s environment and sometimes does not override it.

I agree with Alan. That’s the simplest and principle of least surprise approach.

Alan’s proposal is fine, although I feel arbitrarily puts you at a disadvantage if you use an external node classifier.

I can live with this proposal, if and only if server point 2 is implemented: “The external node classifier should have access to the client’s idea of the environment, and all the facts”

I like that proposal too, but it doesn’t cover all the use cases I’ve had laid out in front of me over this issue.

I’m going to put together a more fleshed out post with all the use cases after the holiday season, as we’re going to have to either swallow complexity or deny some use cases.

I’ve been having similar problems and Nigel asked me to update this bug with what we’re looking for.

Alan’s proposal is absolutely fine for us, but I would suggest an alternative mode whereby if puppet is run without a puppet.conf then it simply attempts to connect to the hostname ‘puppet’ and if it is able to make the connection it retrieves all configuration from the external node classifier. This would allow foreman to be my authoritative source of configuration for clients and remove the need to negotiate anything between server/client.

Alan Barrett wrote:

1. The client/server interaction always begins with the client asking the server “I think my environment should be , and here are my facts and certificates and stuff; what do you think my environment should be?”

In order for the client to present its facts, it must first have downloaded any custom facts from the server. Which environment should it get the facts from?

1. The environment reported by the server to the client in (2) above is always authoritative. If the client doesn’t like it, then its only recourse is to fail.

Really? You’re not very paranoid… I can think of another thing the client can do: continue to use the environment it wants, and see if it can trick the server into giving out things it shouldn’t.

The only way the server can be really authoritative about which environment a client gets, is for the server to decide the environment itself, every time the client asks for something, be it a file (including plugins) or a catalog. It must not trust that the next request from the client will be asking for the environment the server told it to use a nanosecond earlier. That basically means the server must call the node classifier for every client request. (It may be acceptable to cache the results of the classifier for a short time.)

Of course, if you only want to protect against mistakes (someone happened to write “test” instead of “production” in puppet.conf a late night), then your scheme will do. But I suspect some people will want protection against deliberate circumvention attempts as well.

I honestly like the fact that I can run a client with puppet agent —environment=development and have that machine enter that environment.

In fact, I even do something similar to this: puppet.conf.erb:

[agent]
environment = <%= environment %>

After all, I do have the option of “hard coding” an environment on my server with PUPPET_EXTRA_OPTS=“—environment=production” from /etc/sysconfig/puppet

What I don’t like however is that I have to symlink modules from my development modulepath into the production modulepath when I’m testing new modules and providers because the server looks in the wrong environment.

I’m hitting this bug now with 2.6.6

    ...when the client specifies one environment and the classifier specifies another,
 classes are evaluated from the server-specified environment, 
 and yet files are retrieved from the client-specified environment

Our biggest problem now is not which one is authoritative, but the fact that it’s half working when using an external node classifier to set the environment. So it would be nice to have them asap at least consistent (classes and files evaluated from the same environment).

Now we get classes silently evaluated with the wrong files (with nasty results).

Btw, my node classifier is configured to tell “myhost” it should belong to “mycustomenv” environment. Then on the server I’ve noticed this :

> grep environment /var/lib/puppet/yaml/facts/myhost.yaml

  environment: production

> grep environment /var/lib/puppet/yaml/node/myhost.yaml

  environment: mycustomenv
    environment: production

Which just doesn’t seem quite ok to me (although I can’t really judge the impact of having 2 environment entries in the node yaml file).

Thomas Bellman wrote:

The only way the server can be really authoritative about which environment a client gets, is for the server to decide the environment itself, every time the client asks for something, be it a file (including plugins) or a catalog.

I agree, but that’s a much larger change, and I think we can get to a very useful place without implementing that, even though we should probably aim for that in the future.

How about this for a proposal?

We add the client-specified-environment as $2 for the ENC, so people can choose to implement logic around server-specified-environment vs client-specified-environment in their ENC.

If the server specifies an environment, that is what we use.

This seems to solve the most common problem, and we can debate the utility of “is server authoritative?” settings later.

Can somebody put a mention of this bug in the ENC documentation? As a new user trying to setup using environments, modules, and an ENC it was very confusing to have the fileserver broken out of the box. Right now we have http://docs.puppetlabs.com/guides/external_nodes.html which says:

“The value of the environment key is a string representing the master’s preferred environment for this agent node. The interaction between agent-specified and master-specified environments is currently under active design consideration.”

I suggest adding “For now you must set the environment in the client’s puppet.conf in order for the fileserver to work as expected.”

Thank you for that reminder Aaron. #7595

This is the current first step proposal:

If the server sets the environment for a client, that environment is embedded in the catalog the client receives. If the client sees an environment in a catalog it is due to apply, that’s the one it uses.

Additionally, we add the client-specified environment as $2 for the ENC.

Nigel Kersten wrote:

Additionally, we add the client-specified environment as $2 for the ENC.

Passing additional arguments is an ABI change for the ENC script: previously we documented that the script was invoked with a single argument, the node name. However, an easy and compatible work around is to establish the data in the process environment. An aware ENC script can collect from there, and an unaware script will not see the change.

This makes the change safe for the 2.7.x tree.

+1

This is definitely a high priority fix for us as well. I am currently working around this bug with a major hack :(

What is the current status? I’m having the same issue … Only work-around that we use currently is templating

I hope that this will be fixed in 0.25 as well –> Markus Roberts 1 year ago “Fix will be wanted in the 0.25.x series as well.” (we are still on version 0.25.5 currently)

Hi there,

we are using puppet on a very large number of nodes and we are facing the exact same problem. This is really a major issue for us.

so long and thx all the puppets ;)

++ fat

+1: the work-around is kind of a pain when dealing with more than a few servers, not to mention having to remember that you’ve done the work-around in the first place and need to change environments.

Is there any ETA for a fix for this bug? We are also having the same issue in a production environment. I haveto swap environment temporary on servers to work around this bug; and I know its gonna bite me a day..

If you’re struggling with environment changes and your server should be authoritative over the environments: I have pushed a workaround, that is in my opinion not a major hack, to: https://github.com/duritong/handle-puppet-environment

It is simply extending the current dashboard ENC script and is using 2 puppet runs (where the first one is a really safe one) to properly switch the environment. Comments are welcomed on github as pull-requests or whatever.

And as a side-note: Although, making the environment available to the ENC will safe us from storing the previous ENC-data to compare the two environments, we will still have to use 2 puppet runs to safely switch an environment. Otherwise, plugins such as facts and providers will still be synced with the wrong environment and also client file sources will be wrong.

Not sure if this should be a separate bug:

Passing environment on master for compilation results in it being completely ignored:

puppet master --compile raiden.local --environment=test
notice: Scope(Node[default]): scope production
notice: Scope(Node[default]): production

This works from the client, but it’s ineffective to compile large number of catalogs by changing the puppet master configuration:

info: Expiring the node cache of raiden.local
info: Not using expired node for raiden.local from cache; expired at Fri Dec 02 10:58:06 -0500 2011
info: Caching node for raiden.local
notice: Scope(Node[default]): scope production 
notice: Scope(Node[default]): production
notice: Compiled catalog for raiden.local in environment production in 0.03 seconds
info: Expiring the node cache of raiden.local
info: Not using expired node for raiden.local from cache; expired at Fri Dec 02 10:58:18 -0500 2011
info: Caching node for raiden.local
notice: Scope(Node[default]): scope test 
notice: Scope(Node[default]): test

I just wanted to give the watching world an update on this: this is actually a fairly hard problem, because fixing it requires a change to the protocol between the Puppet master and the ENC.

We are currently working on some design around that change, and how to keep it compatible, but ultimately it should empower the ENC to fully own all the data required to make this work.

There is no committed timeline to a fix, but we hope it will be addressed soon – and, ideally, in a way that we can deploy as part of the 2.7 series, rather than forcing everyone to wait on a new major version.

The plan is: If the server sets the environment for a client, that environment is embedded in the catalog the client receives. If the client sees an environment in a catalog it is due to apply, that’s the one it uses.

In that case an attack is possible of retrieving information (and possibly sensitive information like passwords) from side environments.

On Wed, Jan 25, 2012 at 10:08 AM, tickets@puppetlabs.com wrote:

Issue #3910 has been updated by Timur Batyrshin.

In that case an attack is possible of retrieving information (and possibly sensitive information like passwords) from side environments.

Timur,

Could you please explain this attack you’re thinking of in more detail? I don’t understand the attack and I’m trying to understand this better.

-Jeff

I’m sorry I seem to have misunderstood you at first.

If the server is able to override environment sent by client it would be ok.

Yes, but to be clear, if the agent is applying a cached catalog, the environment in the catalog will be trusted as the authoritative source, and the ENC will not be consulted.

This doesn’t expose any more than we have now, where strictly speaking you could retrieve a file that isn’t in your catalog or your environment.

In the future we’ll be looking to enforce better partitioning there, but if you have particularly sensitive data, you can restrict access using auth.conf and/or fileserver.conf with static mountpoints and explicit controls.

Another relatively simple option is to not expose such passwords as files served by the fileserver, but instead use functions to insert them into manifests to avoid leakage.