Puppet

This is a missing feature, not a bug. I don’t think it will be implemented for a while yet, since it requires considerable changes in the library.

This is now not all that difficult, now that resources can generate new resources at run-time using ‘eval_generate’, but it will require additional configuration, maybe ‘recurse => upward’ or something, although that would make it impossible to recurse both directions.

Marked #668 as a duplicate, and fixed description.

A usable workaround exists (courtesy of Volcane on #puppet):

file{ commit:“/foo”, “/foo/bar”, “/foo/bar/baz”: ensure => directory }

I’d also like this fixed too…

berge wrote:

A usable workaround exists (courtesy of Volcane on #puppet):

file{ commit:“/foo”, “/foo/bar”, “/foo/bar/baz”: ensure => directory }

fragfutter wrote:

a discussion on #puppet lead to the question “who owns the directories?”. If puppet creates directories recursive, should it own them (setting permissions and stuff) or not. And should it also set rights on existing folders?

To avoid implicit declaration one possible syntax would be:

file { “/var/long/path/here”: ensure => “directory”, recursive => “/var/long”, owner => “foo”, }

this would update ownership (and create directories if they are missing) for /var/long/path and /var/long/path/here

The solution proposed by berge seems to be less generic but more explicit.

Here’s a suggestion that I made recently on IRC which addresses the issue of predictability.

Given a definition something like this:

file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true }

This should create /some, /some/very, /some/very/deep with the same perms/ownership as / and create /some/very/deep/path with the specified perms/ownership

If the definition changes to:

file{“/some/very/deep/path”: ensure => directory, owner => ‘foo’, create_parents => true }

then the ownership of /some/very/deep/path would be changed, but the parent dirs would not be affected

eh, what’s the difference between:

file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true }

and

file{“/some/very/deep/path”: ensure => directory, owner => ‘foo’, create_parents => true }

then the ownership of /some/very/deep/path would be changed, but the parent dirs would not be affected

as I don’t see the difference (except at one point root is the owner and @ the other foo) I assume you forgot to remove the create_parents?

Still, what about:

file{"/var/lib/cache/foo/bar": ensure => directory, mode => 0700, create_parents => true }

as far as I understood your proposal this would set permission even on /var/lib/cache and upwards, which definately wouldn’t be what you’d like to do. So something like a stop point: /var/lib/cache/foo is imho definately necessary. So something like fragfutter’s solution still seems to be necessary or you go with the current solution.

I’ll try to answer based on my understanding of the IRC conversation yesterday, but of course Robin can followup..

Peter Meier wrote:

as I don’t see the difference (except at one point root is the owner and @ the other foo) I assume you forgot to remove the create_parents?

no, the create_parents can stay. here is the scenario: file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true } You have this code in place for a while and it deploys to 10 hosts.. then you change owner to ‘foo’ which gets updated on those 10 hosts and also deploys to 10 new hosts. With this strategy, any ancestor directories that need to be created get perms/owner/group copied from the deepest existing parent, and the leaf uses the resource settings. So say / has 755/root/root on all your systems. Then the first 10 hosts get /some, /some/very and /some/very/deep also created with 755/root/root. /some/very/deep/path originally gets created with owner root, and the update only changes that dir to owner ‘foo’. The next 10 systems will result in everything matching because /some, /some/very and /some/very/deep again copy upwards from /, and only the leaf gets owner ‘foo’. Hope this helps.

as far as I understood your proposal this would set permission even on /var/lib/cache and upwards, which definately wouldn’t be what you’d like to do. So something like a stop point: /var/lib/cache/foo is imho definately necessary. So something like fragfutter’s solution still seems to be necessary or you go with the current solution.

I think in this example /var/lib/cache/foo would be created with matching perms/owner/group as /var/lib/cache, and “bar” would be created with mode 700.

So far all these suggestions are kind of unappealing to me. The idea of perms being copied from the deepest existing parent is somewhat counter-intuitive to me.

Just my 2c. I realize I’m more than welcome not to use such a feature if it is implemented :)

I’m dead against anything that is non deterministic, giving file this ability means you make it on par with installing a package with yum/apt etc and getting a ton of unknown dependencies. You’d pick up on weird things like the package situation in dev ofcourse but what about file{} as suggested above?

file{“/some/very/deeply/nested/file”: create_parents => true}

in most cases this is fine as described, you get the parent dirs made once in a sane and predictable way.

should I add file{“/some/very”: ensure => directory, owner => “another” } though the file resource above would not fix it. ie. /some/very/deeply directory would not get adjusted to match the new parent dir ownership – see below.

We just do not have state anywhere to say a file as in the first block also created all the parents at some point months ago and store that for all future invocations to then also create these additional resources. Even creating resources on the fly to fill those gaps wont work because presumably on the 2nd run the logic that made all the parent resources will be skipped since hte parents exist already and as described we will leave existing dirs alone, else where do you draw the line between not editing / ? or between not editing /var/cache but letting it edit /var/cache/foo? there’s way too many edge cases.

This further breaks if you have something like class apache::install { } that sets up the parent dir for all your vhosts and such using this feature, on another machine perhaps you created one of the parent dirs in another class already using file{}. The result would be that if you included apache::install on different machines the result across different type of machine would not be the same and would not be predictable.

If you used specifically specified resources for each of the parent dirs you’d know you’re causing a screwup cos it just wont compile thanks to the duplicate resource checks rather than silently creating a false sense of certainty that your machines are all behaving in the same manner because puppet built them and puppet is declaring the full known state.

I can imagine many many cases where this behaviour would result in unpredictable results either through the life of a machine, through the life of other machines that’s supposedly just like this one (by including all the same classes) or through other machines that tries to reuse classes used on this one. The predictable behavior of puppets core types – like file, etc and unlike augeas for example – is what makes it a usable tool.

Any added features that downgrade core types like file{} to something that’s unpredictable should be very very carefully considered.

We just do not have state anywhere to say a file as in the first block also created all the parents at some point months ago and store that for all future invocations to then also create these additional resources. Even creating resources on the fly to fill those gaps wont work because presumably on the 2nd run the logic that made all the parent resources will be skipped since hte parents exist already and as described we will leave existing dirs alone, else where do you draw the line between not editing / ? or between not editing /var/cache but letting it edit /var/cache/foo? there’s way too many edge cases.

This further breaks if you have something like class apache::install { } that sets up the parent dir for all your vhosts and such using this feature, on another machine perhaps you created one of the parent dirs in another class already using file{}. The result would be that if you included apache::install on different machines the result across different type of machine would not be the same and would not be predictable.

good catch about getting unpredictable, even the one over time. running the same puppet manifest on 2 hosts should get the same result.

I have created myseld a patch on version 0.25.4

Then you can create “mkdir -p”

file { "/data/mysql/$hostname/db":
   ensure => directory,
   recurse => true
}

diff -up type/file/ensure.rb_ORIG type/file/ensure.rb
--- type/file/ensure.rb_ORIG    2010-02-09 10:01:29.000000000
+0100
+++ type/file/ensure.rb 2010-02-09 10:14:34.000000000 +0100
@@ -61,10 +61,14 @@ module Puppet
        newvalue(:directory) do
            mode = @resource.should(:mode)
            parent = File.dirname(@resource[:path])
-            unless FileTest.exists? parent
-                raise Puppet::Error,
-                    "Cannot create %s; parent directory %s does not exist" %
-                        [@resource[:path], parent]
+            if @resource.recurse?
+                FileUtils.mkdir_p(parent)
+            else      
+                unless FileTest.exists? parent
+                    raise Puppet::Error,
+                        "Cannot create %s; parent directory %s does not exist" %
+                            [@resource[:path], parent]
+                end       
            end       
            if mode   
                Puppet::Util.withumask(000) do

I have created myseld a patch on version 0.25.4

Then you can create “mkdir -p”

file { “/data/mysql/$hostname/db”: ensure => directory, recurse => true }

this is, hmm, pretty ugly to abuse recurse like that. recurse has basically a totally different meaning. Anyway, what do you do if you set this file resource to absent? As noted in different parts of the discussion, the behavior becomes like that very inconsistent.

Idd it can be confusing and inconsistent If absent you can’t permit to delete also the parents it could be very painful

but I want this feature and I now the limits of it

Maybe if you write into the documents that if you delete a recurse directory that the parents won’t be deleted

Likes fair to me

Lots of reasons why we can’t do this, but it would be nice to have. I think we should list the limitations and side effects and still have it, but not make “absent” work.

Even if we just had …

File { expand(“/tmp/path/one/two”,1): ensure => directory } as shorthand for listing the arrays of each directory without including part of the root (which might be a duplicate).

Or just add mkdir -p into the provider (as mentioned) and surround it will skulls and crossbones about how it determines permissions, following what mkdir -p does for the most part, and listing when it’s a bad idea. After all, even Haskell has some provision for side effects :)

Longer term, if we duplicate resources, and they are 100% the same, maybe that’s not an error and they resolve to the same resource. I don’t know.

—Michael

I disagree.

If it’s just a “nice to have” then we have it. exec { “mkdir -p /foo/bar/foo” }

Consistency is important and I don’t see a good way of offering this functionality in the file type and still being consistent.

Handling creating a directory tree should not be any different than “mkdir -p” when creating a directory tree.

Handling complex permissions should be a separate transaction/resource.

A stipulation can be made that when a directory tree is made, the “tree” only can have one set of permissions/owner/group – as specified in the resource. This would start with the directory that is missing, down to the leaf. –

ie – if /opt/local was there, and the resource called for creation of /opt/local/lib/mylib/mysubdir the permissions/owner/group will start with the newly created directories, starting with “lib”, and including “mylib” & “mysubdir”.

Absent could just be a dangerous “rm -rf” at the leaf level specified. ie – using the above example if one wants to get rid of mylib/mysubdir, one would say

path    => "/opt/local/lib/mylib",

ensure => absent, recurse => true,

As said above, any complex permissions/owner/group should be handled separately.

If people want the file resource to chown, chmod or chgrp with a “-R/r”, that could also be done, but as a separate bug/feature. (I would say something similar to absent… start at the point specified in “path” or “name”)

exec’s ok… but from the http://docs.puppetlabs.com/guides/types/exec.html page:

There is a strong tendency to use exec to do whatever work Puppet can’t already do; while

this is obviously acceptable (and unavoidable) in the short term, it is highly recommended to migrate work from exec to native Puppet types as quickly as possible. If you find that you are doing a lot of work with exec, please at least notify us at Reductive Labs what you are doing, and hopefully we can work with you to get a native resource type for the work you are doing.

Sounds like this is a feature that a lot of people want.

Just need to agree on what the feature does :)

Regards, -Roy

A stipulation can be made that when a directory tree is made, the “tree” only can have one set of permissions/owner/group – as specified in the resource. This would start with the directory that is missing, down to the leaf. –

and this is exactly where this resource becomes unpredictable if on one host /foo/bar/a exists but on one one host only /foo/bar and your tree should create /foo/bar/a/b/c one one host puppet would manage (owner, etc.) folder a on the other not, where it might be different and for example not chmod 0755 which would break access to your leave for the daemon, which would make the puppet-run incomplete and not able to set everything correctly. Or: what if you have 2 trees: /foo/bar/a/b/c and /foo/bar/a/e/f in one tree the owner is set to root in the other it is set to mail. On a vanilla host /foo doesn’t exist. So what is owner /foo? Maybe root maybe mail? depends on which host which tree resource would have been applied earlier.

Furthermore in the second run: how would puppet know from which part of the tree it manages that resource and what should it do if somebody changes in the above example permission of b ? In the first run it would set them correctly and then somebody changes it and what would it do then in the second run?

ie – if /opt/local was there, and the resource called for creation of /opt/local/lib/mylib/mysubdir the permissions/owner/group will start with the newly created directories, starting with “lib”, and including “mylib” & “mysubdir”.

Absent could just be a dangerous “rm -rf” at the leaf level specified. ie – using the above example if one wants to get rid of mylib/mysubdir, one would say

path    => "/opt/local/lib/mylib",

ensure => absent, recurse => true,

and this makes it somehow very inconsistent: you wouldn’t have the same state as if you first manage the tree and then set it to absent, you would have some parts of the tree left, which means that creating and removing a resource wouldn’t do the same.

As said above, any complex permissions/owner/group should be handled separately.

If people want the file resource to chown, chmod or chgrp with a “-R/r”, that could also be done, but as a separate bug/feature. (I would say something similar to absent… start at the point specified in “path” or “name”)

this can imho get very cumbersome. furthermore as puppet has the paradigma to manage a resource only once in a manifest you wouldn’t be able to set the flags with a second file resource.

[…] Sounds like this is a feature that a lot of people want.

Yes, I agree on that. On the other side I can say that within the > 100 modules I wrote I encountered that problem maybe 10 times and half of them I realized that I create anyway a common structure for my systems which I can easily facter out into a seperate module and include that everywhere. And due to the auto-require of paths and the possibility to pass an array to the file resource the overhead is minimal. I know that minimal is somehow ugly, but:

Just need to agree on what the feature does :)

and how it fits into the model that puppet tries to ensure and so far nobody came up with a solution that is consistent and is easy to use.

…

and this is exactly where this resource becomes unpredictable if on one host

/foo/bar/a exists but on one one host only /foo/bar and your tree should create /foo/bar/a/b/c

so – how do you do this now?

one one host puppet would manage (owner, etc.) folder a on the other not, where it might 

Yes, if one wanted/needed a specific set of ownership on a tree, or part of a tree, that could/should be managed separate from the “create”. This would mandate the use of a “before” or “require”…

be different and for example not chmod 0755 which would break access to your 

leave for the daemon, which would make the puppet-run incomplete and not able to set everything correctly. Or: what if you have 2 trees: /foo/bar/a/b/c and /foo/bar/a/e/f in one tree the owner is set to root in the other it is set to mail. On a vanilla host /foo doesn’t exist. So what is owner /foo? Maybe root maybe mail? depends on which host which tree resource would have been applied earlier.

see above.

Furthermore in the second run: how would puppet know from which part of the tree it manages that resource and what should it do if somebody changes in the above example permission of b ? In the first run it would set them correctly and then somebody changes it and what would it do then in the second run?

I see this (#86) as a “create” issue – not an issue of maintaining tree permissions. That should be done as a different resource, with this as a “before”, or be “required”.

I see the “create_dir” option (maybe of ensure => ?) to handle ensuring the directories are created.

but… I’m rather newish to puppet and target managing clients not servers so I may not be seeing this through the same pair of glasses :)

Regards, -Roy

Would it be possible to do something like:

file { “/foo/voo/bar/baz” :

ensure => directory, between => {

start => "/foo", end => "/foo/voo/bar", owner => "foo", group => "foo", mode => 755, }

between => {

start => "/foo/voo/bar/baz", end => "/foo/voo/bar/baz", owner => "baz", group => "baz", mode => 775, }

}

or would that require re-writing chunks of puppet?

Regards, -Roy

I feel like the fact that there are edge cases that might not be handled correctly has stymied implementation of the most common case (create directories up to the specified one with the ownership/perms that are specified) and it’s making a lot of things way more difficult than it needs to be. I get asked incredulously at least a couple times a month by other SAs I’m introducing to puppet “You mean there’s no way to ‘mkdir -p’ an arbitrary path?” Any chance we could machete through the brambles and get a caveat-laden feature in?

I still hear this question sometimes ‘Hey how do you do “mkdir -P” in puppet?’ …

Maybe we should create simply the option “parents => true/false” and ignore that option when using “ensure => absent”

if it is documented I don’t see any problem off-course there are always arguments in some cases, well don’t use it then! (freedom of choice) but if it is not implemented you don’t have a choice

I’ve yet to see a consistent and predictable suggestion for how we accomplish this.

If anyone wants to take a stab at providing that, it would be immensely helpful.

The general principles I believe we should be favoring are: * Explicit is preferable to implicit * Consistency is important * Avoid abusing existing parameters * Avoid a proliferation of extra parameters.

Additionally, can we try to think outside the box a little? Is it possible that we could solve this by being able to instantiate resources with an array of hashes where missing values are undef?

$my_dirs = [ { path => "/a" }, { path => "/a/b" }, { path => "/a/b/c", owner => "root", mode => 0755 } ]
file { $my_dirs: }

What about the original way then?

file { [‘/a’, ‘/a/b’, ‘/a/b/c’]: owner => “root”, mode => 755 }

I’m guessing there are to many thoughts about this topic and it will never be solved sins there is a conflict in the possible solutions

I will continue to patch my puppet with “mkdir -p” where the customer wants to @home I just take the time to configure it like [‘/a’, ‘/a/b’, …

:)

You mean the way that works now? I’m perfectly happy with that, but the whole feature request here is because people aren’t :)

The reason I suggested the hash method is because it sounds like people want to behave like mkdir -p, where they really don’t care about the permissions settings of parent directories.

I think this is a bad idea, because I believe in being explicit, but I offered the hash method as a way for people to only specify directory properties for some of the directories in the tree.

Any other suggestions other than the method that works now?

I’m tempted to close this to reduce some of our immense ticket debt, as I haven’t seen anyone complain about this for a while now.

Hi Nigel,

it will not be possible to satisfy everyone here, so I will continue as it is designed right now For me to “case closed”

greets Mario,

“Ruby DSL”

http://www.puppetlabs.com/blog/ruby-dsl/