The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Feature #86

Directory creation fails if parent directory does not exist

Added by Redmine Admin about 10 years ago. Updated over 1 year ago.

Status:RejectedStart date:
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:file
Target version:-
Affected Puppet version:0.24.7 Branch:
Keywords:feature

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com


Description

I tried

file {“/usr/local/share/puppet/sopext/facter” : ensure => directory, recurse => true }

but get erros that the parent diretories are not available.

dirs_between.rb Magnifier - dirs_between() function, quick and dirty implementation (625 Bytes) Thomas Bellman, 08/19/2009 09:38 am


Related issues

Related to Puppet - Feature #2545: Ability to do "mkdir -p" with a file resource Duplicate 08/17/2009
Related to Puppet - Feature #3522: a "mkdir -p" option for the file type Duplicate 04/08/2010
Duplicated by Puppet - Feature #10340: file resource type should support creating parent tree Duplicate 10/27/2011

History

#1 Updated by Luke Kanies almost 9 years ago

Marked #668 as a duplicate, and fixed description.

#2 Updated by Luke Kanies about 10 years ago

This is a missing feature, not a bug. I don’t think it will be implemented for a while yet, since it requires considerable changes in the library.

#3 Updated by Luke Kanies about 9 years ago

This is now not all that difficult, now that resources can generate new resources at run-time using ‘eval_generate’, but it will require additional configuration, maybe ‘recurse => upward’ or something, although that would make it impossible to recurse both directions.

#4 Updated by Paul Lathrop almost 8 years ago

  • Assignee changed from Puppet Community to Paul Lathrop

I’m gonna take a crack at this.

#5 Updated by Redmine Admin almost 8 years ago

  • Status changed from 1 to Accepted

#6 Updated by Paul Lathrop over 7 years ago

  • Assignee changed from Paul Lathrop to Puppet Community
  • Affected Puppet version set to 0.24.7

I’m gonna have to admit defeat on this one; don’t currently have the bandwidth to get this fixed.

#7 Updated by Berge Schwebs Bjørlo about 7 years ago

A usable workaround exists (courtesy of Volcane on #puppet):

file{ commit:“/foo”, “/foo/bar”, “/foo/bar/baz”: ensure => directory }

#8 Updated by Hari Sekhon about 7 years ago

I’d also like this fixed too…

#9 Updated by Christoph H about 7 years ago

  • Keywords set to feature

a discussion on #puppet lead to the question “who owns the directories?”. If puppet creates directories recursive, should it own them (setting permissions and stuff) or not. And should it also set rights on existing folders?

To avoid implicit declaration one possible syntax would be:

file { “/var/long/path/here”: ensure => “directory”, recursive => “/var/long”, owner => “foo”, }

this would update ownership (and create directories if they are missing) for /var/long/path and /var/long/path/here

#10 Updated by Stéphan Gorget old account about 7 years ago

berge wrote:

A usable workaround exists (courtesy of Volcane on #puppet):

file{ commit:“/foo”, “/foo/bar”, “/foo/bar/baz”: ensure => directory }

fragfutter wrote:

a discussion on #puppet lead to the question “who owns the directories?”. If puppet creates directories recursive, should it own them (setting permissions and stuff) or not. And should it also set rights on existing folders?

To avoid implicit declaration one possible syntax would be:

file { “/var/long/path/here”: ensure => “directory”, recursive => “/var/long”, owner => “foo”, }

this would update ownership (and create directories if they are missing) for /var/long/path and /var/long/path/here

The solution proposed by berge seems to be less generic but more explicit.

#11 Updated by James Turnbull almost 7 years ago

  • Assignee deleted (Puppet Community)

#12 Updated by Robin Bowes over 6 years ago

Here’s a suggestion that I made recently on IRC which addresses the issue of predictability.

Given a definition something like this:

file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true }

This should create /some, /some/very, /some/very/deep with the same perms/ownership as / and create /some/very/deep/path with the specified perms/ownership

If the definition changes to:

file{“/some/very/deep/path”: ensure => directory, owner => ‘foo’, create_parents => true }

then the ownership of /some/very/deep/path would be changed, but the parent dirs would not be affected

#13 Updated by Peter Meier over 6 years ago

eh, what’s the difference between:

file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true }

and

file{“/some/very/deep/path”: ensure => directory, owner => ‘foo’, create_parents => true }

then the ownership of /some/very/deep/path would be changed, but the parent dirs would not be affected

as I don’t see the difference (except at one point root is the owner and @ the other foo) I assume you forgot to remove the create_parents?

Still, what about:

file{"/var/lib/cache/foo/bar": ensure => directory, mode => 0700, create_parents => true }

as far as I understood your proposal this would set permission even on /var/lib/cache and upwards, which definately wouldn’t be what you’d like to do. So something like a stop point: /var/lib/cache/foo is imho definately necessary. So something like fragfutter’s solution still seems to be necessary or you go with the current solution.

#14 Updated by Alan Harder over 6 years ago

I’ll try to answer based on my understanding of the IRC conversation yesterday, but of course Robin can followup..

Peter Meier wrote:

as I don’t see the difference (except at one point root is the owner and @ the other foo) I assume you forgot to remove the create_parents?

no, the create_parents can stay. here is the scenario: file{“/some/very/deep/path”: ensure => directory, owner => ‘root’, create_parents => true } You have this code in place for a while and it deploys to 10 hosts.. then you change owner to ‘foo’ which gets updated on those 10 hosts and also deploys to 10 new hosts. With this strategy, any ancestor directories that need to be created get perms/owner/group copied from the deepest existing parent, and the leaf uses the resource settings. So say / has 755/root/root on all your systems. Then the first 10 hosts get /some, /some/very and /some/very/deep also created with 755/root/root. /some/very/deep/path originally gets created with owner root, and the update only changes that dir to owner ‘foo’. The next 10 systems will result in everything matching because /some, /some/very and /some/very/deep again copy upwards from /, and only the leaf gets owner ‘foo’. Hope this helps.

as far as I understood your proposal this would set permission even on /var/lib/cache and upwards, which definately wouldn’t be what you’d like to do. So something like a stop point: /var/lib/cache/foo is imho definately necessary. So something like fragfutter’s solution still seems to be necessary or you go with the current solution.

I think in this example /var/lib/cache/foo would be created with matching perms/owner/group as /var/lib/cache, and “bar” would be created with mode 700.

#15 Updated by Nigel Kersten over 6 years ago

So far all these suggestions are kind of unappealing to me. The idea of perms being copied from the deepest existing parent is somewhat counter-intuitive to me.

Just my 2c. I realize I’m more than welcome not to use such a feature if it is implemented :)

#16 Updated by R.I. Pienaar over 6 years ago

I’m dead against anything that is non deterministic, giving file this ability means you make it on par with installing a package with yum/apt etc and getting a ton of unknown dependencies. You’d pick up on weird things like the package situation in dev ofcourse but what about file{} as suggested above?

file{“/some/very/deeply/nested/file”: create_parents => true}

in most cases this is fine as described, you get the parent dirs made once in a sane and predictable way.

should I add file{“/some/very”: ensure => directory, owner => “another” } though the file resource above would not fix it. ie. /some/very/deeply directory would not get adjusted to match the new parent dir ownership – see below.

We just do not have state anywhere to say a file as in the first block also created all the parents at some point months ago and store that for all future invocations to then also create these additional resources. Even creating resources on the fly to fill those gaps wont work because presumably on the 2nd run the logic that made all the parent resources will be skipped since hte parents exist already and as described we will leave existing dirs alone, else where do you draw the line between not editing / ? or between not editing /var/cache but letting it edit /var/cache/foo? there’s way too many edge cases.

This further breaks if you have something like class apache::install { } that sets up the parent dir for all your vhosts and such using this feature, on another machine perhaps you created one of the parent dirs in another class already using file{}. The result would be that if you included apache::install on different machines the result across different type of machine would not be the same and would not be predictable.

If you used specifically specified resources for each of the parent dirs you’d know you’re causing a screwup cos it just wont compile thanks to the duplicate resource checks rather than silently creating a false sense of certainty that your machines are all behaving in the same manner because puppet built them and puppet is declaring the full known state.

I can imagine many many cases where this behaviour would result in unpredictable results either through the life of a machine, through the life of other machines that’s supposedly just like this one (by including all the same classes) or through other machines that tries to reuse classes used on this one. The predictable behavior of puppets core types – like file, etc and unlike augeas for example – is what makes it a usable tool.

Any added features that downgrade core types like file{} to something that’s unpredictable should be very very carefully considered.

#17 Updated by Peter Meier over 6 years ago

We just do not have state anywhere to say a file as in the first block also created all the parents at some point months ago and store that for all future invocations to then also create these additional resources. Even creating resources on the fly to fill those gaps wont work because presumably on the 2nd run the logic that made all the parent resources will be skipped since hte parents exist already and as described we will leave existing dirs alone, else where do you draw the line between not editing / ? or between not editing /var/cache but letting it edit /var/cache/foo? there’s way too many edge cases.

This further breaks if you have something like class apache::install { } that sets up the parent dir for all your vhosts and such using this feature, on another machine perhaps you created one of the parent dirs in another class already using file{}. The result would be that if you included apache::install on different machines the result across different type of machine would not be the same and would not be predictable.

good catch about getting unpredictable, even the one over time. running the same puppet manifest on 2 hosts should get the same result.

#18 Updated by Thomas Bellman over 6 years ago

A better way than just saying ‘create_parents => true’, would be ‘parents_upto => “/some/very”’, and having that be exactly equivalent to explicitly specifying every path inbetween “/some/very” and “/some/very/long/path” as file resources with the same parameters.

A slightly different way of achieving this would be to have a function dirs_between() that takes a directory and a subdirectory as arguments and returns a list of all directories inbetween, including the two endpoints. dirs_between(“/some/very”, “long/path”) would return the list [“/some/very”, “/some/very/long”, “/some/very/long/path”].

One would then use that like

    $dirs = dirs_between("/some/very", "long/path")
    file {
        $dirs:
            ensure => directory, owner => "someuser", mode => 0755;
    }

(It would be nice to be able to have the dirs_between() call directly in the file declaration, and not have the temporary variable $dirs, but at least 0.25.0rc1 barfs on that.)

I’m attaching a quick and dirty implementation of dirs_between(). It is sorely lacking in error handling, but seems to work as long as you don’t pass it bad parameters.

#19 Updated by Mario Verbelen about 6 years ago

I have created myseld a patch on version 0.25.4

Then you can create “mkdir -p”

file { "/data/mysql/$hostname/db":
   ensure => directory,
   recurse => true
}

diff -up type/file/ensure.rb_ORIG type/file/ensure.rb
--- type/file/ensure.rb_ORIG    2010-02-09 10:01:29.000000000
+0100
+++ type/file/ensure.rb 2010-02-09 10:14:34.000000000 +0100
@@ -61,10 +61,14 @@ module Puppet
        newvalue(:directory) do
            mode = @resource.should(:mode)
            parent = File.dirname(@resource[:path])
-            unless FileTest.exists? parent
-                raise Puppet::Error,
-                    "Cannot create %s; parent directory %s does not exist" %
-                        [@resource[:path], parent]
+            if @resource.recurse?
+                FileUtils.mkdir_p(parent)
+            else      
+                unless FileTest.exists? parent
+                    raise Puppet::Error,
+                        "Cannot create %s; parent directory %s does not exist" %
+                            [@resource[:path], parent]
+                end       
            end       
            if mode   
                Puppet::Util.withumask(000) do

#20 Updated by Peter Meier about 6 years ago

I have created myseld a patch on version 0.25.4

Then you can create “mkdir -p”

file { “/data/mysql/$hostname/db”: ensure => directory, recurse => true }

this is, hmm, pretty ugly to abuse recurse like that. recurse has basically a totally different meaning. Anyway, what do you do if you set this file resource to absent? As noted in different parts of the discussion, the behavior becomes like that very inconsistent.

#21 Updated by Mario Verbelen about 6 years ago

Idd it can be confusing and inconsistent If absent you can’t permit to delete also the parents it could be very painful

but I want this feature and I now the limits of it

#22 Updated by Mario Verbelen about 6 years ago

Maybe if you write into the documents that if you delete a recurse directory that the parents won’t be deleted

Likes fair to me

#23 Updated by Anonymous about 6 years ago

Lots of reasons why we can’t do this, but it would be nice to have. I think we should list the limitations and side effects and still have it, but not make “absent” work.

Even if we just had …

File { expand(“/tmp/path/one/two”,1): ensure => directory } as shorthand for listing the arrays of each directory without including part of the root (which might be a duplicate).

Or just add mkdir -p into the provider (as mentioned) and surround it will skulls and crossbones about how it determines permissions, following what mkdir -p does for the most part, and listing when it’s a bad idea. After all, even Haskell has some provision for side effects :)

Longer term, if we duplicate resources, and they are 100% the same, maybe that’s not an error and they resolve to the same resource. I don’t know.

—Michael

#24 Updated by Nigel Kersten about 6 years ago

I disagree.

If it’s just a “nice to have” then we have it. exec { “mkdir -p /foo/bar/foo” }

Consistency is important and I don’t see a good way of offering this functionality in the file type and still being consistent.

#25 Updated by Roy Nielsen about 6 years ago

Handling creating a directory tree should not be any different than “mkdir -p” when creating a directory tree.

Handling complex permissions should be a separate transaction/resource.

A stipulation can be made that when a directory tree is made, the “tree” only can have one set of permissions/owner/group – as specified in the resource. This would start with the directory that is missing, down to the leaf. –

ie – if /opt/local was there, and the resource called for creation of /opt/local/lib/mylib/mysubdir the permissions/owner/group will start with the newly created directories, starting with “lib”, and including “mylib” & “mysubdir”.

Absent could just be a dangerous “rm -rf” at the leaf level specified. ie – using the above example if one wants to get rid of mylib/mysubdir, one would say

path    => "/opt/local/lib/mylib",

ensure => absent, recurse => true,

As said above, any complex permissions/owner/group should be handled separately.

If people want the file resource to chown, chmod or chgrp with a “-R/r”, that could also be done, but as a separate bug/feature. (I would say something similar to absent… start at the point specified in “path” or “name”)

exec’s ok… but from the http://docs.puppetlabs.com/guides/types/exec.html page:

There is a strong tendency to use exec to do whatever work Puppet can’t already do; while

this is obviously acceptable (and unavoidable) in the short term, it is highly recommended to migrate work from exec to native Puppet types as quickly as possible. If you find that you are doing a lot of work with exec, please at least notify us at Reductive Labs what you are doing, and hopefully we can work with you to get a native resource type for the work you are doing.

Sounds like this is a feature that a lot of people want.

Just need to agree on what the feature does :)

Regards, -Roy

#26 Updated by Peter Meier about 6 years ago

A stipulation can be made that when a directory tree is made, the “tree” only can have one set of permissions/owner/group – as specified in the resource. This would start with the directory that is missing, down to the leaf. –

and this is exactly where this resource becomes unpredictable if on one host /foo/bar/a exists but on one one host only /foo/bar and your tree should create /foo/bar/a/b/c one one host puppet would manage (owner, etc.) folder a on the other not, where it might be different and for example not chmod 0755 which would break access to your leave for the daemon, which would make the puppet-run incomplete and not able to set everything correctly. Or: what if you have 2 trees: /foo/bar/a/b/c and /foo/bar/a/e/f in one tree the owner is set to root in the other it is set to mail. On a vanilla host /foo doesn’t exist. So what is owner /foo? Maybe root maybe mail? depends on which host which tree resource would have been applied earlier.

Furthermore in the second run: how would puppet know from which part of the tree it manages that resource and what should it do if somebody changes in the above example permission of b ? In the first run it would set them correctly and then somebody changes it and what would it do then in the second run?

ie – if /opt/local was there, and the resource called for creation of /opt/local/lib/mylib/mysubdir the permissions/owner/group will start with the newly created directories, starting with “lib”, and including “mylib” & “mysubdir”.

Absent could just be a dangerous “rm -rf” at the leaf level specified. ie – using the above example if one wants to get rid of mylib/mysubdir, one would say

path    => "/opt/local/lib/mylib",

ensure => absent, recurse => true,

and this makes it somehow very inconsistent: you wouldn’t have the same state as if you first manage the tree and then set it to absent, you would have some parts of the tree left, which means that creating and removing a resource wouldn’t do the same.

As said above, any complex permissions/owner/group should be handled separately.

If people want the file resource to chown, chmod or chgrp with a “-R/r”, that could also be done, but as a separate bug/feature. (I would say something similar to absent… start at the point specified in “path” or “name”)

this can imho get very cumbersome. furthermore as puppet has the paradigma to manage a resource only once in a manifest you wouldn’t be able to set the flags with a second file resource.

[…] Sounds like this is a feature that a lot of people want.

Yes, I agree on that. On the other side I can say that within the > 100 modules I wrote I encountered that problem maybe 10 times and half of them I realized that I create anyway a common structure for my systems which I can easily facter out into a seperate module and include that everywhere. And due to the auto-require of paths and the possibility to pass an array to the file resource the overhead is minimal. I know that minimal is somehow ugly, but:

Just need to agree on what the feature does :)

and how it fits into the model that puppet tries to ensure and so far nobody came up with a solution that is consistent and is easy to use.

#27 Updated by Roy Nielsen about 6 years ago

and this is exactly where this resource becomes unpredictable if on one host

/foo/bar/a exists but on one one host only /foo/bar and your tree should create /foo/bar/a/b/c

so – how do you do this now?

one one host puppet would manage (owner, etc.) folder a on the other not, where it might 

Yes, if one wanted/needed a specific set of ownership on a tree, or part of a tree, that could/should be managed separate from the “create”. This would mandate the use of a “before” or “require”…

be different and for example not chmod 0755 which would break access to your 

leave for the daemon, which would make the puppet-run incomplete and not able to set everything correctly. Or: what if you have 2 trees: /foo/bar/a/b/c and /foo/bar/a/e/f in one tree the owner is set to root in the other it is set to mail. On a vanilla host /foo doesn’t exist. So what is owner /foo? Maybe root maybe mail? depends on which host which tree resource would have been applied earlier.

see above.

Furthermore in the second run: how would puppet know from which part of the tree it manages that resource and what should it do if somebody changes in the above example permission of b ? In the first run it would set them correctly and then somebody changes it and what would it do then in the second run?

I see this (#86) as a “create” issue – not an issue of maintaining tree permissions. That should be done as a different resource, with this as a “before”, or be “required”.

I see the “create_dir” option (maybe of ensure => ?) to handle ensuring the directories are created.

but… I’m rather newish to puppet and target managing clients not servers so I may not be seeing this through the same pair of glasses :)

Regards, -Roy

#28 Updated by Roy Nielsen about 6 years ago

Would it be possible to do something like:

file { “/foo/voo/bar/baz” :

ensure => directory, between => {
start => "/foo", end => "/foo/voo/bar", owner => "foo", group => "foo", mode => 755, }
between => {
start => "/foo/voo/bar/baz", end => "/foo/voo/bar/baz", owner => "baz", group => "baz", mode => 775, }

}

or would that require re-writing chunks of puppet?

Regards, -Roy

#29 Updated by eric sorenson over 5 years ago

I feel like the fact that there are edge cases that might not be handled correctly has stymied implementation of the most common case (create directories up to the specified one with the ownership/perms that are specified) and it’s making a lot of things way more difficult than it needs to be. I get asked incredulously at least a couple times a month by other SAs I’m introducing to puppet “You mean there’s no way to ‘mkdir -p’ an arbitrary path?” Any chance we could machete through the brambles and get a caveat-laden feature in?

#30 Updated by Mario Verbelen over 5 years ago

I still hear this question sometimes ‘Hey how do you do “mkdir -P” in puppet?’ …

Maybe we should create simply the option “parents => true/false” and ignore that option when using “ensure => absent”

if it is documented I don’t see any problem off-course there are always arguments in some cases, well don’t use it then! (freedom of choice) but if it is not implemented you don’t have a choice

#31 Updated by Nigel Kersten over 5 years ago

I’ve yet to see a consistent and predictable suggestion for how we accomplish this.

If anyone wants to take a stab at providing that, it would be immensely helpful.

The general principles I believe we should be favoring are: * Explicit is preferable to implicit * Consistency is important * Avoid abusing existing parameters * Avoid a proliferation of extra parameters.

Additionally, can we try to think outside the box a little? Is it possible that we could solve this by being able to instantiate resources with an array of hashes where missing values are undef?

$my_dirs = [ { path => "/a" }, { path => "/a/b" }, { path => "/a/b/c", owner => "root", mode => 0755 } ]
file { $my_dirs: }

#32 Updated by Mario Verbelen over 5 years ago

What about the original way then?

file { [‘/a’, ‘/a/b’, ‘/a/b/c’]: owner => “root”, mode => 755 }

I’m guessing there are to many thoughts about this topic and it will never be solved sins there is a conflict in the possible solutions

I will continue to patch my puppet with “mkdir -p” where the customer wants to @home I just take the time to configure it like [‘/a’, ‘/a/b’, …

:)

#33 Updated by Nigel Kersten over 5 years ago

You mean the way that works now? I’m perfectly happy with that, but the whole feature request here is because people aren’t :)

The reason I suggested the hash method is because it sounds like people want to behave like mkdir -p, where they really don’t care about the permissions settings of parent directories.

I think this is a bad idea, because I believe in being explicit, but I offered the hash method as a way for people to only specify directory properties for some of the directories in the tree.

#34 Updated by Nigel Kersten over 5 years ago

Any other suggestions other than the method that works now?

I’m tempted to close this to reduce some of our immense ticket debt, as I haven’t seen anyone complain about this for a while now.

#35 Updated by Mario Verbelen over 5 years ago

Hi Nigel,

it will not be possible to satisfy everyone here, so I will continue as it is designed right now For me to “case closed”

greets Mario,

#36 Updated by eric sorenson about 5 years ago

“Ruby DSL”

http://www.puppetlabs.com/blog/ruby-dsl/

#37 Updated by Nigel Kersten almost 5 years ago

  • Status changed from Accepted to Closed

#38 Updated by Nigel Kersten almost 5 years ago

  • Status changed from Closed to Rejected

#39 Updated by James Turnbull almost 5 years ago

  • Target version deleted (4)

#40 Updated by eric sorenson over 1 year ago

2014 update for the sake of posterity and teh googles:

Things have improved in Puppet since this bug came out, namely using an array of directories will auto-require parents, so the pattern of :

file {['/top','/top/second','/top/second/deep','/top/second/deep/deepest']: 
  ensure => directory
}

is both expressive and compact. You can shorten this still further with this module: https://forge.puppetlabs.com/AlexCline/dirtree which will take the path and expand it out.

If you know you really just want a mkdir -p, and really don’t care about the edge cases described in the ticket, this module provides a pretty straightforward way to to do it: https://github.com/ghoneycutt/puppet-module-common/blob/master/manifests/mkdir_p.pp

Due to the complications and edge cases described above, the current file type will not be getting an update to support recursive mkdir.

But FWIW, this will be a design requirement for file/directory management in the rewritten agent announced at PuppetConf this week and described at a high level by Luke in this blog post: http://puppetlabs.com/blog/evolving-puppet-for-next-ten-years

Also available in: Atom PDF