The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Feature #4885

Simplify the syntax for specifying file paths in modules using a '~'

Added by Nigel Kersten about 4 years ago. Updated 10 months ago.

Status:Needs DecisionStart date:09/30/2010
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:3.x
Affected Puppet version: Branch:
Keywords:

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1771


Description

There is too much redundant info in puppet file source specifications in modules.

file { "ugly":
  source => "puppet:///modules/foo/bar",
}

Additionally, we’re inconsistent with our template() and file() functions.

template() takes either a path relative to the modulepath, dereferencing to “templates” sub-directories of the module or an absolute path. file() takes an absolute path only.

I suggest (after a flash of inspiration from Patrick on the mailing list) we use the ‘~’ syntax in a standard unix-y way to refer to module locations as follows.

Each class mentioned below corresponds to a module.

File source:

Reference: $modulepath/modules/foo/files/bar

class foo {
  file { "booyah":
    source => "~/bar",
  }
}

class notfoo {
  file { "booyah2":
    source => "~foo/bar",
  }
}

Template function: Reference: $modulepath/modules/foo/templates/bar.erb

class foo {
  file { "booyah":
    content => template("~/bar.erb"),
  }
}

class notfoo {
  file { "booyah2":
    content => template("~foo/bar.erb"),
  }
}

File function: Reference: $modulepath/modules/foo/files/bar

class foo {
  file { "booyah":
    content => file("~/bar"),
  }
}

class notfoo {
  file { "booyah2":
    content => file("~foo/bar"),
  }
}

I believe we should be able to make this change without breaking any of the existing behavior. I do believe we should aim to deprecate the old template function, but that can be a separate discussion.

I haven’t actually spent time ensuring this is possible. This is just what I would like to see.

Mailing list thread references so we don’t re-hash the same questions over again unless needed:

http://groups.google.com/group/puppet-dev/browse_frm/thread/688050b8f0668ff2

http://groups.google.com/group/puppet-users/browse_frm/thread/688050b8f0668ff2

(was cross-posted, which never works out well. )


Related issues

Related to Puppet - Feature #5158: File resources: Make source/content parameters and the fi... Needs Decision 10/30/2010

History

#1 Updated by Nigel Kersten about 4 years ago

  • Assignee set to Luke Kanies

Luke, let me know if you disapprove of this proposed change.

#2 Updated by Luke Kanies about 4 years ago

  • Status changed from Needs Decision to Accepted
  • Assignee deleted (Luke Kanies)

Entirely brilliant.

#3 Updated by Nigel Kersten almost 4 years ago

  • Target version set to 2.7.x

#4 Updated by Nick Fagerlund almost 4 years ago

(It occurs to me that I should mention our offline convo somewhere related to the issue.)

I like the design; I’m a little weirded out by the tilde, because tildes are things owned by users, not things owned by modules. I can’t for the life of me come up with anything better, though… and it DOES have logic to it, though it requires a bit of a perceptual shift.

#5 Updated by Anonymous over 3 years ago

I think I like this. Let’s see if I understand.

class notfoo {
  file { "booyah2":
    source => "~foo/bar",
  }
}

In this context ~ means: prepend the module path, prepend the class name, after the module name insert the resource type. Put another way, “foo’s ‘bar’ file.”

This is very magical, and different from the way ~ works on Unix. My concern is the expectation mismatch between these:

  • “Unix: tidle adds stuff at the beginning”
  • “Puppet: tilde adds stuff wherever it likes”

#6 Updated by Patrick Mohr over 3 years ago

This wasn’t how my original idea worked when proposed. In unix, “~” means your home directory. “~root” means root’s home.

In puppet the same thing would apply, except that your home “~/” expands to “puppet:///modules/$current_module/”. And “~module_name/” expands to “puppet:///modules/module_name/”

Nothing affects the end of the path. It just allows relative paths.

#7 Updated by Nigel Kersten over 3 years ago

The context Randall was missing was that we already do the “magical” dereferencing in the template function like:

template("foo/bar.erb")

is

$modulepath/foo/templates/bar.erb

The semantics are a bit more complicated than you’ve outlined Patrick, as the tilde-expansion becomes context-sensitive.

e.g.

template("~foo/bar")

is

$modulepath/foo/templates/bar

and

file("~foo/bar")

is

$modulepath/foo/files/bar

#8 Updated by Patrick Mohr over 3 years ago

You have a point. I meant just in the case of “source” but I didn’t make that clear.

Still, only the beginning is changed right?

#9 Updated by Nigel Kersten over 3 years ago

Yeah we can look at it that way. It’s still all about substitution at the start of the string.

#10 Updated by Nigel Kersten over 3 years ago

  • Target version changed from 2.7.x to 3.x

#11 Updated by Digant Kasundra over 3 years ago

I love the ~ syntax as described. Requiring puppet:///modules is way to long and makes pretty syntaxing a pain in the butt (and delivers no new information about what is happening).

#12 Updated by Anonymous about 3 years ago

My one note on this is that ~ alone technically poses some redundancy with file-system conventions. If I write ~pvande/lib/something.rb expecting a reference to a file in the user pvande’s home directory, my expectation will always be broken. In that sense, the puppet “scheme” still seems important.

My take:

  • ~module/foo.txt — looks like a filesystem path, not relative to a module.
  • puppet:~module/foo.txt — Not bad, but doesn’t quite look “right”.
  • puppet://~module/foo.txt — This looks like most URLs, in a good way, but conflates the server portion with the module name.
  • puppet:///~module/foo.txt — This is probably most “correct” as a URI, but the triple slash is unfamiliar to many, and doesn’t add much information in the default case.

#13 Updated by Nigel Kersten about 3 years ago

Pieter van de Bruggen wrote:

My one note on this is that ~ alone technically poses some redundancy with file-system conventions. If I write ~pvande/lib/something.rb expecting a reference to a file in the user pvande’s home directory, my expectation will always be broken. In that sense, the puppet “scheme” still seems important.

I desire a better solution than having to include the puppet scheme in every reference, particularly given the vast vast majority of such references use it.

You can’t currently use the ~username for a local filesystem reference in file locations fwiw

# puppet apply -v /tmp/test.pp 
Parameter source failed: Cannot use relative URLs '~nbk/.zshrc' at /tmp/test.pp:3
  • puppet:~module/foo.txt — Not bad, but doesn’t quite look “right”.
  • puppet://~module/foo.txt — This looks like most URLs, in a good way, but conflates the server portion with the module name.
  • puppet:///~module/foo.txt — This is probably most “correct” as a URI, but the triple slash is unfamiliar to many, and doesn’t add much information in the default case.

I’m really not fond of any of these new suggestions. “puppet” is frustratingly annoying to type on every reference.

#14 Updated by Josh Cooper about 3 years ago

I like Brice’s suggestion source => "module://foo.conf" as it’s compact, simplifies the migration path, and would not affect our ability to support http(s) as the file source (see #5783)

#15 Updated by Nigel Kersten almost 3 years ago

Josh Cooper wrote:

I like Brice’s suggestion source => "module://foo.conf" as it’s compact, simplifies the migration path, and would not affect our ability to support http(s) as the file source (see #5783)

Is that literally “module” ? or the name of the module?

If it’s the former, it doesn’t save anything over “puppet” and how do you distinguish between a file in a subdirectory of the ‘local’ module and a file from a different module?

If it’s the latter, then we’ve got namespace collisions between module names and future supported protocols.

I really think we should be optimizing for the most common cases.

  1. files from within the current module
  2. files from another module
  3. then care about future protocols we haven’t yet implemented…

The vast vast majority of the time you’re dealing with case (1).

#16 Updated by Brice Figureau almost 3 years ago

Nigel Kersten wrote:

Josh Cooper wrote:

I like Brice’s suggestion source => "module://foo.conf" as it’s compact, simplifies the migration path, and would not affect our ability to support http(s) as the file source (see #5783)

Is that literally “module” ? or the name of the module?

I meant literally module. And this would mean get the file in the current module.

If it’s the former, it doesn’t save anything over “puppet” and how do you distinguish between a file in a subdirectory of the ‘local’ module and a file from a different module?

If it’s the latter, then we’ve got namespace collisions between module names and future supported protocols.

I really think we should be optimizing for the most common cases.

  1. files from within the current module

Yes, that’s the more frequent case isn’t it?

  1. files from another module

This would be a very bad practice.

  1. then care about future protocols we haven’t yet implemented…

The vast vast majority of the time you’re dealing with case (1).

Yes and that’s why I thought my proposal would be a good fit.

Either you use “module://foo.txt” or the “puppet://” old syntax. That would cover your 1. and 2.

The problem with the ‘~’ is that we’re adding a complexity to the language, and that’s really not natural. Worst it’s not even an operator, since it is embedded in a string. We’re already have some difficulties to make sure users understand the spaceship operator, this one will be really difficult to explain to new users.

#17 Updated by Nigel Kersten almost 3 years ago

I’m sympathetic to your points Brice, but I’m not convinced “module://” is easier to explain to new users than “~”.

#18 Updated by Nigel Kersten almost 3 years ago

Also I think it’s important to remember that the current mapping of:

puppet:///modules/foo/bar

to

modulepath/foo/files/bar

already contains the equivalent of operators embedded in a string.

#19 Updated by Justin Stoller almost 3 years ago

Forgive my ignorance if this wouldn’t be a backwards compatible change or the parser doesn’t have this information available to it, but could we simply drop the puppet://server/modules/$module_name/ and consider relative paths relative to the module’s file directory?

Then you could prepend puppet://server/ if you needed to specify a different file server? Perhaps modules/$module_name/ if you needed a file outside of the current moduleandpuppet://server/modules/$module_name/` if you need files from a different module on a different file server? …but I don’t think either of those are recommended.

It seems like relative paths from the current module’s file directory would make the best practice the easiest thing to do.

ps. If context switching is important between templates and files you could make them relative to the module’s root directory (source => 'files/my-file.rb', or content => template('templates/my-template.erb') but it doesn’t seem like too hard of distinction for new users to make.

#20 Updated by Nigel Kersten almost 3 years ago

It’s important to keep #5158 in mind.

We’d like to make this consistent between content and source, particularly when thinking about templates vs files.

This is the main reason I didn’t suggest using relative paths to indicate this difference, as we currently have:

template("$modulename/template.erb")

using a relative path to indicate that the template should come from within a module. Note that providing the modulename is mandatory in that case.

#21 Updated by Gary Larizza almost 3 years ago

We had this discussion in class today, and the suggestion was put up as this:

file { '/etc/foo':
  ensure  => file,
  source  => 'module/filename',
}

#Templates
file { '/etc/foo':
  ensure => file,
  content => template('module/filename')
}

This simplifies the syntax AND makes templates and files function similarly. Now, of course, it assumes that we’re filling in “puppet:///modules/‘ beforehand, but Templates ALREADY have a similar 'magic’ about them. We would still support the full syntax, and also have this syntax for modules. Passing a source that doesn’t begin with a ‘/’ would be assumed to be coming from the module.

Thoughts? Comments? Snide Remarks?

Thoughts?

#22 Updated by Anonymous almost 3 years ago

  • Status changed from Accepted to Needs Decision

Gary Larizza wrote:

We had this discussion in class today, and the suggestion was put up as this:

[…]

This simplifies the syntax AND makes templates and files function similarly. Now, of course, it assumes that we’re filling in “puppet:///modules/‘ beforehand, but Templates ALREADY have a similar 'magic’ about them. We would still support the full syntax, and also have this syntax for modules. Passing a source that doesn’t begin with a ‘/’ would be assumed to be coming from the module.

Thoughts? Comments? Snide Remarks?

Of the options, this seems like the best “magic” choice. Allowing anything that looks like a URI to behave like a URI, and a relative path to be treated as “inside the module” seems sensible.

The only other thing I have of value in this thread is to mention the existence of a “protocol independent relative URI”:

://hostname/resource  # <- relative to the current protocol

I would entertain the idea that puppet:${module}/${path} should do the expansion magic, and puppet://${hostname}/${module}/${path} is the full version. Given that, and protocol relative URI expansion, that could abbreviate to :${module}/${path} for power users. (eg: we don’t heavily advertise that ability, but we do allow it, and if you want to abbreviate your manifests, you can go right ahead.)

Finally, I want to remind y'all that we are proposing to support file { "example": source => "http{,s}://example.com/resource.data" }, and that some of the community are threatening to implement that at the moment. I don’t have a strict timeline, but “just Puppet as the protocol” will probably not last forever.

Occasional ideas of experimenting with, eg, the source being a filebucket hash, or using some peer-to-peer protocol as the source are also floated, and I absolutely do not want to lock those options out with the syntax here.

#23 Updated by eric sorenson almost 3 years ago

I beg of you, please don’t let scope creep paralyse you from fixing the original problem Nigel described:

content=>file('/must/be/full/path/to/file')

vs

content=>template('mymodule/template.erb')

This is brutal and leads to people making EVERYTHING a template, just so we can use the automatic this-module-file-slash-template path interpolation magic.

#24 Updated by Nigel Kersten almost 3 years ago

Daniel Pittman wrote:

Of the options, this seems like the best “magic” choice. Allowing anything that looks like a URI to behave like a URI, and a relative path to be treated as “inside the module” seems sensible.

I disagree. :)

I think this is an inconsistent mental model for what’s going on.

source => '/absolute/path/to/local/file/on/agent',
source => 'relative/path/to/file/on/master',

That feels quite horrible to me, and significantly worse than the use of “~” to represent the module on the server.

As Eric has hinted at, the problem with forcing everyone towards the file() and template() functions means that you’re shipping the whole content of the file every single time to the agent, simply because it ends up in the catalog.

In some cases this may result in less load, simply because you’re doing fewer (but much larger) connections vs a gazillion file_metadata requests, but we can do better than that.

#25 Updated by Ryan Coleman almost 3 years ago

Nigel Kersten wrote:

I think this is an inconsistent mental model for what’s going on.

Agreed. All the new Puppet users I train have enough trouble understanding that an absolute path is on the agent and puppet:///… is from the master.

In some cases this may result in less load, simply because you’re doing fewer (but much larger) connections vs a gazillion file_metadata requests, but we can do better than that.

Speaking of those students, not a single one hasn’t been confused by the difference between template() and puppet:/// though I can’t seem to convince any of them to share their thoughts in this ticket — we need to do better soon, even if it’s not perfect. /two cents

#26 Updated by Anonymous almost 3 years ago

Nigel Kersten wrote:

Daniel Pittman wrote:

That feels quite horrible to me, and significantly worse than the use of “~” to represent the module on the server.

…because there is no ambiguity in /foo being evaluated on the client and ~/foo being evaluated on the master, right?

As Eric has hinted at, the problem with forcing everyone towards the file() and template() functions means that you’re shipping the whole content of the file every single time to the agent, simply because it ends up in the catalog.

That is true with the current implementation of the file function. It is absolutely not a requirement of the function, or the syntax, or anything else. It would be absolutely possible right now to implement file so that it stored the content in a file bucket and gave a constant hash reference, for example.

Doing something fancier would require more smarts in the tool, or more pain for the user – because you couldn’t just return a URI from file and have it work, since we have non-polymorphic behaviour in setting where file content comes from – but can absolutely be done.

In some cases this may result in less load, simply because you’re doing fewer (but much larger) connections vs a gazillion file_metadata requests, but we can do better than that.

That wouldn’t be a good choice. However, limiting our design based on some happenstance of current implementation seems as poor a choice to me.

#27 Updated by Anonymous almost 3 years ago

After discussion, it seems like some of the ambiguity comes from being able to specify a file source on the master, and on the client, in the same type.

Would it make sense to consider splitting out a localfile type or an extra parameter for “file on the client” source? That might make it unambiguous if this was “on the master” or “on the client” for both the user, and the programmer.

#28 Updated by Nigel Kersten almost 3 years ago

Daniel Pittman wrote:

Nigel Kersten wrote:

Daniel Pittman wrote:

That feels quite horrible to me, and significantly worse than the use of “~” to represent the module on the server.

…because there is no ambiguity in /foo being evaluated on the client and ~/foo being evaluated on the master, right?

Not none :) I totally take your point, but you can’t argue against “~” and offer the “/foo” vs “foo” as an option.

Maybe we’re overthinking this:

source => modules://foo/bar

That cuts down on a lot of characters, keeps the protocol identification, removes the somewhat annoying ‘modules’ part, but still means you need to type the local $module_name into every reference.

I just think we need a radically simpler syntax for saying “this file in the current module” as that’s the most common case.

If people think this original proposal is fatally flawed, I’m happy to reject it and simply outline the requirements I’d like to see fulfilled.

#29 Updated by Nick Fagerlund almost 3 years ago

Well, we could do what the puppet:// scheme did for servers:

module://my_module/foo/bar == module:///foo/bar

#30 Updated by eric sorenson almost 3 years ago

Ignore ‘source’ urls for the moment, and concentrate on the syntactic split between the file and template content functions, both in how to reference the files and the behaviour if multiples are used… If that’s a separate ticket, let’s talk about it there. But IMO the ‘source’ URLs are the least bad of these things.

#31 Updated by Gary Larizza almost 3 years ago

From a ‘training’ standpoint, the syntax is a MUCH bigger hurdle than having to use the name of the module every time you write out something like:

source => modules://foo/bar

If you’re having a problem with writing out the module, you’re more inclined to fix it yourself with $calling_module or something similar. If you’re having a problem with the syntax, I feel you’re more likely to throw your hands up in the air. I think simplifying it is a bigger win – but just my opinion.

I also agree with Nick about the modules:/// being similar to puppet:///

#32 Updated by Gary Larizza 10 months ago

Redmine Issue #4885 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1771

Also available in: Atom PDF