The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Bug #1565

puppet parser not order independent

Added by Ryan McBride over 5 years ago. Updated 3 months ago.

Status:Needs More InformationStart date:09/08/2008
Priority:NormalDue date:
Assignee:eric sorenson% Done:

0%

Category:language
Target version:3.x
Affected Puppet version:0.22.1 Branch:
Keywords:language

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1417


Description

The puppet parser has number of issues regarding order-of evaluation:

  • It does not always ensure that included classes are evaluated before qualified variables referring to them are evaluated. (LanguageTutorial)
  • It may evaluate classes before the scope in which they are defined is evaluated.
  • the defined() function is “unfortunately dependent on the parse order of the configuration when testing whether a resource is defined.” (FunctionReference)

As far as I understand it, the intent is that puppet is a declarative language, in which order of evaluation should not matter, and it would be nice to have these issues cleared up. Conversely, if puppet continues to be “somewhat order dependant”, stronger mechanisms to provide control over order-of-execution or at least make order-of-execution more predictable are necessary. I believe the former is preferable.

The following example demonstrates the first two of these problems (tested on both 0.24.5 and HEAD):

######################################
# define some classes

class apache {
        apache_setup { "config" : }
}

define apache_setup (
        $chroot = true,
        $ssl = false
) {
        # sanity checking, platform-specific defaults
        $_chroot = $chroot
        $_ssl = $ssl

        class apache_config {
                $ssl = $_ssl
                $chroot = $_chroot
                # more variables here
        }
        include apache_config
        # actuall apache setup goes here
}

class trac {
        case $apache_config::chroot {
        true: {
                notice("chroot = true")
                # do something
        }
        default: {
                # do something else
                notice("chroot = ${apache_config::chroot}")
        }
        }
}


######################################
# Now use what we've set up

class apache_ssl inherits apache {
        # override the defaults
        Apache_setup["config"] {
                ssl => true,
                chroot => false
        }
}

include apache_ssl
#include apache_config
include trac

If this example is executed as-is, we recieve the error, it has clearly been included in apache_setup. (issue 1 above)

Class apache_config has not been evaluated so its variables cannot be referenced at /tmp/foo:26 on node 000AE43B1909

If we uncomment the ‘include apache_config’ directive at the bottom of the example to ensure that puppet sees the class as being included, puppet evaluates apache_config before the apache class and apache_setup define; the scope in which apache_config is defined is not evaluated, and the variables it depends on are evaluated as empty strings. (issue 2 above)

notice: Scope(Class[trac]): chroot =

History

#1 Updated by Ryan McBride over 5 years ago

While I believe that these bugs are independent of what I’m actually trying to do in the example above, it may be helpful if I try to explain the intent:

I’d like to have a relatively generic and reliable mechanism to look into the configuration of one class/module from another. This is particularly useful with modules that have tight dependancy relationships; for example the many web applications which depend on apache. In some cases this may be used to drive configuration in the dependant module, other cases it may be used for sanity checking (eg: throw an error if Apache is configured to run chrooted and you’re installing a module which will not work in a chrooted environment).

In the example above, I’m trying to build the generically-named apache_config module, which can be referenced from other modules regardless of whether apache has been inherited on that specific node. The apache_setup define was an attempt to force puppet to evaluate apache_config in the correct order; this also provides the necessary inheritance support, and provides a nice way to override the module defaults in my site configuration: the standard “Apache_setup { ssl => true }”.

As mentioned on irc/#puppet, it is possible to provide the same behavior with node-scoped variables, something like:

class apache {
    $ssl = $apache_ssl {
       false => false,
       true => true,
       default = > false
    }
    $chroot = $apache_chroot {
       false => false,
       true => true,
       default = > true
    }
    # actual apache setup goes here
}

node "foo" {
    $apache_chroot = false
    $apache_ssl = true
    include apache
    include trac
}

However, this prevents me from using inheritance on the apache module (for example, to include additional files). I’d also prefer to be able to keep all of a module’s configuration in it’s own namespace, to keep things clean. Having the configuration via a define also allows me to force the user to provide a specific configuration value by not providing a default value in the define’s prototype.

#2 Updated by James Turnbull over 5 years ago

  • Category set to language
  • Status changed from Unreviewed to Needs Decision
  • Assignee set to Luke Kanies
  • Target version set to 4

#3 Updated by Ryan McBride over 5 years ago

DavidS pointed out on irc that the tagged() function is also order-dependent, as are variable definitions within a particular scope.

#4 Updated by Luke Kanies almost 4 years ago

  • Assignee changed from Luke Kanies to Markus Roberts
  • Target version changed from 4 to 2.7.x
  • Affected Puppet version set to 0.22.1

I think this will get solved by Markus’s ‘futures’ work.

#5 Updated by James Turnbull about 3 years ago

  • Assignee changed from Markus Roberts to Nigel Kersten

#6 Updated by Nigel Kersten over 2 years ago

  • Status changed from Needs Decision to Accepted
  • Assignee deleted (Nigel Kersten)
  • Target version changed from 2.7.x to 3.x

#7 Updated by James Shubin over 1 year ago

Dear puppet devs,

Would it be acceptable to inquire as to when this bug will be fixed? I have been working on some elaborate puppet code which is blocked by this. Unfortunately re-ordering doesn’t fix this permanently because there are circular dependencies.

I get problems such as:

warning: Scope(Bks::Host[iron1]): Could not look up qualified variable ‘bks::foo::bar’; class bks::foo has not been evaluated

If there is some other workaround, that would be excellent too.

Many thanks in advance, James james@shubin.ca

#8 Updated by Timur Batyrshin over 1 year ago

@James I’m not a puppet dev and I’ve suffered much from the same problem but it seemed to go away when I changed the way I define classes. Try to inherit bks::foo::bar from bks::foo and you’ll get the latter evaluated before a former.

Of course this will affect the way you write your modules but I’ve found that my modules have become more conscise when I switched to this way. And yes, this does not solve the issue of accessing class variables from definitions rather than classes and I have to invent some terrific workarounds each time I need to use a definition.

#9 Updated by James Shubin over 1 year ago

@Timur Thank you for your prompt reply. bar is actually a variable name used inside the foo class.

I’ll be continuing my development on this project with the hopes that this bug gets fixed, since my design relies on this. :) I’ll be posting all of my code in the coming months, so hopefully this will provide another solid use case of this functionality too.

Puppet devs please let me know if you’d like me to test this or otherwise as ready.

Thanks, James

#10 Updated by eric sorenson over 1 year ago

  • Status changed from Accepted to Needs Decision
  • Assignee set to Andrew Parker

I came across this ticket searching for related ordering issues and figured I’d update it while I’m here.

IMO once you leave straight resource declaration syntax in the DSL, you also leave the realm of its claim to declarative programming.

Let me narrow scope to the issue presented by James Shubin in note-7 and ask Andy the question:

warning: Scope(Bks::Host[iron1]): Could not look up qualified variable ‘bks::foo::bar’; class bks::foo has not been evaluated

Assuming the bks::foo is included somewhere, doesn’t referencing that variable now trigger its evaluation? Shouldn’t it?

#11 Updated by eric sorenson over 1 year ago

  • Subject changed from puppet parser not order independant to puppet parser not order independent

#12 Updated by Andrew Parker over 1 year ago

  • Status changed from Needs Decision to Needs More Information
  • Assignee changed from Andrew Parker to eric sorenson
  • Keywords set to language

eric sorenson wrote:

I came across this ticket searching for related ordering issues and figured I’d update it while I’m here.

IMO once you leave straight resource declaration syntax in the DSL, you also leave the realm of its claim to declarative programming.

Yes. I think one of the most confusing things is that there are 2 levels (at least) in puppet: declarative resources and procedural control flow. The language itself is mainly a procedural control flow that decides what resources to declare. Those declarations create model of the world (the catalog) that is interpreted as declarative program on the agent (much like SQL) where it essentially tries to come up with a way of making the real world match the model of the world.

Let me narrow scope to the issue presented by James Shubin in note-7 and ask Andy the question:

[…]

Assuming the bks::foo is included somewhere, doesn’t referencing that variable now trigger its evaluation? Shouldn’t it?

Assuming bks::foo has been included, then it should have already been evaluated. The act of including something should evaluate the body of the class. I don’t understand why it isn’t in this case. There has been talk about having the evaluation of variables be lazy, but that has never moved forward as far as I know, and it also presents a lot of different problems (circular dependencies, side effects during evaluation).

I think this is a real bug in the language as it is currently speced out.

Eric, I think this is related to a whole slew of other bugs around behavior of include and class {...}. A quick search of redmine showed quite a lot, but I haven’t taken the time to discover which are directly related to this. Some of them might even be a non-obvious duplication of this problem.

#13 Updated by Paul Pham 3 months ago

Redmine Issue #1565 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1417

Also available in: Atom PDF