The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

This issue tracker is now in read-only archive mode and automatic ticket export has been disabled. Redmine users will need to create a new JIRA account to file tickets using https://tickets.puppetlabs.com. See the following page for information on filing tickets with JIRA:

Bug #8417

Exported resources is exporting to the database with --noop flag

Added by Andrew Thompson almost 5 years ago. Updated over 2 years ago.

Status:Needs More InformationStart date:07/14/2011
Priority:NormalDue date:
Assignee:Gabriel Filion% Done:

0%

Category:exported resources
Target version:-
Affected Puppet version:3.2.3 Branch:
Keywords:ntbf customer

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This ticket is now tracked at: https://tickets.puppetlabs.com/browse/PUP-1079


Description

When I run puppet with the noop flag resources are being exported to the db.

I would expect it to simulate the transaction with the db…

This is causing things like nagios host resources being creating before the plugins get there, so I get invalid alerts.

-Andrew

History

#1 Updated by Gabriel Filion almost 5 years ago

I experienced that bug with puppet 2.6.2, and also with 0.25.4 (in both cases both puppet master and client are the same version)

#2 Updated by Stefan Schulte over 4 years ago

What about the following suggestion:

  1. normal puppet run: resource gets exported and the collecting host will apply the resource
  2. noop puppet run: resource gets exported but with the noop metaparameter set. On the collecting node, you will now see the resource but it does not get applied
  3. number 1 and 2 can be changed when setting the noop metaparameter on the exported resource, e.g. setting the metaparameter to noop => false will lead to the current behaviour: The collecting node will always apply the exported resource

#3 Updated by James Turnbull over 4 years ago

  • Status changed from Unreviewed to Needs Decision
  • Assignee set to Nigel Kersten

#4 Updated by Nigel Kersten almost 4 years ago

  • Assignee changed from Nigel Kersten to eric sorenson

#5 Updated by eric sorenson almost 4 years ago

  • Status changed from Needs Decision to Needs More Information
  • Assignee changed from eric sorenson to Deepak Giridharagopal
  • Keywords set to ntbf

Deepak can you verify whether this still happens under puppetdb? From a design standpoint it seems wrong (noop should indeed not update the database) but I’m not clear on whether it’s still valid.

#6 Updated by Deepak Giridharagopal almost 4 years ago

This still happens under puppetdb. I actually have always considered this a valid thing, FWIW. I assumed that —noop mode meant that the catalog is still accurate, but it’s application on the local system is side-effect free. However, the presence of storeconfigs means that in order to compile your catalogs, you need to be persisting them centrally even in the noop case. Otherwise your catalogs may not be accurate or have the latest data.

Also, we’ve used this “feature” as a way to help users migrate to PuppetDB. You switch to PuppetDB, then trigger a noop run across your infrastructure (or just wait for machines to check in). Then PuppetDB will be fully populated automatically with the latest data. Zero users have complained about this behavior to me, and in fact many of them seem to expect it (like they’ve asked me if that will still work).

#7 Updated by Deepak Giridharagopal almost 4 years ago

  • Assignee changed from Deepak Giridharagopal to eric sorenson

#8 Updated by eric sorenson almost 4 years ago

  • Status changed from Needs More Information to Closed

OK, intended behavior with a history and valid use cases — closing as not to be fixed.

#9 Updated by Gabriel Filion almost 4 years ago

  • Status changed from Closed to Re-opened

Deepak: humm I don’t agree with the idea that “it doesn’t matter because the catalog is consistent”. it may be true for the local system running with —noop, but it’s not true for system that collect resources possibly exported by this node.

in some cases, the nagios config may break apart and require manual intervention to unbreak (since the nagios types don’t have a working “purge” functionality). this imho is an example of a pretty bad side effect.

and the use case presented is rather a case of using an unintentional behaviour to achieve a one-time shot change (which is a clever thing to do for a one-time change). expecting to be able to achieve the same thing in the long run is not really a good idea since the unintentional behaviour could disappear in the future, especially if it’s not documented anywhere.

I’d still push for fixing the behaviour presented in this issue since it’s causing cases where systems get configured for nodes that were never intentioned to be in the first place.

#10 Updated by Marc Fournier almost 4 years ago

I agree with Gabriel’s point of view ! But Deekpak’s arguments are also valid obviously :–)

Would it be an idea to have a —noexport switch (which would skip all the “@@” resources), as a complement to —noop ? Maybe also a —nocollect one ?

#11 Updated by Nigel Kersten almost 4 years ago

like:

puppet agent --test --noop --no-storeconfigs

?

That should work now I believe.

Or do you still want to collect, but not to export?

#12 Updated by Marc Fournier almost 4 years ago

Nigel Kersten wrote:

like:

puppet agent --test --noop --no-storeconfigs

?

That should work now I believe.

Or do you still want to collect, but not to export?

The use-case where I would have found this useful:

  • install a new node, configure it using puppet, but don’t have it monitored or backuped yet (meaning some resources mustn’t get exported to the central monitoring/backup servers)
  • have people manually do some stuff on it, such as deploy their application, copy data on it, etc
  • once it’s ready, put it in production, meaning among other things: apply the resources it has exported on the central monitoring/backup servers.

Does --no-storeconfigs also prevent @@resources from being stored in storeconfigs_backend ? Or does it only prevent <<| |>> operators from working ? Your comment suggests the latter. My use-case would need the former too, I guess.

But I wouldn’t want to hijack this ticket. Maybe Andrew and Gabriel had something else in mind ?

Thanks, and have a nice WE !

#13 Updated by John Bollinger over 3 years ago

Marc Fournier wrote:

The use-case where I would have found this useful:

  • install a new node, configure it using puppet, but don’t have it monitored or backuped yet (meaning some resources mustn’t get exported to the central monitoring/backup servers)
  • have people manually do some stuff on it, such as deploy their application, copy data on it, etc
  • once it’s ready, put it in production, meaning among other things: apply the resources it has exported on the central monitoring/backup servers.

If I did not want certain resources to be exported for certain nodes, then I would consider that an issue to address on the master, not on the client. It seems strange to me, and a bit dangerous, to allow clients to override the master in the way suggested; moreover, it presents a higher exposure to human error than I would be comfortable with.

The use case seems to be a natural fit for environments, at least conceptually. It could also be addressed via node facts or external data.

In all cases the focus should be on whether the exported resource is declared at all. That is, I would not expect or want a simulation of resource exportation. If a declaration of an exported resource is parsed, then that resource should be recorded in the DB, even if the client is running in —noop mode; in that case it would thereafter be available to be collected by any node.

#14 Updated by Abhay Chrungoo almost 3 years ago

It seems to me that if no resources actually get applied in a —noop run, then no resources should be exported as well. Exported resources being stored in the db is analogous to a resource being applied. This is in principle against the semantics of a —noop run.

To me, exported resources should point to the last successful application. If the catalog was never applied, but the resources exported, it creates problems.

John Bollinger wrote:

Marc Fournier wrote:

The use-case where I would have found this useful:

  • install a new node, configure it using puppet, but don’t have it monitored or backuped yet (meaning some resources mustn’t get exported to the central monitoring/backup servers)
  • have people manually do some stuff on it, such as deploy their application, copy data on it, etc
  • once it’s ready, put it in production, meaning among other things: apply the resources it has exported on the central monitoring/backup servers.

If I did not want certain resources to be exported for certain nodes, then I would consider that an issue to address on the master, not on the client. It seems strange to me, and a bit dangerous, to allow clients to override the master in the way suggested; moreover, it presents a higher exposure to human error than I would be comfortable with.

The use case seems to be a natural fit for environments, at least conceptually. It could also be addressed via node facts or external data.

In all cases the focus should be on whether the exported resource is declared at all. That is, I would not expect or want a simulation of resource exportation. If a declaration of an exported resource is parsed, then that resource should be recorded in the DB, even if the client is running in —noop mode; in that case it would thereafter be available to be collected by any node.

#15 Updated by micah - almost 3 years ago

I agree. This feels like it violates the Rule of Least Surprise. Although I see the interesting upgrade bootstrap case for puppetdb, that sounds like using a misfeature as a hack to do something useful at the expense of doing something non-obvious. If a user choses an option flag that seems to be clear that it doesn’t do anything, but really it does, then that option flag should not be called ‘noop’ and instead be more clearly named. I have no good suggestion as ‘notreallynoop’ seems silly.

#16 Updated by Klavs Klavsen almost 3 years ago

  • Affected Puppet version changed from 2.6.4 to 3.2.3

I definetly agree with micah here. —noop should NOT export resources – as that’s “doing something” – which “no operation” kinda contradicts that should.

#17 Updated by R.I. Pienaar almost 3 years ago

Can see both sides – it’s a compiled catalog, store it. But it should have zero impact.

Resources from a noop compiled catalog shouldnt be collected.

#18 Updated by eric sorenson almost 3 years ago

  • Status changed from Re-opened to Closed

Just had a blinding flash of the obvious here, please correct me if I’ve gotten this wrong.

Storing the exported resources in the database is a byproduct of compiling the catalog, not the agent checking in. If you compile catalogs ahead of time, there’s no way of knowing whether the agent will be checking in with --noop or not. Since Puppet (and the world in general) is moving to decoupled/asynchronous operations this bug seems like it’s not-to-be-fixed.

Seems like it would be much higher value to fix #5239 and make exported resources environment-aware; that way you could really separate/hide nodes that were not supposed to be collected .

#19 Updated by Gabriel Filion almost 3 years ago

  • Status changed from Closed to Re-opened

I’m sorry to re-open this again, but I fail to see how collection by environment can be useful here.

The situation here is that resources get exported with —noop regardless of the environment used and thus can have an impact on systems that would collect them.

In order for #5239 to be relevant, exported resources would need to be automatically flagged with some special value and singled out when collecting…

or just not exported like one would expect out of a no-operation flag.

#20 Updated by eric sorenson almost 3 years ago

  • Status changed from Re-opened to Needs More Information
  • Assignee changed from eric sorenson to Gabriel Filion

Gabriel, the point is the resources get exported completely independently of the client connecting, because the export happens on the server, at the time the catalog is compiled, whereas the noop/enforce switch is a function of the agent. Doing a puppet master --compile node.example.com will cause the resources to be written to the database without any client connection.

Bug #5239 is relevant because environment-aware export and collect would give you control over the resource visibility, as John Bollinger suggests in note 13 — which seems to be the crux of the thing: it’s not whether the resource is exported from the no-op host, it’s whether it get collected and used when you don’t want it to. Does that sound right?

#21 Updated by Charlie Sharpsteen over 2 years ago

  • Keywords changed from ntbf to ntbf customer

#23 Updated by Jason Antman over 2 years ago

Redmine Issue #8417 has been migrated to JIRA:

https://tickets.puppetlabs.com/browse/PUP-1079

Also available in: Atom PDF