The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com
Discovery method should be pluggable
|Assignee:||R.I. Pienaar||% Done:|
|Keywords:||Affected mCollective version:|
Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com
This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.
Since version 2.0.0 the messaging and core system support arbitrary destinations not coupled with filters and broadcasting.
We should build on this and make the actual discovery system pluggable, today we have:
- Discovery against the network using broadcasts and filters
- Discovery against JSON in on STDIN in the rpc application only
I am proposing that we extend this to an actual client side plugin so that any mcollective client can do:
- Discovery against JSON on STDIN
- Querying PuppetDB for nodes matching whatever criteria
- Text files full of lists of hosts
- Querying systems like hiera or ldap
- Arbitrary SQL queries
- Rest calls to systems like CMDBs and dasbhoards like foreman
These should be simple plugins that just returns an array of node identities, they should have names and any rpc client should be able to select one on the CLI using some flag
The default should also be settable in the config file of the client so that someone might decide to never use the network discovery – unless they override it on the CLI
There will need to be a way to parse arguments to these for example if you have a plugin that discovers against a text file you might want to provide the file name on the CLI as well
#1 Updated by R.I. Pienaar over 2 years ago
There is a hacktacular proof of concept in ripienaar/poc/master/discover_plugins that lets you set default_discovery_mode=flatfile in the client.cfg and then it will just never do network discovery at all anymore.
Copying the commit here that details the status of this POC
A pretty hacky proof of concept that makes discovery pluggable it includes 2 plugins one thats the traditional broadcast and one that discovers against a flatfile (hardcoded path now) In the case of the flatfile plugin it shows how plugins can elect to not support certain types of filters - here only supporting identity filters - and then doing discovery against that. RPC clients will force to directed mode for anything but the mc_broadcast plugin You'd elect to use a different discovery mode by setting default_discovery_mode = flatfile in your client config file There are a few issues here: - there's some cases where we dont now force directed mode when we should like in Client#discovered_rec - the RPC::Client#discover still impliments a whole lot of its own discovery magic we should make sure that this work can replace all that stuff there - there is no way to choose a discovery mode on the cli yet - there is no way yet to pass arguments into the discovery plugin like which text file to use for example But it shows conceptually this isnt too hard
#2 Updated by R.I. Pienaar over 2 years ago
Updated the POC you can still supply a default discovery plugin in the config file but now also on the CLI of any application and can also pass in arguments that makes it to the discovery plugins.
I added a mongodb discovery plugin that supports Agent, Facts and Identity filters – still to do compound filters to see how easy that might be.
Major things left to sort out:
These plugins need a DDL, the DDL for them should support some kind of indication of capabilities, it seems common that some plugins will just not be able to do all kinds of discovery. A flat file plugin for example cannot do facts, classes or agent discovery only identity, others might not support regex so the DDL should know this and validate the discovery request prior to dispatching it to the discovery plugin.