The Puppet Labs Issue Tracker has Moved: https://tickets.puppetlabs.com

Feature #14255

Discovery method should be pluggable

Added by R.I. Pienaar over 2 years ago. Updated about 2 years ago.

Status:ClosedStart date:05/01/2012
Priority:NormalDue date:
Assignee:R.I. Pienaar% Done:

0%

Category:Core
Target version:2.1.x
Keywords: Affected mCollective version:
Branch:ripienaar/feature/master/14255

We've Moved!

Ticket tracking is now hosted in JIRA: https://tickets.puppetlabs.com

This issue is currently not available for export. If you are experiencing the issue described below, please file a new ticket in JIRA. Once a new ticket has been created, please add a link to it that points back to this Redmine ticket.


Description

Since version 2.0.0 the messaging and core system support arbitrary destinations not coupled with filters and broadcasting.

We should build on this and make the actual discovery system pluggable, today we have:

  • Discovery against the network using broadcasts and filters
  • Discovery against JSON in on STDIN in the rpc application only

I am proposing that we extend this to an actual client side plugin so that any mcollective client can do:

  • Discovery against JSON on STDIN
  • Querying PuppetDB for nodes matching whatever criteria
  • Text files full of lists of hosts
  • Querying systems like hiera or ldap
  • Arbitrary SQL queries
  • Rest calls to systems like CMDBs and dasbhoards like foreman

These should be simple plugins that just returns an array of node identities, they should have names and any rpc client should be able to select one on the CLI using some flag

The default should also be settable in the config file of the client so that someone might decide to never use the network discovery – unless they override it on the CLI

There will need to be a way to parse arguments to these for example if you have a plugin that discovers against a text file you might want to provide the file name on the CLI as well

History

#1 Updated by R.I. Pienaar over 2 years ago

There is a hacktacular proof of concept in ripienaar/poc/master/discover_plugins that lets you set default_discovery_mode=flatfile in the client.cfg and then it will just never do network discovery at all anymore.

Copying the commit here that details the status of this POC

    A pretty hacky proof of concept that makes discovery pluggable
    it includes 2 plugins one thats the traditional broadcast and one
    that discovers against a flatfile (hardcoded path now)
    
    In the case of the flatfile plugin it shows how plugins can elect to not
    support certain types of filters - here only supporting identity filters
    - and then doing discovery against that.
    
    RPC clients will force to directed mode for anything but the
    mc_broadcast plugin
    
    You'd elect to use a different discovery mode by setting
    default_discovery_mode = flatfile in your client config file
    
    There are a few issues here:
    
     - there's some cases where we dont now force directed mode when we
       should like in Client#discovered_rec
     - the RPC::Client#discover still impliments a whole lot of its own
       discovery magic we should make sure that this work can replace all
       that stuff there
     - there is no way to choose a discovery mode on the cli yet
     - there is no way yet to pass arguments into the discovery plugin like
       which text file to use for example
    
    But it shows conceptually this isnt too hard

#2 Updated by R.I. Pienaar over 2 years ago

Updated the POC you can still supply a default discovery plugin in the config file but now also on the CLI of any application and can also pass in arguments that makes it to the discovery plugins.

I added a mongodb discovery plugin that supports Agent, Facts and Identity filters – still to do compound filters to see how easy that might be.

Major things left to sort out:

These plugins need a DDL, the DDL for them should support some kind of indication of capabilities, it seems common that some plugins will just not be able to do all kinds of discovery. A flat file plugin for example cannot do facts, classes or agent discovery only identity, others might not support regex so the DDL should know this and validate the discovery request prior to dispatching it to the discovery plugin.

#3 Updated by R.I. Pienaar about 2 years ago

  • Branch set to ripienaar/feature/master/14255

Complete branch @ ripienaar/feature/master/14255 will do final testing in a real world scenario before merging

#4 Updated by R.I. Pienaar about 2 years ago

  • Status changed from Accepted to Closed

43 files changed, 1096 insertions(+), 417 deletions(–) http://git.io/34pkNg

Also available in: Atom PDF