List of things and places that interact with the StoreConfig engine directly in the product, to work to build an API around this:

• :async_storeconfigs, :thin_storeconfigs, in defaults.rb, which enable specific features of the tool. No change expected.
• :storeconfigs in defaults.rb, which enables the core feature. Triggered to be true by the previous set.
• lib/puppet/face/node/clean.rb, which interacts directly with the ActiveRecord code to delete / unexport content.
• lib/puppet/indirector/catalog/compiler.rb, which configures the backend during compilation if :storeconfigs are enabled.
• same file, update_node_check, which I can’t trace an invocation of…
• lib/puppet/parser/collector.rb, which does collection from various sources; needs to go through the API rather than direct to Rails.
• lib/puppet/parser/ast/collexp.rb, which does the syntax transformation to “rails” from the input language.
• lib/puppet/application/queue.rb, which hard-codes the target when using async storeconfigs.

Defining the API for StoreConfig backends:

1. add a configuration option for the storeconfig backend / terminus.
2. update existing code to select a storeconfigs terminus, not active_record, when activating storeconfigs.
   • push down failure reporting to the underlying back-end implementation.
3. write the storeconfigs terminus for each class so that it references the configuration option, then picks the right terminus to wrap.
4. ensure that the Rails-specific bits are pushed down into the target, rather than the AST (for collection)
5. push the query into the API; perhaps “find” from an indirection, to return the data.
   • the resource indirection is the right place for this; it should gain a StoreConfig terminus set.

Then we have an API for the final target:

• save and find over catalog, facts, and node indirections that works with the storeconfigs terminus.
• find over the collected resources through a separate indirection.

Alternatively, we could define a sound API over the interface we want – something that supports set and get over catalog, facts, nodes, and collecting resources. I would reach for a face, but the design we received makes it very difficult to do that cleanly; because they are forcibly instances of a class, rather than a class proper, we would need to define a custom API for the back-end, plus replicate all the work the indirector does around configuration, discovery, and loading of terminus instances.

Advice? I don’t like splitting this code into a dozen places, but we can’t pack it sanely into the indirection API without substantial contortion. (eg: a protocol for doing three or four things through a single method, using some dependent dispatch key.)

I lean toward defining a sound API through subclassing, then writing the huge pile of terminus classes to support that, and living with the need to select a backend implementation by hand. (eg: define Puppet::StoreConfigs and Puppet::StoreConfigs::ActiveRecord, then have the parent manage the API so that you get the concrete child instance when you ask for something implementing the API.)

The same branch is updated with the latest version of this code (a forced push, because history has been rewritten during development). This is approaching completion, in that it now has all the code in what looks like the right places, and looking to be doing the right thing.

Still broken, though, are some of the collector tests, and by implication the final round of changes are not actually completed yet.

Once they are fixed up, and finished, the collection expressions are still way too intimate with the current ActiveRecord schema; the subsequent change will be to abstract that, and ensure that we have portable references to the data we use, not to the database schema it happens to be implemented with.

Just pushed another round of updates to GitHub; now we have working everything, and pass an abstract expression rather than the raw SQL compiled out of the collection expression through to the final destination.

Additional changes merged via pull request #98 in commit:74cffa087aa35d31b331033870b026d41e40cb07.

Nick Lewis wrote:

Additional changes merged via pull request #98 in commit:74cffa087aa35d31b331033870b026d41e40cb07.

Specifically, the previous changes missed one limitation of the current StoreConfigs code: previously, the Puppet parser would fail while evaluating the AST when you tried to collect exported resources on more than one condition. This is a fairly arbitrary limit that is really only drawn from the current StoreConfigs implementation. So, the changeset moved the limitation out of the parser, and implemented it instead in the terminus.

That will fail a bit later in the process, but it will still fail cleanly during compilation if the StoreConfig feature is used in ways that are not supported, and the net effect will be pretty much identical.