Feature #6528

Test data: robust, real-looking, repeatable

Added by Randall Hansen about 1 year ago. Updated 2 months ago.

Status:Accepted Start date:03/01/2011
Priority:Normal Due date:
Assignee:- % Done:

0%
Category:-
Target version:-
Keywords: Affected URL:
Branch: Affected Dashboard version:
Votes: 0

Description

We need to be able to test Dashboard with test data that we control. We cannot use customer data for this due to the potential of a leak. Our community needs to be able to see and work with this data as well.

This test data should be able to reflect sensible changes over time. That is, a node should have some number of resources which drift in and out of compliance. We need to be able to report on a node’s compliance over time. Which resource properties change to affect this compliance drift are undefined.

We need a script which generates a configurable number of reports that can be imported into Dashboard with rake reports:import. These reports should:

  • look like real data (e.g., all nodes and resources can’t be called “test”)
  • exercise all user-facing features of Dashboard (e.g., we need nodes and resources in all possible states)
  • be repeatable (e.g., tested)
  • date-sensitive (i.e., since we make decisions in Dashboard based on the last date a node reported, hard-coded dates will not work).

We have a very good start at this, but it’s not yet complete. At minimum (and I’m not convinced this list is complete) we need to be able to generate:

  • nodes (see “Node status proportions” below)
    • compliant/successful
    • failed
    • unresponsive
    • pending
  • resources of status
    • successful
    • pending
    • failed
  • types of resources (see “Resource types proportions” below)
    • file
    • user
    • package
    • service
    • schedule (Puppet built-in schedules are sufficient; no resources need be associated with them)
  • events for these resources
  • inspect reports

Node status proportions: roughly 80% successful nodes (configurable), with the rest distributed roughly evenly.

Resource types proportions: file, user, package, service should each appear in roughly equal proportion. This proportion should be configurable via environment flags, similar to “NUM_EVENTS” and friends. The use case is that I need to be able to generate reports which are 100% files, with no users, packages, or services; but this is an edge case and should not be default behavior.

Also:

  • every failure event should generate a log entry
  • accurate metrics for:
    • changes
    • events
    • resources
    • do NOT need metrics for “Time”

History

#1
Updated by Daniel Pittman about 1 year ago

Doesn’t seem to have audit reports included; we probably need (the capacity) to include those.

#2
Updated by Nigel Kersten about 1 year ago

Daniel Pittman wrote:

Doesn’t seem to have audit reports included; we probably need (the capacity) to include those.

Those are the inspect reports aren’t they? Or did that get added after?

#3
Updated by Pieter van de Bruggen 12 months ago

Since the scope of this ticket has been changed, this note should serve to document my understanding and plan w.r.t. the current work:

  • The test data should appear “real”.
    • The user should be able to choose to generate these reports as inspect reports.
    • This includes schedule resources on every report and some number of resources of arbitrary types (file, user, package, and service for now).
    • There should be events and log entries for these resources.
    • Accurate metrics for changes, resources, and events should be included.
  • The test data should change in realistic ways from report to report.
    • Reports with no changes should be vastly preferred.
    • Reports with changes should contain some small percent of resources that were changed, pending, or failed.

Longer term goals include: * Configurable resource types. * Configurable node statuses.

Please fill any gaps in my understanding.

#4
Updated by Pieter van de Bruggen 12 months ago

This ticket has been sidelined for now, as it does not actually address our current priorities.

#5
Updated by Michael Stahnke 10 months ago

  • Target version set to 140

#6
Updated by Daniel Pittman 2 months ago

  • Target version deleted (140)

Also available in: Atom PDF