A Dashboard from a Store

You have data sitting in a store and you want it on a dashboard. This guide covers the two stores most dashboards read: a Lucene index, which holds events, and a Plan B store, which holds state. The dashboard side is the same for both: a Query component reads the store, a Table shows and aggregates what it returns, and a Visualisation draws the table. What differs is what has to be in place first, and what each store can answer.

If you have not built a dashboard before, Your first dashboard walks through the components once. For how the parts fit together, see Dashboards, and for the text form of the same search, see StroomQL and data sources.

Before you start

Whichever store you use:

• You need Use permission on the index or Plan B document. The dashboard's data source picker only offers documents you can use.
• The store must already hold data. A dashboard reads what is there; it does not trigger processing. Getting data in is covered by Index and search it for an index, and by Plan B for a state store.

From a Lucene index

What must be in place

Open the Lucene Index document and check three things.

The fields you want to search on are Indexed. On the Fields tab, each field has an Indexed and a Stored setting. Indexed means a query can match on it. Stored means its value is kept in the index and can be shown without going back to the original event. They are independent, and the difference decides most of what follows.

The Main Time Field Name is a real, indexed date field. On the Settings tab, Main Time Field Name (default EventTime) names the field the dashboard's time range is applied to. Every search with a time range adds a condition on that field.

There is a Default Extraction Pipeline, if you need fields that are not stored. Also on the Settings tab. For each matching event, Stroom uses this pipeline to go back to the original event and read the fields the index did not keep. See Stored and extracted fields.

Query it

On a new dashboard, which opens in design mode with a Query and a Table already linked:

1. Open the Query component's Settings from its tab menu, and set Data Source to the index.
2. Use Add Term to add conditions, and Add Operator to combine them. The field list offers only the fields that are indexed.
3. Set the dashboard's time range to cover the event times you expect. It is applied to the index's main time field, not to when the data arrived.

The same search written in StroomQL, in a Query document:

from "Authentication Events"
where Outcome = "Failure"
select EventTime, UserId, SourceHost

The table

Click Add Column on the table. Under Data Source are the index's fields; under Counts are ready-made count() and countGroups() columns; Custom gives you an empty column to write an expression in.

Whether a column fills depends on its field:

• Stored fields come straight from the index.
• Fields that are not stored come from extraction. In the table's Settings, Extract Values must be on. If you do not choose an Extraction Pipeline there, the index's default extraction pipeline is used. With Extract Values off, only stored fields have values.

To aggregate, use a column's menu: Group sets a grouping level, and Expression sets the expression, for example floorDay(${EventTime}) to bucket by day, or count() in another column. A visualisation draws the table's rows as they are, so shape the table first.

From a Plan B store

What must be in place

• A Plan B document, with the State Type you need. Its name is the map name your pipeline writes to, and may only contain lower-case letters, digits and underscores.
• A pipeline that writes to it, ending in a PlanBFilter, with a processor filter that has run over your data. See Plan B.
• The Plan B Merge Processor job running on the Plan B storage nodes. Pipelines write fragments; the merge job folds them into the store that queries read. Until it has run, the store is empty to a query.

What a Plan B data source returns

A Plan B document is a data source in its own right: choose it as the Query component's Data Source, or name it in from. There is no extraction step. Every column comes from the store, and which columns exist depends on the state type:

For a Metric store, each value column exists only if the Value Schema stores it (Store Latest Value, Store Min Value and so on); Average needs both count and sum. Trace data is not a Plan B state type in 7.14: it is held in a separate Traces document, which is not a dashboard data source (see Plan B).

The dashboard's query pane offers the key and time fields as terms. Value and ValueType are returned but not offered as terms; to narrow on a value, filter the table column or use filter in StroomQL.

Query it

Set the Query component's Data Source to the Plan B document and add terms as for an index. In StroomQL, using the user_locations Temporal State store from the Plan B example:

from user_locations
where Key = "jbloggs"
select Key, EffectiveTime, Value

This returns every location recorded for that user, one row per change.

What it can and cannot answer

A query returns stored entries, not answers at a point in time. For a Temporal State store, each row is one key at one effective time. Searching it lists the history: when each value took effect. It does not work out which value applied at a given moment, and the time range only selects entries whose effective time falls inside it. A user whose location was last set a month ago has no row in a query over the last week, though that location still applies.

For a point-in-time value, use getState(). It is an expression function, so it goes in a table column or a StroomQL eval, against any data source:

• getState('user_locations', ${UserId}, ${EventTime}) gives the value that applied at each event's time. On a table over an index of events, this labels every event with where the user was when it happened.
• getState('user_locations', ${Key}), with no time, gives the value that applies now. On a table over the Plan B store itself, grouped by Key, this shows the current value per key.

getState() works on State, Temporal State, Range State, Temporal Range State and Session stores. For a Range State, the key must be a number. For a Session store it returns true or false: whether the key was in a session at that time. If the map name does not match a Plan B document, the column is empty rather than an error. Each lookup is done per row, so on large results it is slower than enriching events as they are processed. See Plan B lookups.

A query reads the whole store. Conditions are tested against every entry, so a query on one key in a large store takes as long as a query on all of them. Keep Plan B dashboards for stores of a size you are content to scan.

On a cluster, a query may read a snapshot. A query runs on the node you are using. If that node is not a Plan B storage node, it reads a local snapshot of the store, which is refreshed periodically, so very recent merges may not show yet.

Add a visualisation

1. Shape the table first: group and aggregate it so that each row is one point to draw, for example one row per day with a count.
2. Click Add Component (in design mode) and add a Visualisation.
3. In its Settings, set Table to your table and Visualisation to the chart you want. Visualisations come from content packs such as stroom-visualisations; see Visualisations.
4. Map the table's columns onto the visualisation's settings, such as x and y for a line chart.

To show only the chart, keep the table and hide it; a visualisation always draws from a table. See Dashboards.

In StroomQL, show draws the result directly:

from "Authentication Events"
where Outcome = "Failure"
eval day = floorDay(EventTime)
eval failures = count()
group by day
select day, failures
show LineChart(x = day, y = failures)

The same works on a Plan B store, for example charting how often locations change:

from user_locations
eval day = floorDay(EffectiveTime)
eval changes = count()
group by day
select day, changes
show LineChart(x = day, y = changes)

from "Authentication Events"
eval n = count()
select n

Related pages

• Dashboards: components, tables and visualisations.
• StroomQL and data sources: the query language and the data sources it reads.
• Lucene Index: fields, storage and extraction pipelines.
• Plan B: state types, ingestion and lookups.
• Expression functions: the functions available in table columns and eval.