Pipeline Data Destinations

Once a pipeline has finished transforming data, the results must be stored or sent to another system. This is handled by Appenders and specialised Filters.

The Stroom data store (stream store)

The most common destination for pipeline output is back into Stroom's own data store. This allows for multi-stage processing. For example, Normalisation -> Enrichment -> Indexing.

• StreamAppender: Writes the output as a new stream. You must specify:
  • Feed: Which feed the new data belongs to.
  • Stream type: Typically Events (for processed data) or Context.
• RollingStreamAppender: Used for high-volume output. It pools data and only creates a new stream in the database when a certain size or time threshold is met. This reduces database overhead.

Search indexes

Data often needs to be made searchable. This is achieved by sending XML events into an indexing element.

• IndexingFilter: Sends data to a standard Stroom Lucene index. The XSLT must produce XML that matches the index's field structure.
• ElasticIndexingFilter: Sends data to an external Elasticsearch cluster. Like the Lucene filter, it requires specific XML field mappings.

Plan B (state store)

Plan B is Stroom's high-performance state management system (see Plan B / LMDB data store). It is used for storing "state" that can be looked up by other pipelines. For example, "What was the last known IP for this user?".

• PlanBFilter: Instead of writing to a stream or index, this filter writes data into an LMDB (Lightning Memory-Mapped Database) shard.
• How it works:
  1. The pipeline processes data (often Reference or Raw Reference streams).
  2. The PlanBFilter at the end of the pipe populates an LMDB map (State, Temporal State, Session, etc.).
  3. Once the pipeline completes, the shard is uploaded to storage nodes for central merging.
• Usage: Other pipelines can then perform ultra-fast lookups against these Plan B maps using the stroom:lookup() XSLT function or the getState() Dashboard function.

External systems

Pipelines can also "push" data out of Stroom entirely:

• HTTPAppender: Posts the processed data to a remote URL via HTTP or HTTPS. This is useful for forwarding alerts to a SOC or a third-party ticketing system.
• FileAppender: Writes the output directly to the local file system of the Stroom node.
• HDFSFileAppender: Writes data to a Hadoop Distributed File System.

Summary of destinations