Skip to content

Solr

Indexing into an external Solr cluster, and the two jobs that keep it tidy.

Stroom can index into an external Apache Solr cluster as an alternative to its built-in Lucene indexes. Like Elasticsearch, it moves the search workload onto infrastructure you scale separately from processing.

Choose it when Solr is already what your organisation runs. For a new deployment with no existing preference, built-in Lucene indexing is the path with the least to set up and the most of Stroom's own tooling behind it.

The pieces

  • A Solr Index document represents an index on a Solr cluster and holds the connection to it.
  • A pipeline ending in SolrIndexingFilter writes to it.

The pipeline is shaped exactly as for any other index: a parser reading the translated events, an XSLTFilter that converts them to records:2 fields, then the indexing filter.

The two jobs

Solr indexing depends on two scheduled jobs, and both are enabled by default:

  • Solr Index Optimise compacts the indexes, on a daily schedule.
  • Solr Index Retention applies each index's deletion query, removing documents that have aged out.

Both take a cluster lock, so leave them enabled on every node and let the lock pick one runner. See Scheduled jobs.

If you do not use Solr, both do nothing and can be left alone.

How long: not immediate

Documents are not searchable the moment they are indexed: Solr makes them visible on its own commit and soft-commit settings, which are configured on the Solr side rather than in Stroom.

Optimisation runs daily and on a large index takes a while and is I/O heavy, so it is worth knowing when it is scheduled before investigating a nightly slowdown.

Check it worked

Process one stream, then search for a value you know is in it through a Stroom dashboard built on the Solr Index document.

The check that matters most

A value you chose in advance comes back through a Stroom dashboard.

Going through Stroom rather than querying Solr directly is what proves the Solr Index document, its field definitions and the searching user's permissions are all right, which is where the remaining problems are once documents exist.

This does not prove it worked

Documents in Solr are not proof the dashboard will show them. The Solr Index document and Stroom permissions sit between the two.

An empty result is not proof nothing was indexed. It is also what an uncommitted batch, a field type mismatch and a missing permission all look like.

Fails silently

Disabling Solr Index Retention stops deletion without stopping indexing.

The index keeps growing, searches keep returning data that should have aged out, and nothing reports it. The symptom appears much later as an index that is larger and slower than expected, which is easy to read as a capacity problem.

If it isn't working

In order. Each step rules out a class of cause, so don't skip ahead.

  1. Can Stroom reach the Solr cluster? Connection problems appear in the stream's errors rather than as a pipeline failure.
  2. Did the pipeline produce records:2 output? Step it. A translation producing nothing indexes nothing and reports no error.
  3. Has Solr committed? Recently indexed documents are briefly invisible, and the commit settings are Solr's rather than Stroom's.
  4. Do the field types match the Solr schema? A value Solr rejects is rejected per document, so most of a stream can index while some of it silently does not.
  5. Are the two Solr jobs enabled? See Scheduled jobs.
  6. Can the searching user read the Solr Index document? No permission returns nothing rather than an error.

All documentation