Nodes and Clusters

A Stroom cluster is several nodes running the same software, pointed at one database. There is no separate controller and no per-node build. What makes one node different from another is its configuration file and which jobs are enabled on it.

What each node has of its own

• The reference data store. Loaded on demand, per node. A lookup can succeed on one node and fail on another.
• Plan B shards, written locally and merged. See Plan B.
• Caches, including the property cache, which is why a change made on one node reaches the others only when they next reload.
• Index writers, and whichever index shards that node owns.
• Its own temporary and home directories.

This is why so many jobs have to run on every node: the work is local, and a job switched off on one node stops that node's share of it while the cluster carries on looking healthy. See Scheduled jobs.

What they share

The database, and only the database. It holds stream metadata, processing state, users and permissions, job definitions, configuration overrides, and the cluster locks that stop two nodes doing the same cluster-wide job at once.

Volumes may be local to a node or on shared storage, and that is a deployment choice rather than something Stroom decides. The database always knows which volume a stream is on; whether another node can reach it is up to how you mounted it.

The master node

One node is the master. In the default configuration it holds the processor task queue in memory and hands work to the others, which is why the Nodes screen marks it and why losing it briefly pauses task distribution rather than processing already under way.

From Stroom 7.14, in either mode, each node's Processor Task Heartbeat job renews the tasks it is working on, and the cluster-locked Processor Task Reaper job returns any task whose heartbeat has not been renewed within stroom.processor.taskLeaseTimeout (10 minutes by default) so another node can process it, even when there is no master. These replace the master-only Disown Dead Tasks job, and taskLeaseTimeout replaces disownDeadTasksAfter.

Its role is narrowing. With stroom.processor.claimTasksOnWorker turned on, each worker finds and claims its own tasks straight from the database and the master leaves the processing path entirely. That setting is experimental, off by default, and must be identical on every node: the two modes use different task states and cannot see each other's work in flight, so changing it means stopping the whole cluster, changing it everywhere, and starting again. A mixed cluster leaves tasks unprocessed.

What has to be true everywhere

• One database. Two nodes pointed at different databases both appear to work and never see each other's data.
• Clocks in step. Event times, task heartbeats and lock expiry all depend on time. A node a few minutes out writes data that searches for "the last hour" will miss, and takes tasks other nodes consider expired.
• The same Stroom version. Nodes share a schema, so a cluster part-way through an upgrade is a cluster running two versions against one database.
• The same receipt check mode, if proxies are involved, along the whole chain. See Data receipt.

Adding a node

A new node writes its own job rows when it first starts, taking each job's default, which is why one added later comes up running everything the others do. What it does not inherit is anything you changed by hand on the existing nodes, so a job you disabled everywhere is enabled again on the newcomer.

Nodes, jobs and properties covers managing them, and Requirements covers sizing before you build one.