Upcoming, , 2 min read
What is coming in Stroom 7.14
S3 storage, a rebuilt processor task system, traces as documents and JSON logging are all landing in the 7.14 betas. Here is what is ready, and what is still experimental.
Stroom 7.14 is in beta. Three betas have shipped so far: beta.1 on 3 September, beta.2 on 11 September and beta.3 on 14 September. None of this is a stable release yet, so treat everything below as a preview of 7.14. Some of it is marked experimental even within the betas, and we say so where it is.
S3 at both ends
- An S3 forwarder for Stroom Proxy (#5546) sends proxy-format zips to S3. On the Stroom side, an SQS consumer picks up the S3 create events and registers each file.
- A read-only S3 volume type,
S3 v1 (Read only), lets Stroom read data from an S3 store it only has read access to. S3 v2 (Experimental)(#5279) is a new volume type using framed ZStandard compression. In the changelog's own words it is "work in progress and not for production use".
Processing that copes with losing a node
A run of changes under #5699 rebuilds how processor tasks are tracked:
- Each task now has a heartbeat, so a live long-running task can be told apart from one owned by a node that has died.
- A cluster-locked Processor Task Reaper replaces the master-only Disown Dead Tasks job, and returns stale tasks to the queue even when there is no master node.
- Task creation backs off linearly for filters with little or no data (#5690).
- Each processor profile gets a task creation budget, so every profile gets work (#5691).
There is also an experimental mode, stroom.processor.claimTasksOnWorker, in which each worker claims its own tasks straight from the database. It is off by default and "not yet proven in production", so leave it off unless you have been asked to trial it.
Traces, logging and stepping
- Traces become a first-class document type (#5771): stored, published and browsed on a shared file system.
- JSON-format application and request logging (#5775), for anyone shipping Stroom's own logs into a structured pipeline.
- A stepping data store (#5662) caches stepping data, making pipeline stepping faster.
And AI
7.14-beta.1 also adds askAi() to StroomQL and XSLT, and AI summaries to reports. That has a post of its own.
Beyond 7.14
A Stroom cloud deployment is coming. When there is something concrete to say about it, it will be here first.
Want this running on your own data? The engineers who built it can help.