PravegaArchived

Pravega stores streams: durable, append-only, unbounded sequences of events, grouped into scopes. Each stream is split into segments, and each event carries a routing key that decides its segment, so events with the same key are read in the order they were written. Unlike a Kafka topic, the number of segments isn’t fixed: under a data- or event-rate scaling policy a hot segment splits in two and cold neighbours merge, and readers in a reader group are rebalanced across the new segments. Writers can batch events in transactions that become visible only on commit, and readers persist their positions, so applications such as Flink jobs can process each event exactly once. A state synchronizer lets distributed processes share state through a stream with optimistic, conditional updates.

Two kinds of server do the work. A controller, the control plane, creates, scales and deletes streams. The segment store, the data plane, writes incoming events first to Tier 1 storage, built on Apache BookKeeper, for low-latency durability, and later moves them to Tier 2 long-term storage such as HDFS, a file system or an object store, from which old data can still be read through the same stream API. The project was written in Java, licensed under Apache 2.0 and released 0.13.0, its last version, in September 2023.