Open lakehouse on Iceberg
Kafka streams events into Spark, which writes Iceberg tables on S3 that Trino queries for dashboards.
Click any box to open its page. Open it in the playground to rewire it, change the traffic or take parts down.
This is the common shape when many teams and many engines need the same data. Events flow through Kafka, a Spark job turns them into rows, and Iceberg makes a pile of Parquet files on S3 behave like database tables, with snapshots and safe concurrent writes. Because Iceberg is an open table format, the streaming job, the SQL engine behind dashboards and a data scientist’s notebook can all read and write the same tables without copying them.
How it flows
- 1
Services publish events
Each order service appends events to a Kafka topic and moves on. Kafka keeps them for days, so the downstream job can fall behind, restart or be replaced without losing anything.
ServicesApache Kafka - 2
Spark reads in micro-batches
A Spark Structured Streaming job reads a batch of new events every few seconds, tracking its own offset in the topic, and cleans and reshapes them into rows.
- 3
Iceberg commits a snapshot
Spark writes the rows as new Parquet files on S3, then asks the catalog to swap the table’s pointer to a new metadata file that lists them. That swap is atomic: readers see all of a batch or none of it.
- 4
Trino answers dashboard queries
Trino loads the table’s current metadata, prunes files using partition and column statistics, and reads only the matching Parquet from S3. Spark keeps writing while it reads; snapshot isolation keeps the two from tripping over each other.
Why this shape works
- Storage and compute are separate: S3 holds the data once, and each engine scales on its own
- No lock-in to one engine; Spark, Trino, Flink, DuckDB, Snowflake and others read Iceberg
- Snapshots give time travel, rollback and safe concurrent writers
- Kafka decouples producers from the pipeline, so either side can change or fail independently
Where it strains
- Streaming writes create many small files; schedule compaction and snapshot expiry or queries slow down
- You run a catalog (REST, Glue, Nessie or similar), and it must be highly available
- Several moving parts (Kafka, Spark, Trino, the catalog) means real operational work for a small team
- Freshness is minutes, not milliseconds; for sub-second dashboards a real-time database fits better
Common swaps
Don’t want to run Trino? Athena queries the same Iceberg tables serverlessly, billed per terabyte scanned.
On AWS without running Kafka, Kinesis Data Streams can feed the job, or write to Iceberg tables directly.
If you’re on Databricks, Delta Lake plays Iceberg’s role, and UniForm can expose the same tables to Iceberg readers.
Built from
Apache KafkaOpen sourceEvent streamsThe open-source distributed log that most event streaming is built on, or built to be compatible with.
Apache SparkOpen sourceQuery enginesThe general-purpose distributed engine for large batch jobs, streaming and machine learning, in Python, SQL, Scala, Java or R.
Apache IcebergOpen sourceTable formatsThe most widely supported open table format: snapshots of data files, tracked by a tree of metadata and swapped in atomically through a catalog.
Amazon S3AWSObject storageAWS’s object storage, and the de facto API most object stores copy.
TrinoOpen sourceQuery enginesA distributed SQL engine that queries data where it lives, from Iceberg tables on S3 to a live PostgreSQL database, in one query.