Binary serialization

Avro, Protocol Buffers and Cap’n Proto all start from a schema and encode records in binary, but differently. Avro writes no field names or type tags, so its bytes are small and a reader needs the writer’s schema. Protocol Buffers tags each field with the number it has in the .proto file. Cap’n Proto lays a message out like an in-memory struct, so reading it needs no parse step.

Schemas let producers and consumers upgrade at different times. Avro’s schema resolution matches fields by name and fills in defaults, Protocol Buffers keeps old field numbers stable, and on Kafka a schema registry stores each version and checks that new ones stay compatible.

These are row formats: they write whole records one after another, which suits messages and appends. For scanning one column across millions of rows, data usually lands in columnar files.