Cloud Parts

Apache Avro

A compact, row-based binary format with a JSON schema, widely used for Kafka messages and record files.

What it is
File formats (binary serialization)
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
5 October 2026
Website Source code

What it is

Avro writes one record after another in a compact binary encoding. The bytes carry no field names or type tags, so they’re small, but a reader needs the writer’s schema to decode them. Schemas are written in JSON: a record with named, typed fields, optional defaults, and unions such as ["null", "string"] for values that may be missing.

An Avro container file puts the schema in its header, followed by blocks of records separated by sync markers, so large files can be split and read in parallel. When a reader’s schema differs from the writer’s, Avro’s schema resolution matches fields by name and fills in defaults, which lets producers and consumers upgrade at different times. On Kafka, each message usually carries only a schema ID; a schema registry such as Confluent Schema Registry stores the full schema and checks that each new version stays compatible.

Why teams pick it

  • Small and fast to write, which suits appending one event at a time
  • Clear schema evolution: add a field with a default and old and new readers keep working
  • Self-describing files: BigQuery reads the schema straight from an Avro file and recommends Avro for loads
  • Libraries for Java, Python, C, C#, Rust, JavaScript and more

Watch out for

  • Row-based, so analytical scans read every field; convert to Parquet for querying
  • A single message can’t be decoded without its writer’s schema, so you need a registry or another way to find it
  • Protobuf and JSON Schema compete for the same job; schema registries support all three, so teams have to pick one

Alternatives

What all binary serialization have in common →
CBOROpen sourceConcise Binary Object Representation: a binary format with JSON’s data model plus byte strings and tags, designed for tiny decoders and readable without a schema.

How it fits with other technologies

An Avro schema (order.avsc); the nullable field with a default can be added without breaking old readers · json

{
  "type": "record",
  "name": "Order",
  "namespace": "com.acme.shop",
  "fields": [
    { "name": "order_id", "type": "long" },
    { "name": "amount", "type": "double" },
    { "name": "ts", "type": { "type": "long", "logicalType": "timestamp-millis" } },
    { "name": "coupon", "type": ["null", "string"], "default": null }
  ]
}

See it in a real stack

No architecture uses Apache Avro yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2009–2026

In 2009, Thrift (from Facebook) and Protocol Buffers (from Google) could already encode records compactly across languages, but both expected you to write an IDL file, run a compiler and load generated code for every data type. Doug Cutting, Hadoop’s creator, argued that this was awkward in dynamic environments: a script in Pig, Hive, Python or Perl should be able to open its inputs, look at their types and write its outputs without compiling anything.

On 2 April 2009 Cutting proposed Avro as a new Hadoop subproject, with Sharad Agarwal as the other initial committer. His stated ambition was for Avro “to replace both Hadoop’s RPC and to be used for most Hadoop data files, e.g., by Pig, Hive, etc.” Avro 1.0 followed in July 2009, and in April 2010 Avro became a top-level Apache project.

The RPC half never took over: Hadoop added a Protocol Buffers RPC engine in 2011. The data half did. Avro became a common format for record files on Hadoop, and Confluent’s Schema Registry, first released in February 2015, stored Avro schemas for Kafka topics, which is a large part of why Avro is still the usual encoding for Kafka messages that need a managed schema.

Why it’s built this way

Schemas written in JSON, not a custom IDL
Cutting asked whether anyone would “write parsers for Thrift’s IDL in every language”. Every language already had a JSON parser, which kept new Avro implementations small.
No field numbers or type tags in the data
With the schema in hand, the bytes don’t need to say what each value is, so records are smaller and developers never assign field IDs. Fields are matched by name when versions differ.
Code generation optional
The Java library offered generic, reflection-based and generated representations, so a Pig or Python job could read any file without compiling classes, while Java services could still generate them.
The schema travels with the data
Untagged bytes are unreadable without the writer’s schema, so container files store it once in the header, and schema resolution maps old and new versions onto each other.

Timeline

  1. 2009

    Doug Cutting proposes Avro as a Hadoop subproject in April; 1.0 ships in July

  2. 2010

    Becomes a top-level Apache project in April

  3. 2011

    Hadoop adds a Protocol Buffers RPC engine; Avro’s role settles on data

  4. 2015

    Confluent Schema Registry 1.0 ships in February, storing Avro schemas for Kafka

  5. 2024

    Avro 1.12.0 released in August

  6. 2026

    Avro 1.12.2 hardens readers against malformed data

Sources7 · facts checked 5 October 2026

Improve this page on GitHub