Cloud Parts

Protocol Buffers

Google’s schema-first binary encoding: messages defined in .proto files, fields identified by number, code generated for each language.

What it is
Remote procedure calls,File formats
Type
Open source
Made by
Google LLC
License
BSD 3-Clause (Apache 2.0 until version 2.0.2 in September 2008)
Facts checked
6 October 2026
Website Source code

What it is

You describe messages in a .proto file, where every field has a name, a type and a number. The compiler, protoc, generates classes for C++, Java, Python, Go and other languages, with accessors and methods that turn a message into bytes and back. On the wire a message is a list of records. Each starts with a tag that packs the field number and a wire type into one varint, (field_number << 3) | wire_type, followed by the value. Varints use 7 bits per byte, so small numbers take one byte: field 1 set to 150 encodes as 08 96 01. Strings, nested messages and packed lists carry a length prefix, and unset fields aren’t written at all. Field names never appear, so a reader needs the .proto to know what field 2 means, but the wire type tells it how many bytes to skip. That is how old code ignores fields added later.

The skip rule is the compatibility contract. Adding a field with a new number is safe; removing one is safe if its number is reserved and never reused; changing a field’s number or moving it into an existing oneof breaks readers. The language has three dialects: proto2 (open-sourced in 2008), proto3 (stable in 2016) and Editions (from protoc 27.0 in 2024), which replace both with per-feature settings and leave the bytes unchanged. Protobuf is free under a BSD licence, and Google develops it for its own needs, accepting selected outside pull requests. Beyond RPC, it is gRPC’s default message format, the binary encoding of OpenTelemetry’s OTLP, the body of Prometheus remote-write requests and one of three formats Confluent Schema Registry supports for Kafka.

Why teams pick it

  • Compact: field numbers and varints instead of names and text, and unset fields cost nothing
  • Written-down evolution rules for binary and JSON: add fields with new numbers, reserve deleted ones, and old and new code keep reading each other’s messages
  • One .proto generates code for C++, C#, Java, Kotlin, Objective-C, PHP, Python and Ruby in protoc, Dart and Go through Google’s plugins, and more languages through third-party projects
  • A defined JSON mapping (ProtoJSON) for debugging and for HTTP APIs that also need JSON
  • Used well beyond RPC: OTLP telemetry, Prometheus remote write, Kafka topics with a schema registry, and storage inside Google

Watch out for

  • Not self-describing: without the .proto you see only field numbers and raw values, so Kafka topics need a schema registry and stored data needs its schema kept somewhere
  • Whole messages are parsed into memory before use; the docs suggest another format above a few megabytes, and encoded messages must stay under 2 GiB
  • Encoding isn’t canonical: the same message can produce different bytes, so you can’t compare or hash encoded messages to test equality
  • Row-at-a-time with a parse step: no random access or column scans, so analytics wants Parquet, and Cap’n Proto skips the parse entirely
  • Google steers it to its own needs, and proto2, proto3 and Editions coexist during a long transition

Alternatives

What all remote procedure calls have in common →
Apache ThriftOpen sourceFacebook’s 2007 RPC framework: define types and services in one IDL file, generate clients and servers for over 20 languages, and pick the wire protocol and transport separately.Cap’n ProtoOpen sourceA binary format laid out like an in-memory struct, so reading needs no parse step, plus an RPC system built on object references and promise pipelining.gRPCOpen sourceGoogle’s open RPC framework: describe a service in a `.proto` file, generate typed clients and servers, and call them over HTTP/2 with streaming, deadlines and standard status codes.

How it fits with other technologies

Define a message, then look at its bytes with protoc · bash

cat > order.proto <<'EOF'
edition = "2023";
package shop;

message Order {
  int64 id = 1;
  string customer = 2;
  repeated int32 item_ids = 3;  // packed by default since edition 2023
  reserved 4;                   // was `coupon`: never reuse the number
}
EOF

# Encode a text-format message; field numbers, not names, go on the wire
echo 'id: 150 customer: "ada" item_ids: [1, 2, 3]' | protoc --encode=shop.Order order.proto | xxd
# 00000000: 0896 0112 0361 6461 1a03 0102 03   .....ada.....

# Decode without the schema: only numbers and raw values survive
echo 'id: 150 customer: "ada"' | protoc --encode=shop.Order order.proto | protoc --decode_raw
# 1: 150
# 2: "ada"

protoc --python_out=. --java_out=. order.proto  # generated classes per language

Protocol Buffers, Avro and JSON#

All three can carry the same record. They differ in where the field names live and what a reader needs to make sense of the bytes.

Protocol BuffersAvroJSON
SchemaA .proto file, compiled to codeJSON schema; code generation optionalNone in the format
How a field is found in the bytesBy the field number in its tagBy position, following the writer’s schemaBy its name, written out in every object
Read without the schemaNumbers and raw values onlyNo: values carry no tagsYes
Adding a fieldGive it a new number; old readers skip itGive it a default; fields are matched by nameUp to each application
Bytes for id 150 and customer “ada”8627
  • Sizes are for the bare encodings: 08 96 01 12 03 61 64 61 in Protobuf, a zigzag varint and a length-prefixed string in Avro, and {"id":150,"customer":"ada"} without spaces. Avro container files add a header with the schema, and Confluent’s Kafka serializers put a version byte and a 4-byte schema ID in front of each message by default.

proto2, proto3 and Editions#

The bytes on the wire are the same in all three; what changes is the .proto syntax and the generated code’s behaviour.

proto2

2008

The first open-source version, a rewrite of Google’s internal Proto1. Fields are optional, required or repeated; optional fields remember whether they were set, defaults can be customised, and enums are closed. Still supported by current protoc.

proto3

2016

Released alongside gRPC. Drops required; plain fields have no presence, so zero and unset look the same, and defaults are fixed per type. Repeated numbers are packed and enums open. Unknown fields were dropped at first and kept again from 3.5 (2017); optional returned by default in 3.15 (2021).

Editions

2023 onwards

edition = "2023" replaces the syntax line. Each behaviour (presence, packing, open or closed enums, UTF-8 checks) becomes a feature with a default per edition, which you can override per file, message or field. Edition 2023 needs protoc 27.0 (May 2024), 2024 needs 32.0 (August 2025) and 2026 needs 36.0 (August 2026).

See it in a real stack

No architecture uses Protocol Buffers yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2001–2026

Google used thousands of data formats for messages between servers, index records and more. XML was too expensive at that scale, writing raw in-memory structs broke as soon as a new server version had to talk to old ones, and hand-written parsers didn’t scale to tens of thousands of structures. So, starting in early 2001, Google built Protocol Buffers: define records in a small language and compile them into classes. This first version, Proto1, grew new features whenever a team needed one and became, in Google’s own words, “a bit of a mess”.

Proto2, a complete rewrite whose primary author was Kenton Varda, kept most of the design but dropped the dependencies on Google’s internal libraries. Google open-sourced it on July 7, 2008, partly because other projects it wanted to release depended on it. A new generation, proto3, came out alongside gRPC in February 2015 and was declared stable in version 3.0.0 in July 2016.

In 2023 Google introduced Editions, which turn the all-or-nothing choice between proto2 and proto3 into individual features that can change a little with each yearly edition.

Why it’s built this way

Field numbers on the wire, names only in the schema
Numbers keep messages small, and they let servers running different versions exchange data, which Google needed because new servers almost always start out talking to old ones.
Every record says how to skip it
The wire type tells a parser how long the value is, so old code can step over fields added after it was built instead of failing.
Generated classes instead of hand-written parsers
With tens of thousands of structures in Google’s code base, writing each serializer by hand had already failed. A compiler produces the same optimised code for every type and language.
No required fields in proto3
A required field can never safely be dropped, because any older parser in the path rejects messages without it. Google’s guide says required fields were “considered harmful by so many they were removed from proto3 completely”.

Timeline

  1. 2001

    Work on Proto1 starts inside Google

  2. 2008

    Proto2 is open-sourced on July 7; licence moves to BSD in September

  3. 2015

    proto3 arrives alongside gRPC in February

  4. 2016

    Version 3.0.0 makes proto3 stable in July

  5. 2017

    Version 3.5 keeps unknown fields in proto3 again

  6. 2024

    Editions officially released in protoc 27.0 in May

  7. 2026

    Edition 2026 arrives with protoc 36.0 in August

Sources16 · facts checked 6 October 2026

Improve this page on GitHub