Cloud Parts

Binary parsing and encoding

Reading and writing bytes laid out in a fixed format, such as packet headers, file headers and firmware messages, without trusting what those bytes claim about themselves.

5 technologies on this page ↓

How it works

A binary format fixes where every field sits and how it is stored. Multi-byte numbers have a byte order: big-endian puts the most significant byte first, as the IPv4 header does (hence “network byte order”), while little-endian puts the least significant byte first, as x86 and Apple M1 processors do in memory. Read the same two bytes 00 07 in the wrong order and sensor 7 becomes sensor 1792. Small values share a byte as bit fields: IPv4 packs a 4-bit version and a 4-bit header length into its first byte, and a parser pulls them apart with shifts and masks. Variable-length data needs framing. A length prefix says how many items or bytes follow; tag-length-value (TLV) adds a type tag in front, so an old reader can skip a field it doesn’t know, as Protocol Buffers does. Every length, count and offset in the input is a claim made by whoever sent it, so a parser checks it against the bytes it actually has before reading. Libraries take three broad approaches: annotate a struct and let the library derive the reader and writer (declarative layouts), compose small parsing functions (parser combinators), or write a schema and let a compiler generate the encoder and decoder.

Think of it as

A customs form with numbered boxes. Box 1 holds a four-letter code, box 2 says how many item lines follow, and each line has the same three columns. A careful clerk counts the attached lines before stamping anything, rather than taking box 2 at its word.

Reach for it when

  • Speaking a format someone else defined: network packets, file headers, device and firmware protocols
  • Microcontrollers and radio links, where each byte on the wire and each kilobyte of code matters
  • Large files or high message rates, where fixed offsets beat scanning text

Think twice when

  • You control both ends and only need to move data: use an existing format such as Protocol Buffers, CBOR or JSON instead of inventing a layout
  • Copying an in-memory C struct onto the wire: padding, alignment and byte order then depend on the compiler and the CPU
  • Hand-writing parsers for untrusted input in a memory-unsafe language when a generated or memory-safe parser is available

A real example

Temperature sensors send 10-byte packets over a radio link: the magic bytes SN, one byte holding a 4-bit version and 4-bit flags, a big-endian 16-bit sensor id, a count, then that many signed 16-bit readings in hundredths of a degree. The same data as compact JSON, {"sensor":7,"temps":[2100,-100]}, is 32 bytes. The gateway reads the count, checks that exactly two bytes per reading follow, and drops packets cut short in transit instead of reading past the end of its buffer.

Kinds of binary parsing and encoding

Specific products, and what each does differently.

Three ways to write the parser#

All three read the same bytes. They differ in where the description of the format lives and whether you also get a writer for free.

Declarative layoutsParser combinatorsCode from a schema
You writeA struct or schema object with attributes: sizes, byte order, counts, magic bytesSmall functions (“take 2 bytes”, “match SN”) combined into bigger onesA message definition, such as a .proto file
Who decides the bytesThe format’s author; you describe it as it isThe format’s author; you describe it as it isThe schema compiler’s own encoding
Writing as well as readingYes, from the same descriptionNo, the encoder is separate codeYes, both are generated
Strong atFixed headers, bit fields, counted arraysStreams that arrive in chunks, formats where later fields depend on earlier ones, text as well as binaryFormats you own and evolve; many languages at once
Examplesdeku and binrw (Rust), Construct (Python), Kaitai Struct (YAML, many languages)nom (Rust), Hammer (C)nanopb and the official Protocol Buffers compilers

Treat every length as a claim#

Most binary formats carry their own lengths, and the bug that keeps recurring is believing them. The TLS heartbeat extension (RFC 6520) sends a uint16 payload_length followed by the payload, and the RFC says to discard a message whose length is too large. OpenSSL 1.0.1 before 1.0.1g did not handle these packets properly, and crafted heartbeats made it over-read its buffer and send back process memory, private keys included: Heartbleed, CVE-2014-0160.

The language-theoretic security (LangSec) community argues for a stricter habit: treat the set of valid inputs as a formal language and recognize the whole input before acting on any of it, instead of scattering checks through the program. Libraries help by making the checks structural. nanopb promises it will never read more than the stream’s remaining bytes or write outside the message struct, whatever the input; nom returns an error or “need more data” rather than reading past its slice. Speed shortcuts can quietly drop those checks: Construct’s compiled parsers omit some of them, and its docs say not to use them on corrupted or untrusted data.

In code · python

import struct

# "SN" | version: 4 bits | flags: 4 bits | sensor id: u16 | count: u8 | count × i16
packet = bytes.fromhex("534e 12 0007 02 0834 ff9c")

def parse(buf: bytes) -> dict:
    if len(buf) < 6 or buf[:2] != b"SN":
        raise ValueError("not a reading")
    vf, sensor_id, count = struct.unpack_from(">BHB", buf, 2)  # > = big-endian
    version, flags = vf >> 4, vf & 0x0F                          # two 4-bit fields
    if version != 1:
        raise ValueError(f"unsupported version {version}")
    if len(buf) != 6 + 2 * count:                                # never trust a length
        raise ValueError(f"expected {6 + 2 * count} bytes, got {len(buf)}")
    temps = struct.unpack_from(f">{count}h", buf, 6)
    return {"version": version, "flags": flags, "sensor": sensor_id, "temps": list(temps)}

print(parse(packet))
# {'version': 1, 'flags': 2, 'sensor': 7, 'temps': [2100, -100]}

print(struct.unpack("<H", packet[3:5])[0])  # same two bytes read little-endian: 1792, not 7

try:
    parse(packet[:8])  # truncated in transit
except ValueError as e:
    print(e)  # expected 10 bytes, got 8

Often confused with

File formatsNetwork protocolsRemote procedure callsFunctional programming
Where it came from1980–2020

Byte order was argued over before most of today’s formats existed. In April 1980 Danny Cohen’s note “On Holy Wars and a Plea for Peace” borrowed the Big-Endians and Little-Endians from Gulliver’s Travels to name the two camps. The IPv4 specification, RFC 791 of September 1981, chose big-endian for the internet protocol: the most significant octet of a field goes first, and small fields like the 4-bit version share a byte. Programs read such headers with hand-written code that copied bytes, shifted and masked.

That hand-written code, mostly in C, became a leading source of security bugs. A 2017 paper by Pierre Chifflier and Geoffroy Couprie counted 95 vulnerabilities in the Wireshark protocol analyser in 2016 alone, and Heartbleed in 2014 was a buffer over-read in OpenSSL’s handling of TLS heartbeat messages, which carry their own length field. The LangSec movement, from Len Sassaman and Meredith L. Patterson’s 2010 Black Hat talk onwards, argued that input handling should be a recognizer for a precisely specified language rather than checks scattered through a program.

Libraries took the ad hoc code away in three ways. For declarative layouts, Tomer Filiba’s Construct reached version 2.0 in Python around 2007, and Rust’s deku and binrw (2020) moved the idea into derive macros. Parser combinators, which Graham Hutton described in 1992, reached binary formats through the C library Hammer and through nom (2015) in Rust, which drew on Haskell’s Parsec. Schema compilers such as Protocol Buffers chose their own bytes, and nanopb (2012) fitted that approach onto microcontrollers.

Why it’s built this way

Describe the layout once, derive both directions
A reader and a writer written separately drift apart. Construct’s docs put it plainly: the description is a data structure, not code, so it can parse in one direction and build in the other.
Parsers as pure functions that return the rest of the input
With no mutable state, small parsers can be tested alone and combined freely, and in Rust a parser that returns slices of its input can’t read past the end of them.
Check everything before using anything
LangSec’s argument: when validation is scattered through a program, what it actually accepts no longer matches the programmer’s assumptions, which leaves room for exploits.
Let a compiler choose the bytes when you own the format
If nobody else defined the layout, a schema compiler can pick one that old readers can skip through, as Protocol Buffers’ field tags allow, and generate code for every language in use.

Timeline

  1. 1980

    Danny Cohen names the big-endian and little-endian camps

  2. 1981

    RFC 791 fixes the IPv4 header: bit fields, most significant octet first

  3. 1992

    Graham Hutton’s “Higher-order functions for parsing”

  4. 2007

    Construct 2.0, a declarative binary parser and builder for Python

  5. 2014

    Heartbleed; the first LangSec workshop at IEEE Security & Privacy

  6. 2015

    nom 1.0 brings parser combinators for binary data to Rust

  7. 2020

    deku and binrw derive binary readers and writers from Rust structs

Sources10

Improve this page on GitHub