NimbleExperimental

Nimble stores each column as one or more streams and is designed for tables that are wide rather than long: thousands to tens of thousands of columns, as found in feature engineering and model training at Meta. Its metadata is organized to stay light at that width and is stored in FlatBuffers instead of the Thrift or Protocol Buffers that Parquet and ORC use, so large metadata sections can be accessed efficiently. It encodes data in blocks rather than as one continuous stream, which keeps memory use while reading predictable.

Encoding is the part Nimble makes pluggable. A block can be stored as trivial values, a constant, a mostly-constant value plus exceptions, run-length, dictionary, bit-packed, varint, delta, prefix-compressed strings, or sparse booleans, among others, and encodings cascade: a dictionary’s list of unique values and its index array are each encoded again with whatever fits them best, and Zstd compression is applied at each level only when it helps. Which encoding to use is decided by a selection policy: the default estimates encoded size and weights it by how fast each encoding decodes, an alternative uses a machine-learning model, and a third replays a recorded layout. The library is C++, built with CMake, and is closely tied to Velox, Meta’s execution engine library, which it includes as a Git submodule. Meta asks other projects to use this one library and write bindings rather than reimplement the spec, to avoid the incompatible readers that older formats suffered from.