Cloud Parts

Apache TsFile

A columnar file format for device time series, with per-series indexes and time-series encodings; the storage format of Apache IoTDB and its own Apache project since 2023.

What it is
File formats (columnar files)
Type
Open source
Made by
Apache Software Foundation
License
Apache 2.0
Facts checked
9 October 2026
Website Source code

What it is

A TsFile holds the readings of many devices, organised in four layers. A page stores a run of one series’ timestamps and values in time order, as two separate columns. A chunk is a header plus pages for one series. A chunk group holds the chunks of one device written in the same period. An index at the end of the file locates each series’ chunks and keeps file-level statistics. Minimum and maximum time and value and the count of points are kept for the file, each chunk and each page, so a reader skips pages that can’t match and answers some aggregates without decoding data. Unlike Parquet or ORC, metadata is stored per series, so reading one series never touches the metadata of the others.

Each column is encoded for its shape and then compressed: TS_2DIFF (second-order differences) for timestamps and counters, GORILLA or CHIMP for floats that change slowly, DICTIONARY for repetitive strings, RLE, SPRINTZ, RLBE and others, followed by LZ4 by default, or Snappy, ZSTD, GZIP or LZMA2. Since 2.0 (January 2025) the format also has a table model that matches IoTDB’s: a table per device type, string TAG columns that together identify a device, FIELD columns for the measurements, and a time column that must be in order for each device.

Libraries exist for Java, C++, C and Python (pip install tsfile), and the Python package can open a folder of TsFiles as a lazy, DataFrame-like view. DuckDB reads and writes table-model TsFiles through a community extension. IoTDB loads a TsFile with one LOAD statement or by watching a folder, so an edge device can write files offline and ship them later. TsFile grew inside IoTDB at Tsinghua University and became a top-level Apache project on 15 November 2023, without passing through the Incubator. The latest release is 2.4.0 (July 2026); a 1.1.x line is kept for IoTDB 1.x.

Why teams pick it

  • Per-series indexes and statistics at file, chunk and page level, so a query for one device and time range reads only the chunks it needs
  • Encodings built for sensor data (second-order deltas, Gorilla, Chimp, Sprintz), chosen per column
  • The same files are IoTDB’s storage, so data written on a device or in Python loads straight into the database without conversion
  • Apache 2.0, with native libraries for Java, C++, C and Python, a DuckDB extension, and bundled scripts that convert CSV, Parquet and Arrow files

Watch out for

  • Few engines read it compared with Parquet: outside IoTDB there’s the DuckDB extension (local, table-model files only) and a Spark connector documented only for Spark 2.4.3
  • The Python guide covers only the table model, and TsFileDataFrame reads only numeric fields, skipping STRING, TEXT, BLOB and DATE columns
  • RLE and TS_2DIFF keep only two decimal places of a float by default; GORILLA is the recommended lossless choice
  • Rows must arrive in time order per device (DuckDB’s writer wants them sorted by tag columns, then time), and the 1.1.x and 2.x lines pair with different IoTDB major versions

Alternatives

What all columnar files have in common →
Apache ORCOpen sourceA columnar file format born in Hive: typed columns in large stripes, built-in indexes every 10,000 rows, and the storage behind Hive’s ACID tables.Apache ParquetOpen sourceThe standard columnar file format for analytics: each column stored and compressed on its own, with statistics that let engines skip data.PuffinOpen sourceApache Iceberg’s file format for the data that doesn’t fit in a manifest: a container of typed binary blobs, such as distinct-value sketches and deletion vectors, described by a JSON footer.VortexOpen sourceA columnar file format and toolkit that keeps data compressed with modern encodings, reads single rows fast and runs filters directly on the compressed data, pitched as a faster successor to Parquet.

How it fits with other technologies

Write a table-model TsFile and read it back with the Python library (ran with tsfile 2.4.0) · python

# pip install tsfile   (2.4.0: Python 3.9+, wheels for Linux, macOS and Windows)
from tsfile import (ColumnSchema, ColumnCategory, TableSchema, Tablet,
                    TSDataType, TsFileTableWriter, TsFileReader, to_dataframe)

schema = TableSchema("turbine", [
    ColumnSchema("farm", TSDataType.STRING, ColumnCategory.TAG),
    ColumnSchema("turbine_id", TSDataType.STRING, ColumnCategory.TAG),
    ColumnSchema("wind_speed", TSDataType.DOUBLE, ColumnCategory.FIELD),
    ColumnSchema("power_kw", TSDataType.DOUBLE, ColumnCategory.FIELD),
])
rows = [(1000, "north", "t1", 7.5, 1510.0), (2000, "north", "t1", 8.1, 1720.0),
        (1000, "north", "t2", 6.9, 1380.0), (2000, "north", "t2", 7.2, 1450.0)]

with TsFileTableWriter("turbines.tsfile", schema) as writer:
    tablet = Tablet(["farm", "turbine_id", "wind_speed", "power_kw"],
                    [TSDataType.STRING, TSDataType.STRING,
                     TSDataType.DOUBLE, TSDataType.DOUBLE], len(rows))
    for i, (ts, farm, turbine, speed, power) in enumerate(rows):
        tablet.add_timestamp(i, ts)
        tablet.add_value_by_name("farm", i, farm)
        tablet.add_value_by_name("turbine_id", i, turbine)
        tablet.add_value_by_name("wind_speed", i, speed)
        tablet.add_value_by_name("power_kw", i, power)
    writer.write_table(tablet)

# read two columns for times 0 to 1500: prints "t1 1510.0" and "t2 1380.0"
with TsFileReader("turbines.tsfile") as reader:
    with reader.query_table("turbine", ["turbine_id", "power_kw"], 0, 1500) as rs:
        while rs.next():
            print(rs.get_value_by_name("turbine_id"), rs.get_value_by_name("power_kw"))

print(to_dataframe("turbines.tsfile"))   # all four rows as a pandas DataFrame

Query the same file from DuckDB#

The community extension adds read_tsfile(path, table). Time filters and column choices are pushed down to the TsFile reader. Ran on DuckDB 1.5.6.

sql
INSTALL tsfile FROM community;
LOAD tsfile;

SELECT turbine_id, avg(power_kw) AS avg_kw
FROM read_tsfile('turbines.tsfile', 'turbine')
WHERE time >= 1000
GROUP BY turbine_id ORDER BY turbine_id;
-- t1  1615.0
-- t2  1415.0

TsFile next to Parquet#

Both are columnar; they split the data differently.

TsFileParquet
Unit of groupingChunk group: one device’s series for a periodRow group: a slice of rows across all columns
TimestampsStored with every series, so series sampled at different instants need no nullsAn ordinary column, shared by the row
Nested dataNone; flat device, series, time, valueStructs, lists and maps, via repetition and definition levels
StatisticsPer file, chunk and page, indexed by seriesPer column chunk in the footer, plus an optional page index
Who reads itIoTDB, the TsFile libraries, DuckDBNearly every engine and dataframe library

See it in a real stack

No architecture uses Apache TsFile yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2018–2026

TsFile was born inside IoTDB at Tsinghua University. When the team proposed IoTDB to the Apache Incubator in 2018, TsFile was the part already open source on GitHub, described as a columnar format similar to Parquet and CarbonData but built for time series of the form device, series, timestamp, value.

Its design started from how sensors behave. Two sensors on one device rarely sample at the same millisecond, so forcing them into rows means storing nulls, and Parquet’s repetition and definition levels for nested data add work that flat series don’t need. TsFile stores each series with its own timestamps, puts time-series statistics in every page header, and keeps a device’s chunk groups in time order so a time-range scan can stop early. IoTDB used it as its storage format, and Hadoop, Hive and Spark could read the same files directly, so analysts didn’t need an export.

In October 2023 the IoTDB community discussed splitting TsFile off so it could serve as a general file format for IoT rather than a database internal. On 15 November 2023 the ASF board established Apache TsFile directly as a top-level project, with an initial PMC of IoTDB committers, and handed it the IoTDB TsFile subproject. Version 1.0.0 followed in February 2024, and 2.0.0 in January 2025 added a format version for the table model. Later releases built out the C++ and Python libraries.

Why it’s built this way

Timestamps stored with each series
Sensors on one device sample at slightly different times. Giving every column chunk its own timestamps avoids padding with nulls and the repetition and definition fields Parquet needs.
Statistics in every page header
Minimum and maximum values and timestamps let a reader skip pages that can’t match a filter, which is most of them in a narrow time-range query.
Metadata indexed per series at the end of the file
A query usually asks for a few series of one device. Keeping each series’ metadata separate means a reader doesn’t load metadata for columns it won’t read.
Its own Apache project
The IoTDB community argued that a format with its encodings and self-describing files could become a standard for IoT data beyond one database, which needed separate governance and releases.

Timeline

  1. 2018

    Already open source when IoTDB enters the Apache Incubator in November

  2. 2020

    VLDB paper describes it as IoTDB’s storage format, readable from Spark and Hive

  3. 2023

    Established as a top-level Apache project on 15 November

  4. 2024

    1.0.0 released in February

  5. 2025

    2.0.0 adds the table-model format in January

  6. 2026

    2.4.0 in July speeds up the C++ library with SIMD and parallel reads

Sources18 · facts checked 9 October 2026

Improve this page on GitHub