BentoML

A Python library that turns model inference code into an HTTP service with adaptive batching, then packages code, models and dependencies into a versioned “Bento” and a Docker image.

What it is: Model serving and MLOps
Type: Open source
Made by: Modular
License: Apache 2.0
Facts checked: 23 October 2026

What it is

A BentoML service is a Python class with the @bentoml.service decorator, usually in service.py. Its methods marked @bentoml.api become HTTP endpoints, POST by default, with inputs and outputs declared as ordinary type hints: strings, Pydantic models, numpy.ndarray, torch.Tensor, data frames or pathlib.Path for files. The decorator also sets resources (resources={"gpu": "1"}), timeouts (traffic={"timeout": 10}) and the number of worker processes. Model loading goes in __init__, so it runs once per worker. One service can call another through bentoml.depends(), which looks like a local method call but lets each part of a pipeline run, and scale, as its own service, for example CPU preprocessing in front of a GPU model.

Batching is opt-in per endpoint with @bentoml.api(batchable=True): the method then takes and returns a list, and a dispatcher collects concurrent requests into one call. The dispatcher adapts the batch window to recent traffic, within the max_batch_size and max_latency_ms you set, and returns HTTP 503 when it can’t keep up. For deployment, the runtime is declared in code with bentoml.images.Image (Python version, packages, a requirements file), bentoml build packages the project into a Bento with a unique version, and bentoml containerize turns that Bento into a Docker image whose tag matches the Bento’s. bentoml serve runs the same service locally on port 3000.

BentoML started in 2019 at a startup of the same name, founded by Chaoyu Yang, an early Databricks engineer. Modular, the company behind the Mojo language and the MAX model-optimization stack, bought it in February 2026 and said the project would stay Apache 2.0; Qualcomm in turn completed its purchase of Modular in July 2026. The managed service is sold as the Bento Inference Platform. The latest release is 1.4.39, from May 2026.

Why teams pick it

Any Python inference code becomes a typed HTTP API with a few decorators, and the same code runs locally, in Docker or on Kubernetes
Adaptive batching adjusts the batch window to traffic instead of using a fixed wait, within latency and size limits you set
Multi-service pipelines with bentoml.depends(): each stage gets its own resources and workers, for example a CPU tokenizer in front of a GPU model
Reproducible builds: a Bento records code, model versions and dependencies, and becomes an image with one command

Watch out for

Scaling across many machines is left to you or to the paid Bento Inference Platform: the open-source Kubernetes operator, Yatai, was archived in June 2026
Python only: services, pipelines and image definitions are all Python code
Ownership has changed twice in 2026, from an independent startup to Modular and then to Qualcomm, and releases slowed: none between May and October 2026
The 1.2 release in 2024 introduced service classes in place of Runners; old code still runs, but many older tutorials and examples use the previous style

The BentoML family

1 technologies built from or on BentoML

Tools

YataiArchivedOpen sourceA Kubernetes operator and web console for deploying BentoML services, with custom resources for building Bento images and running them as autoscaled deployments. Archived in June 2026; it never supported BentoML 1.2’s service API.

Alternatives

What all model serving and mlops have in common →Compare side by side →

Ray ServeOpen sourceRay’s library for online inference: Python classes become deployments whose replicas run as Ray actors across a cluster, with request batching, model composition, autoscaling to zero and an OpenAI-compatible LLM server built on vLLM.

How it fits with other technologies

What BentoML uses

service.py from the README: a batched summarization API · python

import bentoml

@bentoml.service(
    image=bentoml.images.Image(python_version="3.11").python_packages("torch", "transformers"),
)
class Summarization:
    def __init__(self) -> None:
        import torch
        from transformers import pipeline

        device = "cuda" if torch.cuda.is_available() else "cpu"
        self.pipeline = pipeline('summarization', device=device)

    @bentoml.api(batchable=True)
    def summarize(self, texts: list[str]) -> list[str]:
        results = self.pipeline(texts)
        return [item['summary_text'] for item in results]

# bentoml serve                              -> http://localhost:3000
# bentoml build                              -> a versioned Bento
# bentoml containerize summarization:latest  -> a Docker image

See it in a real stack

No architecture uses BentoML yet. Drop it into the playground and wire it up yourself.Open playground →

Where it came from2019–2026

Chaoyu Yang, an early engineer at Databricks, started BentoML to make it easier to put models into production. The project was open-sourced in 2019; its GitHub repository dates from April 2019. When BentoML raised a $9 million seed round led by DCM Ventures in June 2023, TechCrunch counted more than 3,000 members in its open-source community and named Line and Naver as early users, and Yang said a demo app could take three to six months to become production-ready.

BentoML 1.0 shipped in July 2022, built around Runners, separate units for model inference that could scale apart from the API code. As LLMs and generative AI grew, the team found that approach “less optimal”. Version 1.2, in February 2024, replaced it with service classes and @bentoml.api methods, moved configuration from YAML into Python, and added bentoml deploy to build and ship to BentoCloud in one step.

On February 10, 2026 Modular announced that BentoML was joining it, citing more than 10,000 organizations using BentoML, including over 50 of the Fortune 500. Modular said the open-source project would continue under Apache 2.0 with the same docs, community and contribution process. Qualcomm announced a deal to buy Modular in June 2026 and completed it in July.

Why it’s built this way

Plain Python classes and type hints for the API: Many people building AI services come from full-stack backgrounds rather than ML infrastructure. Version 1.2 moved from decorator parameters and YAML files, which became “cumbersome” for complex schemas, to classes and functions Python developers already know.
Batching that adapts instead of a fixed window: Traffic changes during the day. Learning from recent request patterns lets the server build large batches under load and small, fast ones when traffic is light, while staying inside the latency limit you set.
A Bento as the unit of deployment: Models break when code, weights and library versions drift apart. Packaging all three into one versioned artifact, then into an image, makes a deployment reproducible on any machine that runs Docker.

Timeline

2019
BentoML is open-sourced (GitHub repository created in April)
2022
BentoML 1.0 (July); Yatai 1.0 for Kubernetes (November)
2023
$9 million seed round led by DCM Ventures (June)
2024
BentoML 1.2 replaces Runners with service classes (February)
2025
BentoML 1.4 (February)
2026
Modular acquires BentoML (February); Qualcomm completes its purchase of Modular (July)

Sources9 · facts checked 23 October 2026

Improve this page on GitHub