cloudparts

Model serving and MLOps

The work between a trained model and a running product: recording every training run and model version, packaging a model with its code and dependencies as a service, and serving it with batching and autoscaling so you always know which model answers and how it was made.

4 technologies on this page ↓

How it works

Three jobs make up the loop. Tracking comes first: each training run logs its parameters, metrics and output files to a tracking server. In MLflow, a run’s metadata goes to a backend store, a database such as PostgreSQL, MySQL or SQLite, and large files such as model weights go to an artifact store such as Amazon S3. A registry turns the good runs into named, numbered versions: the first model registered as churn becomes version 1, the next version 2, and a mutable alias such as champion points at one of them, so a server loads models:/churn@champion and a rollout or rollback is a change of pointer. Packaging bundles the model with the code that calls it and the libraries both need, usually into a container image: bentoml build makes a Bento and bentoml containerize turns it into a Docker image; mlflow models build-docker does the same for an MLflow model. Serving runs that image as an HTTP service with several replicas. Because one model call can keep a CPU or GPU busy for tens of milliseconds, model servers batch: Ray Serve’s @serve.batch waits up to 10 ms by default for up to 10 requests and runs them as one call, and BentoML’s adaptive batching tunes the batch window itself within a max_latency_ms limit, returning 503 when it can’t keep up. Autoscaling then adds or removes replicas by load, not CPU: Ray Serve aims for an average of 2 ongoing requests per replica by default, can go down to zero replicas, and asks the cluster autoscaler for new machines when there is no room for another GPU replica. New versions go out beside old ones: an Azure Machine Learning endpoint can send 90% of traffic to a blue deployment and 10% to green, or copy traffic to green without returning its answers.

Think of it as

A bakery with a test kitchen. The test kitchen keeps a notebook of every batch: the flour, the oven temperature and how it tasted (tracking). Recipes that pass go into a binder as numbered versions, and a sticky note marks the one the shop bakes today, so moving the note changes the bread without reprinting the binder (registry and alias). The shop doesn’t bake one loaf per customer: it fills a tray before it fires the oven (batching), and lights more ovens when the queue gets long (autoscaling).

Reach for it when

  • Models are retrained regularly and you must be able to say which data, code and parameters produced the model that is live right now
  • Several people train models for the same problem and compare results: one tracking server replaces spreadsheets and folder names like model_final_v3
  • A model runs on GPUs and requests arrive one at a time: server-side batching turns many small calls into one pass over the hardware
  • Traffic varies and GPU time is expensive: replicas scale with the number of requests in flight, and rarely used models scale to zero
  • You want to try a new version on a share of real traffic, and roll back by moving an alias or a traffic split instead of rebuilding

Think twice when

  • You call a hosted model API such as Claude or GPT: the provider does the serving, and what you need is a gateway and tracing, not a model server
  • The model scores a table once a night: a batch job over the data is simpler and cheaper than an endpoint that runs all day
  • One person, one small model, few requests: a FastAPI app in a Docker container is enough, and a tracking database, registry and Kubernetes operator are more to run than the model itself
  • Every millisecond of latency counts for a small, fast model: batching makes the first request wait for others, so keep the wait short or turn batching off

A real example

A team retrains a churn model every Monday. Each training run logs its parameters and validation AUC to a shared MLflow tracking server, which keeps metadata in PostgreSQL and model files in S3. This week’s best run is registered as version 7 of churn, and after checking it on held-out data the team moves the champion alias from version 6 to 7. The model is served by a Ray Serve deployment whose replicas load models:/churn@champion when they start, batch up to 32 requests per call and autoscale between 1 and 10 replicas to keep about 2 requests in flight per replica. After the alias moves, the team redeploys, Ray Serve starts new replicas that load version 7, and the old ones drain. When the support dashboard shows odd scores on Tuesday, rolling back is moving champion back to version 6 and redeploying: nothing is rebuilt, and the run that produced version 6 still shows exactly what it was trained on.

Technologies

Specific products, and what each does differently.

Also used for model serving and mlops

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
Amazon SageMaker AI: real-time endpoints with automatic scaling (down to zero instances), SageMaker Model Registry (Model Groups, versions, approval status), and fully managed MLflow (MLflow Apps and MLflow tracking servers)
Google Cloud
Agent Platform Endpoints and Online Inference, Agent Platform Model Registry and Experiments on Agent Platform: the former Vertex AI services, renamed in April 2026 as part of Gemini Enterprise Agent Platform
Azure
Azure Machine Learning: managed online endpoints that split or mirror traffic between deployments, batch endpoints, and workspaces that act as an MLflow tracking server and model registry
Open source & others
Serving: BentoML, Ray Serve (on Kubernetes with KubeRay), KServe; tracking and registry: MLflow (Apache 2.0, a Linux Foundation project)

Who does which job#

Open-source tools usually do one or two of the jobs; cloud platforms bundle all of them.

JobMLflowBentoMLRay ServeAmazon SageMaker AI
Track runsRuns, parameters, metrics and artifacts; autologging for scikit-learn, PyTorch and othersNoNoManaged MLflow
RegistryRegistered models, numbered versions, aliases and tagsA local Model Store and Bentos with versionsNoSageMaker Model Registry, fed automatically from MLflow’s registry
PackageMLflow Models with flavors; mlflow models build-dockerbentoml build makes a Bento; bentoml containerize makes an imagePython code plus a Serve config fileModelBuilder prepares MLflow models for an endpoint
Servemlflow models serve, or deploy to SageMaker, Azure ML or KubernetesHTTP server with adaptive batching, workers and multi-model pipelinesReplicas on a Ray cluster, @serve.batch, model composition, LLM servingReal-time endpoints, which can also be invoked through OpenAI-compatible APIs
AutoscaleLeft to the platform it deploys toOn BentoCloudOngoing requests per replica, scale to zero, cluster autoscalerApplication Auto Scaling policies, scale to zero instances

Batching trades latency for throughput#

A GPU that runs one request at a time spends most of its time idle between kernels, while 32 requests stacked into one tensor cost little more than one. Clipper, a research serving system from UC Berkeley published in 2016, put batching, caching and model selection into a layer between applications and ML frameworks, and the idea is now standard. Ray Serve’s @serve.batch is a fixed rule: wait up to batch_wait_timeout_s (10 ms by default) for up to max_batch_size requests (10 by default), then call the method once with a list. BentoML’s dispatcher learns from recent traffic: it builds larger batches when traffic is heavy and smaller ones when it is light, within the max_batch_size and max_latency_ms you set.

The cost is that the first request in a batch waits for the others, and a batched method must take a list and return a list of the same length. Large language models are a special case: engines such as vLLM batch continuously at the level of individual tokens, so LLM serving layers such as Ray Serve LLM put the engine inside a replica and concentrate on routing, autoscaling and an OpenAI-compatible API around it.

In code · python

import mlflow, pandas as pd
from ray import serve

# Training: log the run, register a new version, point the alias at it
with mlflow.start_run():
    mlflow.log_param("max_depth", 6)
    mlflow.log_metric("val_auc", 0.91)
    mlflow.sklearn.log_model(sk_model=model, name="model", registered_model_name="churn")
mlflow.MlflowClient().set_registered_model_alias("churn", "champion", version=7)

# Serving: load whatever "champion" points to, batch requests, autoscale
@serve.deployment(autoscaling_config={
    "min_replicas": 1, "max_replicas": 10, "target_ongoing_requests": 2})
class Churn:
    def __init__(self):
        self.model = mlflow.pyfunc.load_model("models:/churn@champion")

    @serve.batch(max_batch_size=32, batch_wait_timeout_s=0.01)
    async def score(self, rows: list[dict]) -> list[float]:
        return list(self.model.predict(pd.DataFrame(rows)))

    async def __call__(self, request):
        return await self.score(await request.json())

serve.run(Churn.bind())

Often confused with

Large language modelsModel hubsRunning models locallyModel routers and gatewaysAutoscalingContainersCanary releasesObservabilityML frameworks
Where it came from2015–2026

By the mid-2010s many companies had working models and no reliable way to run them. In 2015 a Google team led by D. Sculley described machine learning in production as a source of “massive ongoing maintenance costs”, from tangled data dependencies to feedback loops. When Uber described its Michelangelo platform in September 2017, it wrote that before it “there was no established path to deploying a model into production”: teams built a custom serving container for every project. Michelangelo gave each model a UUID and an optional tag or alias, so a new model could replace an old one without changing clients.

Serving became its own research topic. Google described TensorFlow Serving, its internal and open-source model server, and UC Berkeley’s RISELab published Clipper (2016), which added caching, batching and model selection between applications and ML frameworks. Then came open tools for the rest of the loop. Databricks released MLflow in June 2018 to track experiments, package code and save models in a common format, and gave it to the Linux Foundation in 2020. BentoML was open-sourced in 2019 by Chaoyu Yang, an early Databricks engineer. Ray Serve followed in 2020, arguing that wrapping a model in Flask didn’t scale and that specialized servers forced a “tensor-in, tensor-out” API on developers.

Large language models shifted the work again. MLflow 3 (June 2025) added tracing and evaluation for LLM apps and agents, and Ray 2.44 (March 2025) gave Ray Serve an LLM module with vLLM inside. The tools also consolidated: Modular bought BentoML in February 2026, and BentoML’s Kubernetes operator, Yatai, was archived in June 2026 after two years without a release.

Why it’s built this way

Batch requests in the server, within a latency budget
Models run far faster on a batch than on single inputs, but callers send one request at a time. Clipper showed that the serving layer can collect requests for a few milliseconds and run them together, without changing the ML framework; BentoML and Ray Serve both ship this as a decorator.
Name versions, and point at them with aliases
Clients should not change when the model does. Michelangelo let a tag stand for a model UUID, and MLflow’s registry uses mutable aliases such as champion, so promoting or rolling back a model means moving one pointer.
Serve Python code, not just a model file
Real requests need preprocessing, business logic and often several models. Ray Serve’s authors saw specialized model servers split request handling and inference into two services; BentoML and Ray Serve let one Python class hold both and scale it as a unit.
Open interfaces instead of one platform
Companies used many ML libraries and deployment targets. MLflow’s launch post set out to work “with any ML library, algorithm, deployment tool or language” through REST APIs and simple file formats, which is why cloud platforms such as SageMaker AI and Azure Machine Learning now speak MLflow’s API.

Timeline

  1. 2015

    “Hidden Technical Debt in Machine Learning Systems” published at NIPS

  2. 2016

    Clipper paper from UC Berkeley’s RISELab (December)

  3. 2017

    Uber describes Michelangelo (September); TensorFlow-Serving paper (December)

  4. 2018

    Databricks releases MLflow in alpha (June)

  5. 2019

    BentoML is open-sourced

  6. 2020

    Ray Serve is released (July post); MLflow joins the Linux Foundation (June)

  7. 2025

    Ray Serve LLM in Ray 2.44 (March); MLflow 3 with GenAI tracing and evaluation (June)

  8. 2026

    Modular acquires BentoML (February); Yatai is archived (June)

Sources17

Improve this page on GitHub