cloudparts

Ray Serve

Ray’s library for online inference: Python classes become deployments whose replicas run as Ray actors across a cluster, with request batching, model composition, autoscaling to zero and an OpenAI-compatible LLM server built on vLLM.

PyTorch Foundation project
What it is
Model serving and MLOps
Type
Open source
Made by
Anyscale (Ray is a PyTorch Foundation project)
Foundation
PyTorch Foundation, joined October 2025. Part of Ray. Part of the Linux Foundation
License
Apache 2.0
Facts checked
23 October 2026
Website Source code

What it is

A Ray Serve app is ordinary Python. The @serve.deployment decorator turns a class or function into a deployment, and each deployment runs as one or more replicas, copies of the class in separate Ray actor processes that can sit on any machine in the Ray cluster. Replicas can ask for CPUs, GPUs or a fraction of a GPU, so several small models can share one card. The top-level, or ingress, deployment receives HTTP requests, directly or through a FastAPI app attached with @serve.ingress, and calls other deployments through DeploymentHandles, which look like method calls but run on other replicas. That is how a pipeline of preprocessing, several models and business logic becomes one application, with each step scaled on its own hardware. It works with any framework, from PyTorch to scikit-learn, because it serves Python code rather than a model file.

@serve.batch collects concurrent requests into one call, by default up to 10 requests within 10 ms. Autoscaling counts requests rather than CPU: with num_replicas="auto" Serve aims for 2 ongoing requests per replica, caps each at 5, and scales between 1 and 100 replicas; min_replicas: 0 scales an idle deployment to zero. When there is no room for a new replica, the Ray autoscaler asks the cloud for another node, and on the way down Serve packs replicas so whole nodes can be released. Ray Serve LLM, added in Ray 2.44, wraps vLLM (or SGLang) in Serve deployments and exposes an OpenAI-compatible API, with multi-node tensor and pipeline parallelism, prefix-aware routing, separate scaling of prefill and decode, and multiple LoRA adapters on one base model.

On Kubernetes, Ray Serve runs through KubeRay’s RayService resource, which holds a Ray cluster spec and the Serve config together. KubeRay applies config-only changes in place and, for cluster changes, starts a new Ray cluster, switches the Kubernetes service to it once it is ready, and then deletes the old one. Ray was developed at UC Berkeley and by Anyscale, and joined the PyTorch Foundation in October 2025. The latest release is Ray 2.59, from October 2026.

Why teams pick it

  • Model composition in plain Python: preprocessing, several models and business logic in one app, each part with its own replicas, resources and autoscaling
  • Fractional GPUs and multi-node placement, so small models share a GPU and large ones span several machines
  • Request-based autoscaling down to zero, tied to the Ray autoscaler so the cluster adds and removes machines with demand
  • The same library and cluster serve classic models and LLMs, with an OpenAI-compatible server around vLLM

Watch out for

  • You run a Ray cluster: a head node, workers and, on Kubernetes, the KubeRay operator, which is more to operate than a container behind a load balancer
  • Not a full ML platform: Ray’s own docs say it lacks model lifecycle management, so tracking and the registry come from tools like MLflow
  • No model-specific optimizations of its own; speed comes from what runs inside a replica, such as vLLM, ONNX Runtime or a compiled PyTorch model
  • A cluster-level change on Kubernetes starts a second Ray cluster, so the Kubernetes cluster needs room for both during the switch

Alternatives

What all model serving and mlops have in common →Compare side by side →
BentoMLOpen sourceA Python library that turns model inference code into an HTTP service with adaptive batching, then packages code, models and dependencies into a versioned “Bento” and a Docker image.+1 Yatai

How it fits with other technologies

Ray Serve LLM quickstart: an OpenAI-compatible endpoint for an open model · python

from ray import serve
from ray.serve.llm import LLMConfig, build_openai_app

llm_config = LLMConfig(
    model_loading_config={
        "model_id": "qwen-0.5b",
        "model_source": "Qwen/Qwen2.5-0.5B-Instruct",
    },
    deployment_config={
        "autoscaling_config": {
            "min_replicas": 1,
            "max_replicas": 2,
        }
    },
    # Pass the desired accelerator type (e.g. A10G, L4, etc.)
    accelerator_type="A10G",
    # You can customize the engine arguments (e.g. vLLM engine kwargs)
    engine_kwargs={
        "tensor_parallel_size": 2,
    },
)

app = build_openai_app({"llm_configs": [llm_config]})
serve.run(app, blocking=True)
# Any OpenAI client can now call http://localhost:8000/v1

See it in a real stack

No architecture uses Ray Serve yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2020–2026

Ray was developed at UC Berkeley, where the RISELab had earlier built Clipper, a research model server with batching and caching. In a July 2020 post, Ray Serve engineer Simon Mo described the gap Ray Serve was released to fill. Wrapping a model in Flask is simple but becomes unmanageable with many models, since each model needs its own replicas, routing and hundreds of megabytes of weights in memory. Specialized servers such as TensorFlow Serving and TorchServe scale, but expose a “tensor-in, tensor-out” API, so request handling stays in a separate web service and teams end up running two systems.

Ray 1.4, in June 2021, rebuilt Serve around deployments, replacing the earlier “backends”, and added @serve.batch and FastAPI integration. Ray 2.0, in August 2022, added a deployment graph API for composition, a Kubernetes-native way to deploy Serve through KubeRay’s RayService, survival of Ray GCS failures, and scale to zero, with autoscaling marked stable.

As LLMs took over inference workloads, Anyscale moved its separate RayLLM project into Ray itself: Ray 2.44, released in March 2025, added ray.serve.llm with vLLM integration and an OpenAI-compatible API. In October 2025 Anyscale contributed Ray to the PyTorch Foundation, which also hosts vLLM and DeepSpeed.

Why it’s built this way

Serve Python code, not a model format
Real requests need parsing, preprocessing and business logic around the model. Keeping that code in the same deployment as the model avoids the split between a web server and a “tensor-in, tensor-out” model server.
Replicas as Ray actors
Ray already scheduled stateful processes with CPU, GPU and fractional-GPU requests across a cluster. Building replicas on actors let Serve reuse Ray’s scheduler for placement and multi-node scaling instead of writing its own.
Autoscale on requests in flight
Model calls are slow and vary in length, so the number of requests waiting on each replica is a direct measure of load. Counting ongoing requests per replica reacts to that queue, and tying it to the Ray autoscaler adds machines only when replicas need them.

Timeline

  1. 2020

    Ray Serve is released; “Machine Learning Serving is Broken” (July); Ray 1.0 (September)

  2. 2021

    Ray 1.4: deployments API, @serve.batch, FastAPI ingress (June)

  3. 2022

    Ray 2.0: deployment graphs, RayService on Kubernetes, scale to zero (August)

  4. 2025

    Ray 2.44 adds Ray Serve LLM (March); Ray joins the PyTorch Foundation (October)

  5. 2026

    Ray 2.59 (October)

Sources9 · facts checked 23 October 2026

Improve this page on GitHub