cloudparts

Model routers and gateways

A proxy between your app and many model providers: one API and one key in front of OpenAI, Anthropic, Google and others, which routes each request, falls back when a provider fails, and caches, rate-limits, logs and bills the traffic.

3 technologies on this page ↓
Live model
What you’re watchingRequests from the app reach the gateway, which answers identical repeats from its cache (HIT) and forwards the rest to a router that spreads them over two providers. When provider A goes down, the router sends everything to provider B until A recovers.

How it works

Your app sends its model requests to the gateway instead of to each provider, usually in OpenAI’s chat completions format: OpenRouter at https://openrouter.ai/api/v1/chat/completions, Cloudflare AI Gateway by swapping the provider’s base URL for https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway_id}/openai, or a self-hosted LiteLLM proxy on your own network. The gateway checks the caller’s key, picks an upstream, translates the request if that provider speaks another format, forwards it and streams the answer back. Routing decides the upstream. OpenRouter, by default, prefers providers of the requested model that had no significant outages in the last 30 seconds, then picks among the cheapest of those, weighted by the inverse square of their price; provider.order or sort override that. Fallbacks handle failure: OpenRouter’s models list is tried in order when a model returns an error, including rate limits, downtime, moderation refusals and context-length errors, and the request is billed at the model that answered. Cloudflare’s dynamic routes add conditions, percentage splits and per-key rate or budget limits that switch to a fallback model. Caching skips the provider entirely: Cloudflare hashes the provider, endpoint, model, auth header and full request body with SHA-256 and serves an identical request from cache (cf-aig-cache-status: HIT). Rate limits return 429 after, for example, 100 requests per 60 seconds; Azure API Management counts tokens per minute instead. Every request can be logged with its prompt, response, provider, token usage, cost and duration. Billing goes one of two ways: you bring your own provider keys, or you buy credits from the gateway, which passes provider prices through and charges a fee on the credits (5% at Cloudflare, 5.5% on OpenRouter’s Standard plan).

Think of it as

A travel agent instead of booking each airline yourself. You give one agent your plans and one card. They know every airline’s booking system, rebook you on another carrier when a flight is canceled, hand you the same ticket again if you ask for exactly the same trip, keep a record of every booking, and send one invoice with their fee on top.

Reach for it when

  • One product uses models from several providers, and you want one API, one key and one bill
  • Uptime matters more than which provider answers: fallbacks move a request to another provider or model when one returns errors or rate limits
  • You need per-team or per-user limits and budgets, and one log of prompts, tokens and cost across providers
  • Many requests are identical, such as a support bot with fixed menu prompts, so an exact-match cache answers them without a provider call

Think twice when

  • Prompts must not pass through another company: a hosted gateway sees and, by default, may log every prompt and response, so turn logging off, restrict routing to zero-data-retention providers, or self-host LiteLLM
  • You depend on one provider’s newest or most specific options: a common format can lag or reject them, for example OpenRouter’s Anthropic-style fallbacks entries accept only a model name and return a 400 for per-attempt max_tokens or thinking
  • You spend heavily with one provider: a 5% to 5.5% fee on credits is real money at scale, where a direct contract or your own keys cost less
  • Every prompt is different: exact-match caching only helps identical requests, and any change to the messages, tools or parameters is a miss

A real example

OpenRouter’s docs walk through a request for one open-weight model that three providers host at $1, $2 and $3 per million tokens, where provider B has had a few recent outages. The router puts B last and weights the other two by the inverse square of their price: A gets 1/1² = 1 and C gets 1/3² = 1/9, so A is tried first 9 times as often as C. If A fails, C is tried next, and B only after that. A team using that model adds a models list with a second model behind it. When every provider of the first model fails or rate-limits, OpenRouter sends the same messages to the second, and the response’s model field shows which one answered and was billed. To keep prompts away from providers that store data, the team also sets provider.zdr so only zero-data-retention endpoints are used, and pays with credits, which carry a 5.5% platform fee on the Standard plan while token prices pass through unchanged.

Technologies

Specific products, and what each does differently.

Also used for model routers and gateways

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
No multi-provider gateway service: Amazon Bedrock intelligent prompt routing picks between models of one family per request, and cross-Region inference profiles spread requests across AWS Regions
Google Cloud
Apigee API Management with LLM token policies (PromptTokenLimit, LLMTokenQuota) in front of model APIs
Azure
AI gateway in Azure API Management: token-per-minute limits per consumer, semantic caching, load balancing with circuit breakers, and a unified OpenAI-compatible model API (preview) for Foundry and other providers such as Amazon Bedrock
Open source & others
LiteLLM AI Gateway (self-hosted, one OpenAI-compatible endpoint for 100+ providers, virtual keys and budgets); hosted: OpenRouter, Cloudflare AI Gateway

What a gateway does to each request#

The same features show up in hosted and self-hosted gateways, under different names.

FeatureWhat it doesOpenRouterCloudflare AI GatewayLiteLLM (self-hosted)
One APIAccepts one format and translates to each providerOpenAI-style chat completions, plus an Anthropic Messages endpointProvider URLs behind one gateway, plus OpenAI-compatible endpointsOne OpenAI-compatible endpoint for 100+ providers
Routing and fallbackChooses an upstream and retries elsewhere on failurePrice-weighted load balancing; models fallback listDynamic routes with conditions, percentage splits and fallbacksLoad balancing, fallbacks, retries and cooldowns
CachingAnswers repeats without calling the providerUses providers’ prompt caching, with sticky routing to the same provider to keep cache hitsExact-match cache, off by default, per-request TTL headersCaching
LimitsCaps requests, tokens or spend per callermax_price per requestRequests per window (fixed or sliding); rate and budget nodes in routesBudgets and rate limits per virtual key
Logs and billingRecords cost and usage; one invoice5.5% fee on credits (Standard); no markup on tokensLogs with prompt, response, tokens and cost; 5% fee on Unified Billing creditsSpend per key, team and user; you pay providers directly

Exact-match or semantic caching#

A model call is expensive and slow compared with a cache lookup, but a cache only helps if requests repeat. Cloudflare AI Gateway uses an exact-match key: a SHA-256 hash of the provider, endpoint, model, the provider auth header and the full request body. Any difference in the messages, tools or parameters makes a new entry, so Cloudflare suggests it for apps with limited prompt options, such as a support bot that offers a fixed menu of questions. Caching is off by default and set per gateway, with per-request headers to change the TTL or the key.

Azure API Management offers semantic caching instead: it stores completions with an embedding of the prompt in a Redis-compatible vector store and reuses a cached completion when a new prompt is close enough. That raises hit rates on prompts that are worded differently but mean the same, at the risk of returning an answer to a question that only looked similar.

In code · bash

# One endpoint and one key for every provider OpenRouter routes to
curl https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "models": ["~anthropic/claude-sonnet-latest", "gryphe/mythomax-l2-13b"],
    "provider": { "sort": "price", "zdr": true },
    "messages": [{"role": "user", "content": "Summarize this ticket in one line: ..."}]
  }'

# models: tried in order if the first one errors, is rate-limited or refuses
# provider.sort: cheapest provider first, instead of price-weighted load balancing
# provider.zdr: only providers with zero data retention
# The response's "model" field says which model answered (and was billed).

Often confused with

Large language modelsAPI gatewaysLoad balancersCachingObservability
Where it came from2023–2026

The first generation of LLM apps called one provider’s API directly, with that provider’s SDK and key. As apps started to mix providers, every team rebuilt the same plumbing: retries, fallbacks, cost tracking and caching. When Cloudflare announced the beta of AI Gateway on September 27, 2023, it noted that basic usage analytics were “surprisingly difficult” to get from the typical model providers. Its gateway sat between the app and APIs such as OpenAI, Hugging Face and Replicate, needed only a changed base URL, and could cache, rate-limit, retry and fall back, for example from GPT-3 on OpenAI to GPT-2 on Hugging Face.

OpenRouter, founded in 2023 by Alex Atallah and Louis Vichy in New York, took the marketplace route: one API and one bill for models from many developers and inference providers, with routing by price and performance. It grew to more than 500 models from over 80 providers and raised a $113 million Series B in May 2026 at a valuation of about $1.3 billion; in August 2026, Bloomberg and The Wall Street Journal reported that Stripe had agreed to acquire it for more than $7 billion.

The same features moved into self-hosted and enterprise tools. LiteLLM offers an open-source gateway with virtual keys, budgets and spend tracking, and Azure API Management added an AI gateway that counts tokens per consumer and caches by meaning. By 2026 Cloudflare’s gateway also sold models from several providers on one bill, with a 5% fee on credits and provider prices passed through.

Why it’s built this way

Copy the provider’s API instead of inventing one
If the gateway accepts the same requests as OpenAI’s API, an existing app switches by changing its base URL. Cloudflare’s 2023 launch post said it took one line of code and left the payload untouched.
Pass token prices through and charge on credits
Both OpenRouter and Cloudflare charge the provider’s own per-token price with no markup and take a fee when you buy credits, so comparing models stays simple and the gateway’s cost is one visible percentage.
Route by health and price, not a fixed provider
The same open model is often served by several providers at different prices and with different uptime. OpenRouter puts providers with recent outages last and weights the rest by the inverse square of their price, which favors cheap providers while keeping others warm as backups.
Exact-match cache keys
Hashing the whole request, including the auth header, means a cached answer is only reused for exactly the same request with the same credentials, never for a near match or another key. Cloudflare planned semantic caching to raise hit rates; Azure API Management ships it, trading exactness for more hits.

Timeline

  1. 2023

    OpenRouter is founded in New York; Cloudflare announces the AI Gateway beta (September 27)

  2. 2025

    OpenRouter raises a seed round led by Andreessen Horowitz and a Series A led by Menlo Ventures

  3. 2026

    OpenRouter raises a $113 million Series B (May); Stripe is reported to be acquiring OpenRouter (August)

Sources19

Improve this page on GitHub