cloudparts

Large language models

A neural network trained on huge amounts of text to predict the next token; run in a loop, one token at a time, it turns a prompt into answers, summaries or code.

1 technology on this page ↓

How it works

Text is first cut into tokens: words, pieces of words or bytes from a fixed vocabulary, each mapped to an integer. Gemini’s docs put a token at about 4 characters, so 100 tokens are 60 to 80 English words. The model itself is a set of weights, billions of numbers learned in training: Qwen3-8B has 8.2 billion, stored as BF16 in five .safetensors files totaling about 16.4 GB. Inference means running those weights over the tokens. For the next position the network scores every token in its vocabulary, one token is picked, appended to the input, and the loop repeats until the model emits an end-of-sequence token or reaches the max_tokens limit. Everything the model can take into account in one request is its context window: the system prompt, the conversation so far, tool definitions, pasted documents and its own output, up to a fixed size (1M tokens on current Claude models, 32,768 natively on Qwen3-8B). Nothing carries over between requests, so a chat app resends the whole conversation every turn. The roles are themselves text: a chat template wraps each message in special tokens, such as Qwen3’s <|im_start|>system ... <|im_end|>, and the system prompt is simply the first message, which the model was trained to treat as instructions. Picking the token is called sampling. Greedy decoding always takes the most likely one; sampling draws at random from the scores, and temperature flattens or sharpens them. That draw is why the same prompt returns different answers, and Anthropic notes that even at temperature 0 the results are not fully deterministic.

Think of it as

An autocomplete that has read a library. While it writes, it doesn’t look anything up: it only continues the page in front of it, one word at a time, choosing words that usually come next in text like this. Hand it a page that starts with instructions and a question, and the most likely continuation is an answer. Anything it should know has to be written on that page, and the page has a size limit.

Reach for it when

  • Turning messy text into something else: summaries, translations, drafts, labels, or fields extracted as JSON
  • Answering questions about documents you put in the prompt, where a person or a check can compare the answer with the source
  • Writing or reviewing code, where tests, a type checker and a compiler catch mistakes
  • A natural-language front end to tools and APIs, which is how agents are built

Think twice when

  • The result must be exact and repeatable, such as totals on an invoice or a record lookup: compute it in code, because sampled output can change between identical calls
  • The answer depends on facts the model never saw, such as last week’s orders or a private wiki, unless you put them into the context, which is what retrieval-augmented generation does
  • A regular expression, a lookup table or a small classifier already does the job: every call is billed per token, for example $2 per million input tokens and $10 per million output tokens on Claude Sonnet 5.5
  • Nobody checks the output: models write fluent text that can be confidently wrong

A real example

A support team labels 10,000 incoming tickets a day with one of 12 categories. Each request carries a system prompt of about 400 tokens (the category list and three short examples) plus the ticket, about 500 tokens, and asks for JSON such as {"category": "billing"}, about 30 tokens. That is 9 million input tokens and 0.3 million output tokens a day. On Claude Sonnet 5.5 at $2 and $10 per million, the bill is $18 plus $3, or $21 a day. Because models released after Claude Opus 4.6 accept no temperature other than 1.0, the team doesn’t rely on low temperature for consistency: the code parses the JSON, rejects any category that isn’t on the list, and retries those tickets once. The context window is never a concern here, since each request is under 1,000 of the model’s 1M tokens.

More on large language models

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

What each provider calls it

AWS
Amazon Bedrock: 100+ foundation models from Amazon (Nova), Anthropic, DeepSeek, Moonshot AI, MiniMax, OpenAI, xAI and others, through the Converse, Invoke, Chat Completions and Responses APIs
Google Cloud
Gemini API (keys from Google AI Studio, with an OpenAI-compatible endpoint); Gemini Enterprise Agent Platform, renamed from Vertex AI in April 2026, with Gemini plus partner and open models in Model Garden
Azure
Microsoft Foundry Models: Azure OpenAI models plus models sold by Azure from providers such as Cohere, DeepSeek, Meta, Mistral AI and xAI, and a catalog of partner and community models
Open source & others
Open-weight models such as Qwen3 (Apache 2.0) and Llama 3.1 (Llama 3.1 Community License), downloaded from Hugging Face and run with llama.cpp, Ollama or LM Studio; hosted APIs from model makers, such as Anthropic’s Claude API

Hosted API or open weights#

The same idea ships two ways: as an API in front of weights you never see, or as files you download and run yourself.

Hosted APIOpen weights
Where the weights areOn the provider’s servers; you send text and get text backOn your disk: Qwen3-8B is five safetensors files, about 16.4 GB in BF16
What you pay forTokens, priced per million, with output usually dearer: $2 in and $10 out on Claude Sonnet 5.5Hardware, power and your time; the download is free
Where your prompts goTo the provider (and to any cloud or gateway in between)Nowhere: Ollama’s FAQ says “We don’t see your prompts or data when you run locally”
Size and contextThe largest models, with windows up to 1M tokens on current Claude modelsWhatever fits in your memory; Qwen3-8B handles 32,768 tokens natively
TermsThe provider’s terms of serviceThe license on the weights: Apache 2.0 for Qwen3; the Llama 3.1 Community License requires a separate license from Meta above 700 million monthly active users

Sampling settings#

Greedy decoding, which always takes the most likely token, is the default in Hugging Face Transformers. Its docs say it suits short, input-grounded tasks but “begins to repeat itself” on long outputs. Sampling draws from the whole distribution instead, and three settings shape it: temperature (Transformers’ docs suggest above 0.8 for creative tasks and below 0.4 for tasks that require “thinking”), top_k (only sample from the k most likely tokens) and top_p, or nucleus sampling (rank the tokens by probability and cut the list off once their probabilities add up to p). Both cutoffs remove the “long tail” of unlikely tokens.

Model makers publish the settings their models were tuned for. Qwen3-8B’s card asks for temperature 0.6, top-p 0.95 and top-k 20 in thinking mode and warns not to use greedy decoding, which “can lead to performance degradation and endless repetitions”. Some hosted APIs now fix these values: on models released after Claude Opus 4.6, Anthropic’s Messages API rejects any temperature other than 1.0, any top_k, and any top_p below 0.99.

In code · python

from openai import OpenAI

# Many providers speak the chat completions format; this one is Gemini’s endpoint.
client = OpenAI(
    api_key="GEMINI_API_KEY",
    base_url="https://generativelanguage.googleapis.com/v1beta/openai/",
)

stream = client.chat.completions.create(
    model="gemini-3.8-flash",
    messages=[
        # The system prompt is just the first message the model reads.
        {"role": "system", "content": "Answer in one sentence."},
        {"role": "user", "content": "What is a context window?"},
    ],
    stream=True,                              # tokens arrive as they are sampled
    stream_options={"include_usage": True},  # the last chunk reports token counts
)
for chunk in stream:
    if chunk.choices:
        print(chunk.choices[0].delta.content or "", end="", flush=True)
    if chunk.usage:
        print("\n", chunk.usage.prompt_tokens, "in,", chunk.usage.completion_tokens, "out")

Often confused with

AI coding agentsAI agentsRetrieval-augmented generation (RAG)Vector databasesModel serving and MLOpsJEPA (Joint-Embedding Predictive Architecture)LLMs or JEPA?
Where it came from2017–2026

Language models that guess the next word are old, but the current ones rest on one 2017 paper. In “Attention Is All You Need”, Google researchers (Ashish Vaswani and seven coauthors) replaced the recurrent networks used for translation with the Transformer, built only from attention. It trained faster because it parallelized better: their English-to-French model set a new record after 3.5 days on eight GPUs. OpenAI’s decoder-only GPT models followed. GPT-2, with 1.5 billion parameters in 2019, drew attention because OpenAI at first held it back over fears of misuse. GPT-3, described in May 2020, had 175 billion parameters, 10 times more than any earlier non-sparse model, and showed that one model could do new tasks from a few examples written in the prompt instead of being fine-tuned for each.

A pretrained model only continues documents, which makes it hard to direct. In March 2022, OpenAI researchers fine-tuned GPT-3 on demonstrations and human rankings of its answers (reinforcement learning from human feedback) and found that people preferred a 1.3-billion-parameter InstructGPT over the 175-billion-parameter GPT-3. That recipe produced ChatGPT, released on November 30, 2022, which reached 100 million monthly users two months later.

Until 2023 the strongest models were mostly reachable only through an API. On February 24, 2023, Meta released LLaMA in four sizes from 7 to 65 billion parameters under a noncommercial research license; within days the weights were on BitTorrent, and llama.cpp ran them on a laptop. Llama 2 followed on July 18, 2023, free for research and commercial use, and in January 2025 DeepSeek released the 671-billion-parameter DeepSeek-R1 as open weights. By 2026 the clouds resell many vendors’ models side by side: Amazon Bedrock lists more than 100.

Why it’s built this way

Train on raw text to predict the next token
Any text is training data, with no labels needed, so models could learn from trillions of tokens. Meta trained LLaMA 65B on 1.4 trillion; the cost is a model that only continues documents until it is fine-tuned to follow instructions.
Attention instead of recurrence
Recurrent networks process a sequence one step at a time. The Transformer’s attention looks at all positions at once, which the 2017 paper showed was “more parallelizable” and took “significantly less time to train”, the property that made very large models affordable.
Scale up and instruct in the prompt instead of fine-tuning per task
Fine-tuning needed thousands of labeled examples for every task. The GPT-3 paper found that a large enough model could do many tasks from instructions and a few examples in the prompt, with no gradient updates.
Fine-tune on human feedback
The InstructGPT paper put it plainly: “Making language models bigger does not inherently make them better at following a user’s intent.” Ranking outputs and training toward the preferred ones made a model 100 times smaller win in human evaluations.

Timeline

  1. 2017

    Google researchers publish the Transformer in “Attention Is All You Need” (June 12)

  2. 2019

    OpenAI’s GPT-2 (1.5 billion parameters) is first held back over misuse fears

  3. 2020

    GPT-3 paper: 175 billion parameters and few-shot prompting (May 28)

  4. 2022

    InstructGPT paper on fine-tuning with human feedback (March 4); ChatGPT is released (November 30)

  5. 2023

    Meta releases LLaMA to researchers (February 24) and Llama 2 for commercial use (July 18)

  6. 2025

    DeepSeek releases the 671-billion-parameter DeepSeek-R1 as open weights (January)

  7. 2026

    Google renames Vertex AI to Gemini Enterprise Agent Platform (April 22)

Sources18

Improve this page on GitHub