Documentation sites
Docs written as Markdown files in a repository, built into a searchable website on every merge, with API reference generated from the API’s own description and plain-text copies for AI agents.
1 technology on this page ↓How it works
Docs as code means writing documentation with the same tools as code: plain-text markup (Markdown, reStructuredText or AsciiDoc) in version control, reviewed in pull requests and checked by automated tests, so a feature can be blocked from merging until it is documented. A static site generator turns the files into a website. Sphinx, originally created for Python’s documentation, reads reStructuredText or MyST Markdown and cross-references code objects; Docusaurus and Astro’s Starlight read Markdown and MDX, Markdown that can import and embed JSX components such as tabs, callouts or a live chart. A config file sets the navigation, and CI builds the site on every merge, often with a preview for each pull request and a check for broken links. API reference is generated rather than written: an OpenAPI document, a language-agnostic description of an HTTP API, becomes one page per endpoint with its parameters, schemas and example requests (Redoc, Mintlify and others render it). Search comes from an index built after the site, as with Pagefind, which needs no server, or from a hosted crawler such as Algolia DocSearch. Since 2024 sites also publish for AI agents: an /llms.txt file lists the important pages as Markdown links with one-line notes, and each page is also served as clean Markdown at its URL plus .md.
A cookbook typeset straight from the test kitchen’s notebook. Every recipe change gets the same tasting review as the dish, the book is reprinted automatically after each change, the ingredient tables are printed from the pantry inventory rather than copied by hand, and a plain-text edition goes to the delivery robots.
Reach for it when
- Developer docs for an API, SDK or library that changes with every release
- Docs that engineers and writers both edit, where review in pull requests keeps them accurate
- An HTTP API with an OpenAPI document: generate the reference from it instead of writing it by hand
- Docs that coding agents will read: publish
llms.txtand Markdown copies of the pages
Think twice when
- Contributors who will never use Git, unless the platform has a web editor that commits for them
- Hand-written API reference next to a generated one: the two drift apart, so keep one source
- A small library whose README already says everything; a site adds a build to maintain
A real example
An API team keeps docs/ next to the service code: 40 Markdown pages, a config file for the navigation, and the openapi.yaml the service already publishes. A pull request that adds a refund endpoint changes the spec and adds a guide page. CI builds a preview, and the link checker fails on a mistyped link until it is fixed. After the merge the site redeploys with a new reference page generated from the spec, and the same build rewrites /llms.txt, so a coding agent asked to “add refunds” fetches the Markdown of that page instead of scraping HTML.
Technologies
Specific products, and what each does differently.
Reading the llms.txt above#
The file sits at /llms.txt (or under a subpath such as /docs/llms.txt, covering the pages below it). The H1 is the only required part. A blockquote summary and plain paragraphs follow, then H2 sections that are lists of [name](url) links, each with an optional note after a colon. A section called “Optional” holds links an agent can skip when it needs a shorter context.
The links point at Markdown, not HTML, so an agent reads the index, picks the pages it needs and fetches them without navigation, scripts or ads. Some platforms also write /llms-full.txt with every page’s text in one file.
Common tools#
Generators, hosts and the pieces they plug together.
Sphinx
generator, PythonCreated for Python’s own docs, first released in March 2008. Reads reStructuredText or MyST Markdown; generates API docs from Python, C++ and other code; outputs HTML, PDF via LaTeX and ePub.
Docusaurus
generator, ReactBuilt by Facebook’s open source team in 2017 so projects could write Markdown instead of building websites. Now MDX-based, with versioning, i18n and search.
Starlight
generator, AstroA documentation theme for the Astro framework: Markdown, MDX and Markdoc pages in src/content/docs/, with built-in components such as cards and tabs.
Read the Docs
hostingHosts docs built by Sphinx and other tools, built from your Git repository, with pull request previews; started to remove the barrier of hosting documentation.
Mintlify
hosted platformMDX pages and a docs.json in your repository, deployed on push, with an OpenAPI playground, llms.txt, Markdown copies and an MCP server generated for you.
Redoc
API referenceTurns an OpenAPI description into a three-panel reference page: navigation, documentation, and request and response examples.
Pagefind
static searchRuns after any static site generator, indexes the built HTML and writes a static search bundle, so search needs no server.
Algolia DocSearch
hosted searchA crawler extracts docs into an Algolia index and a frontend package searches it; free for eligible documentation sites.
In code · text
# Acme Parcels API
> REST API for creating and tracking parcels. Every request needs an API key in the `Authorization` header.
Amounts are in cents. Webhooks are signed with HMAC-SHA256.
## Docs
- [Quickstart](https://docs.acme.example/quickstart.md): create a key and send a first parcel
- [Webhooks](https://docs.acme.example/webhooks.md): parcel status events and how to verify signatures
## API reference
- [OpenAPI document](https://docs.acme.example/openapi.yaml): every endpoint, schema and error code
## Optional
- [Changelog](https://docs.acme.example/changelog.md)Often confused with
Where it came from2004–2026
The plain-text half came first. John Gruber’s Markdown, at version 1.0.1 in December 2004, let writers use an easy-to-read text format and convert it to valid HTML. Python’s documentation went through its own toolchain, built by Fred L. Drake, Jr., and then Sphinx, first released in March 2008 and written for Python’s docs, which turned reStructuredText into cross-referenced HTML. Read the Docs followed, to make hosting such sites simple and give documentation one place to be found.
In the 2010s the practice got a name. Write the Docs talks from 2015 onwards described teams, Google’s among them, treating docs like code: Git, plain text, code review and automated tests. API reference became generated too: the Swagger specification (2011) was donated to the OpenAPI Initiative at the end of 2015 and became OpenAPI 3.0 in 2017, a machine-readable description that tools render as reference pages. In December 2017 Facebook released Docusaurus because each of its open source projects had built its own website, and it wanted teams to write Markdown instead.
Then the readers changed. In September 2024 Jeremy Howard of Answer.AI proposed /llms.txt, a Markdown index of a site for language models, because HTML wraps the facts in navigation, ads and JavaScript and whole sites don’t fit in a context window. By the second version of the proposal in August 2026, thousands of sites published one, documentation platforms generated it automatically, and Chrome’s Lighthouse checked for it.
Why it’s built this way
- Plain text in the code repository
- Writers work in the same workflow as developers, developers draft docs for their own features, and a merge can be blocked until the docs exist.
- Generate reference from a machine-readable description
- An OpenAPI document lets people and tools understand an API without its source code, so the reference pages and the interactive playground can be rebuilt from the same file the API is described in.
- Markdown with components (MDX)
- Markdown is more natural to type than HTML for headings and emphasis; MDX keeps that and lets authors import components, such as interactive charts or alerts, and embed them in long-form content.
- A second, Markdown copy for agents
- Turning HTML pages built for people back into clean text is difficult and imprecise, and every wasted token costs time and money, so
llms.txtand per-page.mdfiles give them a concise path to the same content.
Timeline
- 2004
Markdown 1.0.1 released on 17 December
- 2008
Sphinx 0.1, written for the Python documentation, in March
- 2011
First release of the Swagger Specification in August
- 2015
Swagger 2.0 donated to the OpenAPI Initiative on 31 December
- 2017
OpenAPI 3.0 in July; Facebook introduces Docusaurus in December
- 2024
Jeremy Howard proposes /llms.txt on 3 September
- 2026
llms.txt v2: thousands of sites, generated by docs platforms, checked by Lighthouse
- Daring Fireball: Markdown (version 1.0.1, 17 December 2004)
- Python docs: About this documentation (Sphinx originally created for Python; Fred L. Drake, Jr.)
- Sphinx changelog: Release 0.1 (March 2008)
- Read the Docs: The story of Read the Docs
- Write the Docs: Docs as Code (talks from 2015 on)
- OpenAPI Specification 3.1.0, Appendix A: Revision History
- Docusaurus blog: Introducing Docusaurus (14 December 2017)
- Jeremy Howard: The /llms.txt file, v2 (first published 3 September 2024, modified 10 August 2026)
- MDX docs: What is MDX?
Sources14
- Write the Docs: Docs as Code
- Sphinx documentation (reStructuredText or MyST, cross-references, API docs, HTML, LaTeX and ePub output)
- Python docs: About this documentation
- MDX docs: What is MDX?
- Docusaurus docs: Introduction (MDX, versioning, i18n, search)
- Starlight docs: Getting started (Astro, Markdown, MDX and Markdoc)
- OpenAPI Specification 3.1.0 (what the OAS defines)
- Redocly docs: Redoc CE (reference from an OpenAPI description, three-panel layout)
- Pagefind docs: Getting started (runs after the static generator, static search bundle)
- Algolia: What is DocSearch? (crawler, frontend packages, free for eligible sites)
- Read the Docs: The story of Read the Docs
- Jeremy Howard: The /llms.txt file, v2 (format, Optional section, .md copies of pages)
- Mintlify docs: llms.txt (llms.txt and llms-full.txt generated automatically)
- Mintlify docs: OpenAPI setup