Cloud Parts

CI/CD pipelines

An automated path from a commit to production.

4 technologies on this page ↓
Live model
What you’re watchingEach push is built and tested. About one in five fails its tests and goes back to the author; the rest move on through staging to production.

How it works

Continuous integration builds and tests every change as soon as it’s pushed, so problems show up within minutes. Continuous delivery then packages the tested build and promotes it through environments, usually automatically to staging and with checks or an approval before production. The pipeline itself is defined in code next to the application.

Think of it as

An assembly line with inspection stations. Every part is checked at each station, and faulty ones come off the line before they reach a customer.

Reach for it when

  • Any codebase with more than one contributor
  • Frequent, small, low-risk releases
  • Enforcing tests, linting and security scans before merge

Think twice when

  • Pipelines so slow that people route around them; keep feedback fast
  • Skipping tests to ship sooner; a pipeline is only as good as its checks

A real example

A push triggers a build and 900 tests that finish in 6 minutes. When a test fails, the author is notified and the change stops there. Green builds deploy to staging, then to production.

More on ci/cd pipelines

Narrower ideas that build on this one.

Technologies

Specific products, and what each does differently.

Also used for ci/cd pipelines

Products of other kinds that are often used this way, grouped by what they are.

What each provider calls it

AWS
CodePipeline, CodeBuild
Google Cloud
Cloud Build, Cloud Deploy
Azure
Azure Pipelines
Open source & others
GitHub Actions, GitLab CI, Jenkins, Argo CD

In code · yaml

# .github/workflows/ci.yml
on: [push]
jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest
  deploy:
    needs: test                     # only runs if tests pass
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - run: ./deploy.sh production

Worth exploring

WebsitePlatform tooling landscapePlatformEngineering.orgA filterable map of tools for building an internal developer platform. Each tool sits in one of five “planes” (developer control, integration and delivery, resource, observability, security) and a category such as IaC, CD pipeline, secrets or developer portal, with a one-line description. Vendors can submit their own tools, so treat it as a list of what exists rather than a ranking.
WebsitePlatformEngineering.orgThe Platform Engineering communityA community hub for teams that build internal developer platforms: a blog, a Slack workspace, the PlatformCon conference, courses, and a map of platform tooling.
All resources →

Often confused with

Infrastructure as codeContainersObservabilityLocal cloud emulatorsVersion controlBuild toolsBranching strategiesRolling deploymentsBlue-green deploymentsCanary releases
Where it came from2000–2019

For decades, programmers worked apart for weeks or months and then tried to merge everything before a release. Martin Fowler recalls a large electronics company in the 1980s where programmers had been integrating a project for months with no end date in sight. Long-lived branches drifted apart, and releases needed code freezes and whole stabilisation phases. Microsoft was known for daily builds, usually overnight, but without the testing regimen or the focus on fixing defects that continuous integration would demand.

Kent Beck defined continuous integration in the 1990s as part of Extreme Programming, following its rule that if something is painful you should do it more often: everyone merges into the mainline at least daily, and an automated build and test suite checks every merge. Fowler’s 2000 article with Matthew Foemmel, drawing on ThoughtWorks’ Atlas project, spread the practice. ThoughtWorks open-sourced CruiseControl, an early CI server, and Kohsuke Kawaguchi’s Hudson, renamed Jenkins in 2011, followed.

In 2010 Jez Humble and David Farley’s book Continuous Delivery carried the idea through to production: every change flows through one deployment pipeline that builds, tests and can release it, so deployments become “predictable, routine affairs”. Pipelines then moved into the repository as code, from Jenkins 2.0’s Pipeline in 2016 to hosted YAML workflows like GitHub Actions in 2019.

Why it’s built this way

Merge into one mainline at least daily
Branches that live for weeks diverge, and every extra branch multiplies the testing needed to merge it. Small, frequent merges keep conflicts small and the software always in a working state.
Every change is built and tested automatically
A build that only compiles, like the old nightly builds, says little. Running the tests on each change, and fixing a red build before anything else, is what makes the mainline trustworthy.
One deployment pipeline all the way to production
If every change goes through the same automated stages, releasing becomes a routine, low-risk step that can happen on demand, instead of a rare event that needs its own project.
Pipelines defined as code in the repository
Jobs set up through a CI server’s web interface lived outside version control. A Jenkinsfile or workflow YAML lives next to the code, changes with it and goes through the same review.

Timeline

  1. 2000

    Martin Fowler and Matthew Foemmel publish “Continuous Integration” in September

  2. 2010

    Humble and Farley publish Continuous Delivery

  3. 2011

    Hudson’s community renames the project Jenkins

  4. 2016

    Jenkins 2.0 adds Pipeline, pipelines as code

  5. 2019

    GitHub Actions adds CI/CD and becomes generally available

Sources10

Improve this page on GitHub