Omicron

Omicron is everything in the Oxide rack that decides rather than does. Its center is Nexus, one program that hosts every user-facing API (for users and operators), the web console and the internal APIs other components report to. Nexus keeps no state of its own: all configuration, inventory and runtime state lives in a replicated CockroachDB cluster, and Nexus is the only component that talks to it. So any number of Nexus copies can run, coordinating only through the database. A change follows four steps: Nexus receives the request, validates it, writes it to CockroachDB (the point where it is ordered against concurrent changes), and then propagates it to the components that need to know.

On every sled a sled agent does the local work. To start a VM it creates an illumos zone for a Propolis server and sets up the VM’s OPTE network ports; Propolis then connects the VM’s disks to Crucible. Other services run as zones that can be placed on any sled and multiplied for availability: Oximeter collects metrics into ClickHouse, internal DNS lets components find each other, external DNS answers for the API and console names, and Crucible Pantry handles detached volumes. On the two sleds wired to the switches, Dendrite programs the switch and the Management Gateway Service talks to every service processor. All Oxide-written services are in Rust and talk HTTP with OpenAPI specs generated by Dropshot.

Multi-step operations, such as creating an instance with its disks and IP addresses, run as distributed sagas whose log records which steps have run, and periodic background tasks re-check the state of the world and reconcile it, rather than retrying single calls. The design goals are to stay available through any two hardware or software failures and to support on the order of 100 servers and 10,000 VMs per rack. Omicron also has a simulated mode, with a fake sled agent, that runs on Linux and macOS for development; the real one runs only on Helios.