cloudparts

Propolis

Oxide’s virtual machine monitor: a Rust program that pairs with the bhyve hypervisor in the illumos kernel and emulates the devices each VM in an Oxide rack sees.

What it is
Hypervisors
Used for
Virtual machines
Type
Open source
Made by
Oxide Computer Company
License
MPL 2.0
Facts checked
13 October 2026
Website Source code

What it is

A hypervisor has two halves. The kernel half, here bhyve in illumos, uses the CPU’s virtualization extensions to run guest code directly and to map guest memory. The user-space half, the virtual machine monitor (VMM), emulates everything else: the chipset, serial ports, disks and network cards, plus tasks like live migration. On Linux that half is usually QEMU. Propolis is Oxide’s replacement for it on illumos, written in Rust and talking to bhyve through its ioctl interface.

Propolis emulates a deliberately small set of devices: NVMe and virtio block disks, virtio network cards backed by illumos’s in-kernel viona driver, a UART for the serial console, PS/2 and the PCI and chipset plumbing around them. Guests boot UEFI firmware built from EDK2’s OVMF. A disk can be a local file or a Crucible volume: Propolis links in Crucible’s client, which sends each read, write and flush over the network to three storage servers.

It comes as propolis-server, which runs one VM and is driven over a REST API (in an Oxide rack, by the control plane, with one server per VM in its own illumos zone), propolis-standalone, which runs a VM from a TOML file for quick experiments, and propolis-cli. Live migration of a running VM between hosts is supported, primarily on AMD processors.

Why teams pick it

  • A memory-safe device model: the code that parses everything a possibly hostile guest writes to its devices is Rust, not C
  • A small surface: only the devices modern Linux and Windows guests need, rather than QEMU’s catalog of emulated hardware
  • Built for a control plane: a REST API with an OpenAPI description, metrics through Oximeter, and live migration between sleds
  • Disks can come straight from Crucible’s replicated, encrypted network block storage, with no separate storage target to configure

Watch out for

  • It only runs on illumos with bhyve, and needs recent illumos builds because it is tightly coupled to bhyve’s kernel interface
  • It works best on AMD hosts, where Oxide runs its CI; Intel hosts can run VMs, and live migration may work there
  • Guests must boot through UEFI on x86 and have VirtIO drivers; nested virtualization and a virtual TPM are not offered in the Oxide rack
  • Known issue in release v23: guests treat the TSC clock source as unreliable and fall back to much slower timestamp syscalls, unless you apply the documented workaround

Alternatives

What all hypervisors have in common →Compare side by side →
ACRNDormantOpen sourceA small type-1 hypervisor for Intel-based embedded and industrial systems that runs real-time and safety VMs on dedicated cores next to Linux, Windows or Android guests on the same board.Cloud HypervisorOpen sourceA Rust virtual machine monitor for modern cloud VMs on KVM or Microsoft’s hypervisor: virtio devices only, plus hotplug, VFIO passthrough, live migration, snapshots and Windows guests.HeliosOpen sourceOxide’s distribution of illumos, the operating system on every sled’s host CPU in the Oxide rack, where it runs VMs on its bhyve hypervisor and the control plane in zones.KVM (Kernel-based Virtual Machine)Open sourceThe hypervisor built into the Linux kernel: it runs guest code directly on the CPU through /dev/kvm, and leaves device emulation to a user-space program such as QEMU or Firecracker.libvirtOpen sourceA stable API, daemon and virsh command line for managing virtual machines, storage and networks, with drivers for QEMU/KVM, Xen, LXC, Cloud Hypervisor, VMware ESX, Hyper-V and more.Microsoft Hyper-VCommercialMicrosoft’s type-1 hypervisor, built into Windows Server and Windows 10 and 11 Pro and Enterprise: a thin hypervisor under a Windows root partition, which also underpins Azure’s hypervisor and Windows features like VBS, WSL 2 and Windows Sandbox.Oracle VirtualBoxOpen sourceOracle’s free, open-source desktop hypervisor: a type-2 hypervisor that runs VMs as an application on Windows, macOS, Linux and Solaris, scriptable with VBoxManage and the default provider for Vagrant.Proxmox Virtual EnvironmentOpen sourceA Debian-based server platform from Vienna that runs KVM virtual machines and LXC containers, with clustering, high availability, Ceph and ZFS storage, backups and a web interface in one free ISO.QEMUOpen sourceThe open-source machine emulator and virtualizer: it emulates whole computers in software, or runs them at native speed on a hypervisor such as KVM, and supplies the virtual devices either way.VMware ESXiCommercialVMware’s bare-metal hypervisor (called ESX again since VCF 9), whose own kernel runs VMs directly on servers while vCenter adds vMotion, HA, DRS and vSAN; since Broadcom bought VMware in 2023 it is sold only as per-core subscriptions.Xen Project hypervisorOpen sourceThe open-source type-1 hypervisor from Cambridge that ran the first Amazon EC2 instances: a small microkernel-style core that boots first, with drivers and management in a privileged VM called dom0.

How it fits with other technologies

A `propolis-standalone` config: 4 vCPUs, 1 GiB, an ISO on virtio-block and a viona NIC (`pfexec propolis-standalone vm.toml`) · text

[main]
name = "testvm"
cpus = 4
bootrom = "/path/to/bootrom/OVMF_CODE.fd"
memory = 1024

[block_dev.alpine_iso]
type = "file"
path = "/path/to/alpine-extended-3.12.0-x86_64.iso"

[dev.block0]
driver = "pci-virtio-block"
block_dev = "alpine_iso"
pci-path = "0.4.0"

[dev.net0]
driver = "pci-virtio-viona"
vnic = "vnic_name"
pci-path = "0.5.0"

See it in a real stack

No architecture uses Propolis yet. Drop it into the playground and wire it up yourself.Open playground →
Where it came from2020–2024

Choosing a hypervisor for the Oxide rack meant choosing an operating system for the host, and Oxide’s RFD 26 weighed the two open-source options: KVM on Linux and bhyve on illumos. KVM was the most feature-rich, but its usual partner, QEMU, was ruled out as too complex and too often the subject of reliability and security bugs. bhyve, which began on FreeBSD, required modern CPU virtualization features and skipped nested virtualization, which made its code base much smaller, but its own C userspace had also been a source of bugs.

So Oxide decided to keep bhyve in the kernel and write a new userspace in Rust. The Propolis repository started in October 2020, and at Oxide’s May 2021 milestone demo Propolis booted an Alpine Linux guest on Helios. RFD 26’s determination, as of June 2021, was Helios as the host OS and “Propolis, our Rust-based userspace, and bhyve in the illumos kernel for guest workloads”.

Propolis was open source throughout its development, and shipped in every Oxide rack from 2023. It runs each customer VM, in its own zone, under the control of Omicron’s sled agent.

Why it’s built this way

bhyve rather than KVM
bhyve’s requirement for hardware virtualization and lack of nested virtualization kept it small and manageable. RFD 26 also weighed the host OS by how well a small team could find and fix defects from as few observed incidents as possible.
A new Rust userspace instead of QEMU or bhyve’s C one
Device emulation is where much of a VMM’s attack surface and bugs live. QEMU’s breadth came with reliability and security problems, and bhyve’s C userspace had its own, so Oxide wrote the device models in a memory-safe language.
Live migration as a requirement
RFD 26 calls live migration necessary: moving VMs off a server lets operators balance capacity and update or repair a node without taking guests down.

Timeline

  1. 2020

    Repository started (October)

  2. 2021

    Boots an Alpine Linux guest at Oxide’s May milestone demo; RFD 26 commits to bhyve plus Propolis (June)

  3. 2023

    Ships in the first Oxide racks

  4. 2024

    RFD 26 postscript: Propolis open source throughout, Helios open-sourced in January

Sources9 · facts checked 13 October 2026

Improve this page on GitHub