OPTE (Oxide Packet Transformation Engine)

OPTE runs on every sled, in the illumos kernel, between each VM’s virtual NIC and the physical network. Every packet a VM sends goes into OPTE first. Each virtual NIC port has a series of layers that handle firewalling, routing, NAT and encapsulation. A layer either looks up state that already exists (a routing rule) or creates state (a firewall connection), and adds a transformation to a list. When every layer has had its say, OPTE applies the list: rewrite the addresses for NAT, say, and wrap the packet in a Geneve header addressed to the sled hosting the other VM. Packets coming in go through the same layers in reverse.

OPTE works off a connection-oriented flow table, as Microsoft’s VFP design recommends, because firewalls and NAT care about connections rather than single packets. Between sleds, encapsulated packets travel over the rack’s IPv6 underlay, routed by Oxide’s delay-driven multipath protocol, through the two rack switches, which decapsulate traffic leaving the rack. The project is split into a generic engine (opte), the Oxide VPC built on it (oxide-vpc), the xde kernel module that runs it on Helios, and opteadm, an operator tool for inspecting ports and rules; DTrace scripts trace it at run time.

Customers never touch OPTE directly. They see its effects through the Oxide API: VPCs with private RFC 1918 subnets, firewall rules, custom routers, internet gateways, outbound NAT, and ephemeral and floating external IPs.