Architecture

The substrate#

Every sandbox, runner, and simulator Sandal runs comes from the same substrate: bare-metal machines (Hetzner, OVH, Leaseweb, plus dedicated Mac minis for iOS/macOS work) running microVMs. No shared kernels, no noisy-neighbor containers pretending to be isolated. Each guest gets its own kernel.

$ npx sandal init

We use two hypervisors depending on the job:

• Firecracker for sandboxes and the code interpreter, sub-second cold boot, minimal device model, built for exactly this: many small, short-lived, hostile-tenant VMs per host.
• Cloud Hypervisor for bigger workloads, GitHub Actions runners, iOS/Android simulator hosts, anything that wants more devices (GPU passthrough, larger disks, USB/ADB tunnels) than Firecracker's minimal model exposes.

Different doors, same substrate: the scheduler, the image format, the network fencing, and the billing meter are shared across both. A runs-on: sandal-ubuntu-24.04 runner and a sandal.sandboxes.create() call are provisioned by the same control plane, just routed to different VMM backends.

Host agent + privileged split#

Every physical host runs a small host agent as the only privileged process. The host agent is the sole thing on the box permitted to:

• launch or tear down a VMM process,
• attach a TAP device / network namespace to a guest,
• mount or unmount a generation's block device,
• rotate or destroy a per-workload encryption key.

Nothing else on the host, not the VMM, not the guest, not the API layer, holds those privileges. Guest-facing agents (guestd, see below) run unprivileged inside the microVM and talk to the host agent over a narrow, versioned control socket. That socket is the entire trust boundary between "code we didn't write, running on a stranger's behalf" and "the thing that can touch other tenants' data."

Signed, fenced commands#

Nothing reaches a guest, and nothing a guest does reaches the host, without going through signed, fenced command paths:

• Every control-plane instruction to a host (create, pause, fork, resume, destroy) is a signed, single-use command tied to a specific VM identity. Replaying an old command against a rebuilt VM with the same name fails closed.
• Every guest-originated action that needs a privileged host operation (expose a port, request a snapshot, extend a lease) is fenced: checked against the workload's live capability grant before the host agent will act on it, not just authenticated at connection time.
• Commands carry a monotonic fence token per VM generation. A command stamped for generation N is rejected once the VM has moved to generation N+1, this is what stops a paused-and-forked sandbox from racing its own fork over stale credentials.

This is the same pattern distributed storage systems use to stop a zombie primary from writing after failover, applied to VM lifecycle instead of disk I/O.

Warm pools#

Cold boot on Firecracker is fast, under a second, but "fast" still isn't "instant," and agents don't wait well. So most workloads are served from warm pools: pre-booted, pre-attached microVMs sitting at a snapshot boundary, ready to be claimed.

• Claiming a warm instance means handing it a fresh identity, a fresh network fence, and a fresh set of credentials, then resuming, targeting under 200ms from request to a runnable guest.
• Pools are templated per image (node-24, sandal-ubuntu-24.04, ios-simulator, etc.) and kept at a configured minWarm per pool, replenished as they're drained.
• Runner pools use the same warm-pool machinery under a different name: sandal.runners.pools.create({ os: "macos", labels: ["sandal-macos"], minWarm: 2 }) keeps N macOS runners hot so a JIT-registered job doesn't sit in a GitHub queue waiting on a cold Mac mini.

Paused sandboxes cost nothing while idle, a paused generation is just bytes at rest, not a running VM, so pools can afford to stay generously sized without turning into a standing compute bill.

Generations: encrypted, content-addressed, durable#

A sandbox's on-disk state is never "the VM's disk" as a single mutable blob. It's a chain of generations, content-addressed, per-minute incremental snapshots, each one encrypted with a key scoped to that workload.

• Every generation is a V4 image (RAFS-formatted), addressed by the hash of its content, so identical layers across generations and across tenants are never stored twice.
• Generations write incrementally: a paused sandbox that's been running for an hour isn't re-uploading an hour of disk, it's uploading the last minute's delta.
• S3 is the durability floor. Every generation lands in object storage as the source of truth, if a host dies, the data isn't gone, because nothing durable ever lived only on that host's local disk.
• Local NVMe is the accelerator, not the store. Hosts keep a hot cache of recently used generations on local NVMe so resume and fork are fast, but that cache is disposable, it's rebuilt from S3 on demand, never the only copy of anything.
• Resume and fork both use lazy restore with prefetch playbooks: the guest starts running before the whole image is local, with a playbook of "the files this workload touches first" pulled ahead of the fault, and the rest streamed in behind it on access.

Forking a sandbox is a generation-graph operation, not a disk copy: const child = await sb.fork() creates a new content-addressed generation that shares every block with its parent until one of them writes.

guestd and the held-stream transport#

Inside every guest, an unprivileged agent (guestd) is the only thing the host agent talks to. It's the process that turns sb.exec("pnpm test"), sb.fs.write(...), and sb.expose(3000) into actual syscalls inside the VM.

The transport between host agent and guestd is a held-stream design: rather than opening a new connection per call (slow, and a new thing to authenticate every time), the control channel is a single long-lived stream held open for the life of the VM, multiplexing exec, filesystem, and port-forward traffic over it. Commands are framed and fenced individually (see above) even though they share a transport, so a long-lived stream never becomes a long-lived trust grant, each frame is checked on its own.

One substrate, many doors#

The product surface looks like four different things, sandboxes, GitHub Actions runners, iOS/Android simulators, a code interpreter, but underneath there's one substrate exposed through different doors:

Same scheduler, same generation format, same host agent, same fencing. New doors get added by pointing a new SDK surface at the existing substrate, not by standing up new infrastructure.

Evidence-gated capability matrix#

Every number on this site, cold create time, warm claim latency, per-minute price, throughput, comes from a qualification run, not a spec sheet. We publish a capability matrix that only advances a claim once it's been produced by an actual measured run against the actual substrate, on the record, re-run as the substrate changes. If a number in the docs looks stale or the qualification run that produced it isn't linked, that's a bug, file it.

Regions#

Sandal runs in EU and US regions on the same substrate and control-plane design. SOC2 is in progress; see Security for the threat model and current compliance status.