Architecture

ScaiBunker is a thin product layer over ScaiGrid's existing primitives — tenancy, quotas, audit, S3-backed storage — wrapped around a fleet of worker nodes that run Firecracker microVMs. The controller never executes code itself; it schedules work onto workers and relays bytes between workers and object storage.

Components#

Three tiers: the ScaiGrid controller (FastAPI process, owns the database and Redis), the worker fleet (separate hosts that run Firecracker microVMs), and S3 (snapshots, images, audit batches, file staging). Workers connect outward to the controller; they never accept inbound connections from the caller.

There's no separate ScaiBunker deployment of the controller — it runs as a ScaiGrid module in the main FastAPI process, behind the same auth, charged against the same accounting. Workers are separate hosts; they connect outward to the controller.

The storage proxy can additionally run as a stripped-down SCAIGRID_MODE=bunker_proxy process: same /v1/modules/scaibunker/storage/* routes, no DB or Redis, horizontally scalable behind a load balancer. Workers never see S3 credentials.

Request flow for one exec call#

1. Caller sends POST /v1/modules/scaibunker/bunkers/{id}/exec with a command, timeout, optional env and stdin.
2. Auth validates the bearer token and confirms the caller has scaibunker:execute.
3. Bunker lookup confirms the bunker is in this tenant and in running state; otherwise 404 or 409.
4. Worker pool picks the worker the bunker is scheduled on and opens an HTTP connection.
5. Execution. The controller forwards the exec request to the worker. The worker runs the command inside the microVM, captures stdout / stderr / exit code, enforces the timeout.
6. Output handling. Small outputs come back inline; large outputs are streamed to S3 via the storage proxy and the worker returns a full_output_ref instead.
7. Exec log. A row in mod_scaibunker_exec_log records the command, exit code, duration, output preview.
8. Activity timestamp. The bunker's last_activity_at is bumped so the idle-timeout sweeper leaves it alone.
9. Response. Inline mode returns the full result; stream: true returns Server-Sent Events with stdout, stderr, and exit events.

State#

• Bunkers, workers, images, snapshots, quota profiles, availability groups, bridges, exec logs — in ScaiGrid's MariaDB.
• Live status, quota counters, worker heartbeats — in Redis. The Redis state is rebuildable from MariaDB plus worker heartbeats; losing Redis is recoverable.
• Ext4 rootfs images, snapshots, large exec outputs, audit batches, file staging — in S3 via the storage proxy.
• Per-worker image cache — local on each worker; reconciled with the controller's mod_scaibunker_image_cache ledger every heartbeat.

Scheduler and image fan-out#

When a bunker create lands, the scheduler picks a worker that (a) has the requested image cached ready (unless the image is lazy_pull: true), (b) has free capacity for the requested CPU / memory / disk / GPU, and (c) for transit bunkers, owns every bridge the interfaces reference. If no worker fits, the call fails with WORKER_UNAVAILABLE or NO_SUITABLE_WORKER.

Availability groups scope the fan-out: adding an image to a group warms it on every worker in the group; adding a worker to a group warms every image already in the group. Workers that share a group can also fetch ext4 bytes from each other via P2P discovery (GET /peers/{sha256}), falling back to the controller storage proxy only when no peer has the sha cached yet.

Lifecycle state machine#

Every bunker moves through a fixed set of states:

pending → provisioning → running ⇄ paused / snapshotting → terminated

Failure can intrude from any non-terminal state and lands the bunker in failed. terminated and failed are terminal — bunkers don't come back. The transitions are enforced controller-side; an illegal move (e.g. resume on terminated) returns BUNKER_INVALID_TRANSITION (409).

A sweeper background task runs every 30 seconds and terminates bunkers that have exceeded max_lifetime_s or sat idle longer than idle_timeout_s. Defaults are lifecycle-dependent: ephemeral bunkers default to 1 hour max / 5 minutes idle; session bunkers to 8 hours / 15 minutes; persistent bunkers to no hard cap on lifetime but 15 minutes idle (the scheduler passivates them via snapshot).

Trust boundaries#

The worker token (SCAIBUNKER_WORKER_TOKEN) is the only credential workers carry. They use it for heartbeats, storage-proxy reads and writes, and peer discovery. The token never crosses the bunker boundary — code running inside a microVM has no way to reach it.

Bunker → outside-world traffic is governed by the bunker's network_profile. Even on unrestricted, all egress goes through the worker's network namespace and is optionally NDJSON-audited. Bunker → controller (the agent backchannel) is the only path the bunker has to call back into ScaiGrid; it is rate-limited and tenant-scoped.

Caller → ScaiBunker is the normal ScaiGrid bearer flow — JWT or API key, with scaibunker:* module permissions evaluated per request. The storage proxy uses either the shared worker bearer (workers) or short-lived capability tokens (admin clients that prefer not to hand workers more privilege than they need).

How it differs from docker run on your own host#

ScaiBunker is multi-tenant, quota-aware, auditable, and hardware-isolated by default. A docker host gives you the runtime; ScaiBunker gives you the runtime plus the surrounding machinery a multi-tenant product needs.

For a one-off CI container or a developer laptop, you don't need ScaiBunker. For multi-tenant sandboxed compute under quotas with auditing, the surface area saved is most of the work.

Background tasks#

The controller runs six cron-style background tasks per minute via arq:

• Sweep timeouts (every 30s) — terminates bunkers past their idle or lifetime caps.
• Monitor heartbeats (every 30s) — flips workers to offline when their heartbeat goes stale.
• Cleanup snapshots (every 5 min) — deletes snapshots past their expires_at.
• Emit usage ticks (every minute) — feeds the accounting pipeline.
• Scan pending images (every 2 min) — drains the image-scan queue.
• Request periodic rescans (daily at 03:17 UTC) — flips aged scan results to pending so the scanner re-checks for new CVEs in unchanged images.

These tasks run on the same worker pool that backs the rest of ScaiGrid's background work — no separate process to deploy.

Heartbeats#

Workers POST /v1/modules/scaibunker/heartbeat every ten seconds (WORKER_HEARTBEAT_INTERVAL_S). Each heartbeat carries the worker's current capacity (available CPU / memory / disk / GPU), active bunker count and per-bunker statuses, and optionally a cached_images list. If three intervals pass without a heartbeat, the monitor flips the worker to offline and the scheduler stops handing it new bunkers; active bunkers keep running until the heartbeat returns or the worker is drained.

The optional cached_images list is the worker's authoritative view of its on-disk image cache. When present, the controller reconciles mod_scaibunker_image_cache against it — marking entries the worker has but the controller lost (drift recovery after a DB issue) and evicting rows the worker no longer reports. This is what lets you wipe a worker's /var/lib/scaibunker and have the controller's view self-heal without manual intervention.

Quota counters#

Quota accounting lives in Redis under quota:bucket:{kind}:{id}:{resource} keys, one counter per (bucket, resource) pair. A bunker's lifecycle hooks increment the counters on create and decrement them on terminate, across every bucket the bunker contributes to (user-individual, group-shared, group-per-user — all evaluated at create time). The decrement set is stashed in quota:bunker:{bunker_id} so the right counters are touched even if the bunker's profile assignment changed mid-life.

The same QuotaResolver evaluates every applicable profile on create; the strictest cap wins. Callers see GET /quota-profiles/usage for a per-bucket breakdown — current, cap, remaining — without having to sum active bunkers client-side.

If Redis is wiped, counters rebuild from the active-bunker rows on the next create attempt; the recovery is implicit, not a separate migration step.