Platform
ScaiWave ScaiGrid ScaiCore ScaiBot ScaiDrive ScaiKey Modellen Tools & Services
Oplossingen
Organisaties Ontwikkelaars Internet Service Providers Managed Service Providers AI-in-a-Box
Kenniscentrum
Ondersteuning Documentation Blog Downloads
Bedrijf
Over ons Onderzoek Vacatures Investeren Contact
Inloggen

Architecture

ScaiBunker is a thin product layer over ScaiGrid's existing primitives — tenancy, quotas, audit, S3-backed storage — wrapped around a fleet of worker nodes that run Firecracker microVMs. The controller never executes code itself; it schedules work onto workers and relays bytes between workers and object storage.

Components#

Three tiers: the ScaiGrid controller (FastAPI process, owns the database and Redis), the worker fleet (separate hosts that run Firecracker microVMs), and S3 (snapshots, images, audit batches, file staging). Workers connect outward to the controller; they never accept inbound connections from the caller.

flowchart LR C[Caller] CTRL["Controller<br/>Bunker svc<br/>Quota svc<br/>Scheduler<br/>Image svc<br/>Snapshot svc"] SP["Storage proxy<br/>/storage/..."] S3[Garage S3] WD[scaibunker-worker daemon] FC["Firecracker microVM<br/>(ext4)"] S3C[S3 client] C -- "/v1/modules/scaibunker/..." --> CTRL CTRL -- "bunker, exec results, files" --> C CTRL -- HTTP --> WD WD --> CTRL WD --- FC WD --- S3C SP <-- stream --> S3C CTRL --> SP SP -- S3 PUT --> S3 S3C --> S3 subgraph SG [ScaiGrid] CTRL SP S3 end subgraph WF [Worker fleet] WD FC S3C end

There's no separate ScaiBunker deployment of the controller — it runs as a ScaiGrid module in the main FastAPI process, behind the same auth, charged against the same accounting. Workers are separate hosts; they connect outward to the controller.

The storage proxy can additionally run as a stripped-down SCAIGRID_MODE=bunker_proxy process: same /v1/modules/scaibunker/storage/* routes, no DB or Redis, horizontally scalable behind a load balancer. Workers never see S3 credentials.

Request flow for one exec call#

  1. Caller sends POST /v1/modules/scaibunker/bunkers/{id}/exec with a command, timeout, optional env and stdin.
  2. Auth validates the bearer token and confirms the caller has scaibunker:execute.
  3. Bunker lookup confirms the bunker is in this tenant and in running state; otherwise 404 or 409.
  4. Worker pool picks the worker the bunker is scheduled on and opens an HTTP connection.
  5. Execution. The controller forwards the exec request to the worker. The worker runs the command inside the microVM, captures stdout / stderr / exit code, enforces the timeout.
  6. Output handling. Small outputs come back inline; large outputs are streamed to S3 via the storage proxy and the worker returns a full_output_ref instead.
  7. Exec log. A row in mod_scaibunker_exec_log records the command, exit code, duration, output preview.
  8. Activity timestamp. The bunker's last_activity_at is bumped so the idle-timeout sweeper leaves it alone.
  9. Response. Inline mode returns the full result; stream: true returns Server-Sent Events with stdout, stderr, and exit events.

State#

  • Bunkers, workers, images, snapshots, quota profiles, availability groups, bridges, exec logs — in ScaiGrid's MariaDB.
  • Live status, quota counters, worker heartbeats — in Redis. The Redis state is rebuildable from MariaDB plus worker heartbeats; losing Redis is recoverable.
  • Ext4 rootfs images, snapshots, large exec outputs, audit batches, file staging — in S3 via the storage proxy.
  • Per-worker image cache — local on each worker; reconciled with the controller's mod_scaibunker_image_cache ledger every heartbeat.

Scheduler and image fan-out#

When a bunker create lands, the scheduler picks a worker that (a) has the requested image cached ready (unless the image is lazy_pull: true), (b) has free capacity for the requested CPU / memory / disk / GPU, and (c) for transit bunkers, owns every bridge the interfaces reference. If no worker fits, the call fails with WORKER_UNAVAILABLE or NO_SUITABLE_WORKER. Among the workers that fit, a bunker without GPUs avoids GPU workers, so their GPUs stay available. A worker can still turn a create away when it's already full (its last heartbeat didn't show the newest bunkers yet); the bunker then moves to the next worker that fits, and the call fails with WORKER_UNAVAILABLE if there is none.

GPU bunkers run on GPU workers in a separate VM runtime with PCI passthrough: each bunker gets whole GPUs, never a share of one, and the GPUs are reset before the next bunker gets them. The NVIDIA driver runs inside the bunker. Because a GPU's state can't be saved, GPU bunkers have no snapshots or checkpoints; pause and resume work.

Availability groups scope the fan-out: adding an image to a group warms it on every worker in the group; adding a worker to a group warms every image already in the group. Workers that share a group can also fetch ext4 bytes from each other via P2P discovery (GET /peers/{sha256}), falling back to the controller storage proxy only when no peer has the sha cached yet.

Lifecycle state machine#

Every bunker moves through a fixed set of states:

pending → provisioning → running ⇄ paused / snapshotting → terminated

Failure can intrude from any non-terminal state and lands the bunker in failed. terminated and failed are terminal — bunkers don't come back. The transitions are enforced controller-side; an illegal move (e.g. resume on terminated) returns BUNKER_INVALID_TRANSITION (409).

If a bunker's worker stops sending heartbeats for 5 minutes, the bunker becomes lost. Its VM may still be running, but it can't be reached and isn't billed. When the worker reports the bunker again, ScaiGrid puts it back in the state the worker reports (running or paused), or terminates it if it has passed its max_lifetime_s. A lost bunker the returning worker no longer has becomes failed. The same applies to a single running bunker whose worker reports that its VM stopped answering (no response from inside the VM for about 70 seconds): it becomes lost, isn't billed, and comes back when the worker reports it running again. ScaiGrid never destroys it for that alone. You can terminate a lost bunker at any time. Bunkers still pending or provisioning when their worker is lost fail instead. A lost bunker past its max_lifetime_s is terminated even if its worker never returns.

If a worker isn't coming back, an admin retires it (POST /workers/{id}/retire). A worker silent for 2 hours is retired automatically, as long as another worker is still running. Each of its lost bunkers is then recreated on another worker, under the same bunker id, from its latest snapshot: it goes provisioning and then running, with what it had stored up to that snapshot. Anything written after the snapshot is gone. Ephemeral bunkers, and bunkers without a snapshot, become failed. If no worker can take a bunker right now, it stays lost and is tried again every minute. Nothing is billed while a bunker is lost; billing resumes on the new worker.

Persistent bunkers are snapshotted automatically for this (checkpoints, trigger: "checkpoint"): every 6 hours while they run, and before they are paused for being idle if they changed since their last snapshot. The newest two checkpoints per bunker are kept; snapshots you take yourself are never removed automatically. While a checkpoint is taken the bunker is snapshotting, as for a requested snapshot.

A sweeper background task runs every 30 seconds. It terminates bunkers that have exceeded max_lifetime_s, in any state. A bunker idle longer than idle_timeout_s is terminated if it's ephemeral, and paused if it's session or persistent: the VM is frozen in place and keeps its id, and only persistent bunkers are snapshotted first (a checkpoint). A paused bunker stays paused until you call POST /bunkers/{id}/resume; exec, file and shell calls on it return 409 BUNKER_NOT_RUNNING. Defaults are lifecycle-dependent: ephemeral bunkers default to 1 hour max / 5 minutes idle; session bunkers to 8 hours / 15 minutes; persistent bunkers to no hard cap on lifetime but 15 minutes idle. You can set max_lifetime_s up to 24 hours for ephemeral and 72 hours for session bunkers, and idle_timeout_s up to the bunker's lifetime.

Only API calls count as activity: exec, file reads and writes, and typing in a shell. A running exec keeps a bunker from going idle for its timeout_s. A process left running in the background (nohup, setsid) keeps running after its exec returns, but doesn't count: give such a bunker a long idle_timeout_s, or check on the job more often than the timeout.

Trust boundaries#

The worker token (SCAIBUNKER_WORKER_TOKEN) is the only credential workers carry. They use it for heartbeats, storage-proxy reads and writes, and peer discovery. The token never crosses the bunker boundary — code running inside a microVM has no way to reach it.

Bunker → outside-world traffic is governed by the bunker's network_profile. Even on unrestricted, all egress goes through the worker's network namespace and is optionally NDJSON-audited. Bunker → controller (the agent backchannel) is the only path the bunker has to call back into ScaiGrid; it is rate-limited and tenant-scoped.

Caller → ScaiBunker is the normal ScaiGrid bearer flow — JWT or API key, with scaibunker:* module permissions evaluated per request. The storage proxy uses either the shared worker bearer (workers) or short-lived capability tokens (admin clients that prefer not to hand workers more privilege than they need).

How it differs from docker run on your own host#

ScaiBunker is multi-tenant, quota-aware, auditable, and hardware-isolated by default. A docker host gives you the runtime; ScaiBunker gives you the runtime plus the surrounding machinery a multi-tenant product needs.

Concern DIY container host ScaiBunker
Isolation Linux namespaces + seccomp Firecracker hardware virtualisation
Multi-tenancy You design it Built-in (tenant column, quota profiles, audit)
Image distribution You orchestrate Availability groups + P2P peer fetch
Network policy iptables rules you write Five named profiles, per-flow audit
Snapshots You wire up Built-in, S3-backed
Quotas You build it Per-user / per-group, composable, Redis-backed
Audit You instrument Every exec (command line, exit code, first 4 KiB of output) and file op (path) logged for 30 days; shell sessions logged (who, when, how long), not their content

For a one-off CI container or a developer laptop, you don't need ScaiBunker. For multi-tenant sandboxed compute under quotas with auditing, the surface area saved is most of the work.

Background tasks#

The controller runs these background tasks via arq:

  • Sweep timeouts (every 30s) — pauses or terminates bunkers past their idle or lifetime caps, lost ones included.
  • Monitor heartbeats (every 30s) — flips workers to offline when their heartbeat goes stale, and marks their bunkers lost after 5 minutes of silence.
  • Recreate bunkers (every minute) — recreates the lost bunkers of retired workers on other workers.
  • Checkpoints (every minute) — snapshots persistent bunkers that are due.
  • Collect image builds (every minute) — applies finished custom-image builds.
  • Scan pending images (every 2 min) — records the build scan of new image content.
  • Usage checkpoint (every 5 min) — cuts open usage segments at each full hour.
  • Cleanup snapshots (every 5 min) — deletes snapshots past their expires_at.
  • Reconcile quota (every 5 min) — rebuilds quota counters from the database.
  • Roll out images (every 15 min) — moves platform images to new versions, one worker at a time.
  • Snapshot orphans (hourly) — removes unfinished uploads and stored objects without a snapshot.

These tasks run on the same worker pool that backs the rest of ScaiGrid's background work — no separate process to deploy.

Heartbeats#

Workers POST /v1/modules/scaibunker/heartbeat every ten seconds (WORKER_HEARTBEAT_INTERVAL_S). Each heartbeat carries the worker's current capacity (available CPU / memory / disk / GPU), active bunker count and per-bunker statuses, and optionally a cached_images list. If three intervals pass without a heartbeat, the monitor flips the worker to offline and the scheduler stops handing it new bunkers; active bunkers keep running. After 5 minutes of silence (WORKER_LOST_THRESHOLD_S) the worker's bunkers become lost. Once heartbeats resume, each lost bunker the worker reports is taken back (or terminated if past its lifetime), and one it leaves out of 3 heartbeats in a row becomes failed. Bunkers the controller has already ended but the worker still reports are destroyed.

The optional cached_images list is the worker's authoritative view of its on-disk image cache. When present, the controller reconciles mod_scaibunker_image_cache against it — marking entries the worker has but the controller lost (drift recovery after a DB issue) and evicting rows the worker no longer reports. This is what lets you wipe a worker's /var/lib/scaibunker and have the controller's view self-heal without manual intervention.

The same reconcile removes deactivated images: a reported image that has been deactivated is deleted from that worker's cache. Bunkers still running on it keep working; the worker frees the disk once the last one ends.

Quota counters#

Quota accounting lives in Redis under quota:bucket:{kind}:{id}:{resource} keys, one counter per (bucket, resource) pair. A bunker's lifecycle hooks increment the counters on create and decrement them on terminate, across every bucket the bunker contributes to (user-individual, group-shared, group-per-user — all evaluated at create time). The decrement set is stashed in quota:bunker:{bunker_id} so the right counters are touched even if the bunker's profile assignment changed mid-life.

The same QuotaResolver evaluates every applicable profile on create; the strictest cap wins. Callers see GET /quota-profiles/usage for a per-bucket breakdown — current, cap, remaining — without having to sum active bunkers client-side.

If Redis is wiped, counters rebuild from the active-bunker rows on the next create attempt; the recovery is implicit, not a separate migration step.

Updated 2026-11-17 10:07:48 View source (.md) rev 43