Troubleshooting
A short list of things that go wrong and how to fix them. If none of these match, check the request id in the response envelope and grep the ScaiGrid logs.
Bunker stays in pending or provisioning#
The worker hasn't acknowledged the create yet, or the scheduler can't find a worker that fits.
WORKER_UNAVAILABLEon create. No worker has enough free CPU / memory / disk / GPU. Either reduce the request, wait for other bunkers to terminate, or add a worker.WORKER_UNAVAILABLEon a GPU create. No GPU of the requested type is free, or no worker has thatgpu_type. GPU capacity isn't available yet on the managed service.NO_SUITABLE_WORKERon create. No worker has the image cachedready. Either setlazy_pull: trueon the image, force aPOST /images/{id}/warm, or wait for fan-out to complete.- Stuck in
provisioningfor more than a minute. The worker took the assignment but never reported back. Check the worker's status withGET /workers/{id}— iflast_heartbeatis stale, the bunker fails automatically once the worker has been silent for 5 minutes.
BUNKER_QUOTA_EXCEEDED#
The caller's resolved quota would be over a cap.
- Read
GET /quota-profiles/usage— you'll see one row per profile applying to your user, with the bucket, current usage, cap, and headroom. - Terminate idle bunkers; quota is decremented across every bucket they contributed to.
- Ask a tenant admin to assign a more generous profile or raise the existing one.
NETWORK_PROFILE_DENIED or LIFECYCLE_MODE_DENIED#
You picked a profile or lifecycle that needs a permission you don't have.
registryneedsscaibunker:network:registry,allowlistedneedsscaibunker:network:allowlisted, etc.sessionneedsscaibunker:create:session,persistentneedsscaibunker:create:persistent.- Ask a tenant admin to grant the specific key — these are deliberately granular.
INTERFACES_NOT_ALLOWED / TRANSIT_MISSING_INTERFACES#
The interfaces[] array is only valid with network_profile: "transit", and transit requires at least one interface.
- For transit bunkers, name a
bridge_namefor each interface; the bridge must already exist on a worker viaPOST /bridges. - For everything else, drop the
interfacesfield.
exec returns a 0 exit code but stdout is empty#
Two common causes:
- Output went to a file inside the bunker. Read it back with
GET /files/.... - Output was truncated to S3. Check
truncated: trueandfull_output_refon the response; fetch viaGET /storage/output/{bunker_id}/{name}.
exec times out mid-command#
The default timeout_s is 60. For builds, installs, or long-running scripts:
- Bump
timeout_s(no hard ceiling at the API; the bunker's ownmax_lifetime_sis the upper bound). - Use
"stream": trueso you see progress as it happens and can decide when to abort. - Move long work into a snapshot-able session bunker so a partial result survives a controller restart.
Files PUT returns 413 or fails on a large file#
The inline PUT path is fine for files under ~10 MB. For larger:
- Call
POST /files/uploadfor a pre-signed S3 URL andkey. - Upload the file directly to S3.
- Call
POST /files/commitwith{key, dest_path}— the worker injects the file into the bunker.
Image registered but no bunkers can use it#
- Check
GET /images/{id}/cache— if every row ispendingorfailed, the fan-out didn't reach the workers. - Check the image is in an availability group containing your workers (
POST /availability-groups/{group_id}/images). - Re-trigger with
POST /images/{id}/warm(idempotent). - Inspect a failing worker row's
errorfield — usually a registry auth failure, an OOM duringmkfs.ext4, or a size cap that was set too low.
Custom image build failed#
build_error on the image says why:
image has no /bin/sh— distroless andscratchimages can't run as bunkers. Build from a base with a shell.fixable CRITICAL vulnerabilities— the image contains a critical vulnerability that already has a fix. Rebuild it from an updated base and register it again.pull failed: …— the reference doesn't exist, isn't public, has no linux/amd64 variant, or is over the size limit.boot test failed— the image didn't start in a microVM. Check it runs/bin/shwithout extra setup.
Register the fixed image under a new name.
Image scan stuck on pending#
- The scanner runs every 2 minutes as a background task. Wait a couple of minutes.
- Images record the scan from their build; if a built image stays
pending, the image store couldn't be reached, and the next run retries. - An image that's still building stays
pendinguntil its build isready; checkbuild_status.
GPU bunker can't see the GPU#
nvidia-smi: not foundin an Alpine image. The driver tools and libraries in/usr/local/nvidiaare built for glibc. Use a glibc-based image (ubuntu-24.04,python-3.12,python-3.12-ml, or your own Debian/Ubuntu-based image).- CUDA reports no device or a driver mismatch. The driver supports CUDA 13 and older runtimes; check that your image's CUDA runtime isn't newer than what
nvidia-smireports.
SNAPSHOT_NOT_SUPPORTED#
GPU bunkers can't be snapshotted: the GPU's memory can't be saved with the VM. Keep the state you need on the bunker's disk with your own copy (files API, or push to your storage), and terminate without snapshot=true.
Bunker shows lost#
- Its worker hasn't sent a heartbeat for 5 minutes, or its worker reports that the bunker's VM stopped answering (about 70 seconds without a response from inside the VM). The VM may still be running, but the bunker can't be reached and isn't billed while it's lost.
- A bunker whose VM stopped answering comes back by itself if the VM answers again. If it doesn't, terminate it, or create a new bunker from one of its snapshots (
snapshot_idon create). - When the worker's heartbeats resume, the bunker comes back by itself in the state the worker reports (
runningorpaused). If it passed itsmax_lifetime_smeanwhile, it's terminated; if the worker no longer has it, it becomesfailed. - If the worker won't come back, an admin retires it (
POST /workers/{id}/retire): its session and persistent bunkers are recreated on another worker from their latest snapshot, under the same id. Anything written after that snapshot is lost. You can also terminate the bunker yourself (DELETE /bunkers/{id}). - A bunker stays
lostafter its worker was retired when no other worker can take it (resources, labels, or the bridges atransitbunker needs). It's tried again every minute.
Worker shows offline but its host is up#
- The heartbeat is what makes a worker
online. Default cadence is 10 seconds;WORKER_STALE_THRESHOLD_MULTIPLIER(default 3) means missing 30 seconds flips itoffline. - Check the worker's own logs for heartbeat send failures (wrong
SCAIBUNKER_WORKER_TOKEN, controller URL, network reachability). - The auto-detected status is in Redis; if Redis was wiped, the next heartbeat re-populates it.
Snapshots filling up S3#
- Snapshots default to a 7-day retention (
DEFAULT_SNAPSHOT_RETENTION_DAYS). The cleanup background task runs every 5 minutes. - Anything older than the retention with
expires_atset will be deleted automatically. - Manual snapshots without
expires_atare kept indefinitely — set lifecycle on the S3 bucket if you want hard caps.
Storage proxy 401s#
- The proxy verifies a constant
SCAIBUNKER_WORKER_TOKENplus, optionally, capability tokens minted viaPOST /storage/capabilities. - Mismatch between the worker's token and the controller's is the most common cause — they have to share the same string.
- For capability tokens,
scaibunker_capability_secretmust be set on the proxy as well as the controller.
Bunker can't reach hosts you expected#
- On
isolated, it can't reach anything. Onregistry, only the platform's package mirrors. Onallowlisted, only what you listed. - Allowlist entries are plain hostnames or
*.domain.com. URLs, paths, and double wildcards are rejected at create time. - For one-off "I just need to curl this URL" cases, use
unrestrictedplus enable the egress audit so the run is logged.
Conversations and bunkers double-charged#
- A persistent bunker holds quota for its entire lifetime, even when paused. Pause counts toward
max_concurrent_bunkers, not toward CPU usage on Redis. - If you want to give back resources, terminate (optionally with snapshot) — pause doesn't release.