Provisioning workflows

When a tenant subscribes to a ScaiLabs service, something has to happen on the other side — a tenant record gets created in the service, credentials get minted, the service's health gets verified, an email goes out. ScaiControl owns that orchestration through a small workflow engine. This page covers the model.

The four nouns#

	What it is	Lives in
Workflow definition	A JSON recipe: ordered steps with dependencies, handler refs, retries, timeouts. One row per `(service, type, version)`. Immutable once any workflow attaches.	`provisioning_workflow_definitions`
Workflow (instance)	A specific execution of a definition for a specific subscription. Has a status, a snapshot of context (tenant_id, plan, URLs, callback token), and per-step rows.	`provisioning_workflows`
Step	One node in the DAG. Per-step status, retry counters, configured handler, dependency list, mapped inputs, captured outputs.	`provisioning_workflow_steps`
Handler	The Python class that actually does something. Built-in (`ValidateDependencies`, `CallProvisioningApi`, …) or shipped as a plugin by a service team via setuptools entry points.	`services/provisioning/handlers/`

Definitions are templates; workflows are runs; steps are units of execution; handlers are code.

The DAG#

A definition is a directed acyclic graph. Each step declares which other steps it depends_on by name. The scheduler walks the DAG forward: a step becomes runnable the moment all its dependencies are in a terminal "succeeded" state (completed or skipped). Independent branches run in parallel within the same scheduler tick.

flowchart TD validate[validate] gen[gen_creds] vault[store_vault] api[call_provisioning_api] health[health_check] notify[notify] validate --> gen gen --> vault vault --> api api --> health health --> notify

Cycles are rejected at validation time (CYCLE), so the graph is guaranteed acyclic by the time the scheduler runs.

Step types#

Type	When to use it	Handler base class
`sync`	The handler completes in one call (preflight check, credential mint, vault write).	`StepHandler`
`async_callback`	The handler kicks off external work (POST to the service's provisioning API) and waits for the service to POST back to `/provisioning/callback` with the result.	`AsyncCallbackHandler`
`async_poll`	The handler kicks off external work and returns a poll URL — the engine polls it on a schedule until terminal.	`AsyncPollHandler`

The state column tracks where each step is in its lifecycle:

stateDiagram-v2 [*] --> pending pending --> running running --> completed running --> failed running --> waiting_callback running --> waiting_poll waiting_callback --> completed waiting_callback --> failed waiting_poll --> completed waiting_poll --> failed failed --> pending: retry (auto, until max_retries) pending --> skipped: workflow rollback / cancel completed --> rolled_back: workflow rollback completed --> [*] failed --> [*] rolled_back --> [*]

A step that fails with retries remaining flips back to pending and is picked up again after retry_delay_seconds. Only when retry_count == max_retries does it become terminally failed.

Inputs and outputs#

Each step's handler reads from an inputs dict and writes to a result dict. Wires are explicit:

json
{
  "name": "call_api",
  "handler_class": "CallProvisioningApi",
  "depends_on": ["gen_creds"],
  "input_mapping": {
    "client_id":     "gen_creds.client_id",
    "client_secret": "gen_creds.client_secret",
    "tenant_label":  "context.tenant_id"
  }
}

Source expressions:

<step_name>.<output_key> — pull from an upstream step's result. The step must be in depends_on (the validator enforces this; the visual designer auto-creates the edge when you pick a source step).
context.<key> — pull from the workflow's frozen context (tenant_id, partner_id, service_slug, plan_slug, service_base_url, etc. — see the context spec below).

What outputs a handler produces is declared in the handler's produces_outputs class attribute, surfaced via the handler registry.

Workflow context#

Set once at workflow start, frozen for the lifetime of the run:

Key	Source
`tenant_id`, `partner_id`, `subscription_id`, `service_slug`	Subscription row + service registry
`plan_slug`, `plan_tier`, `plan_id`	Plan associated with the subscription
`service_base_url`, `provisioning_api_url`, `health_check_url`	Service registry record
`callback_url`	`<API_URL>/api/v1/provisioning/callback`
`capabilities`, `features`, `quotas`	Catalog snapshots — service capabilities, plan features, plan quotas

Compensation and rollback#

Every step can declare a compensate_handler — a handler the engine calls in reverse topological order when the workflow is rolled back. Rollback is admin-triggered (POST /admin/provisioning/workflows/{id}/rollback) or automatic on a fatal failed workflow state.

Steps in pending / running / waiting_* are marked cancelled (no compensation — they never had a side effect).
Steps in completed get their compensate_handler invoked, then move to rolled_back.
Steps that already failed are left as-is.

So if gen_creds ran and minted a credential, compensate_handler: "GenerateServiceCredentials" would revoke that credential during rollback. The compensator is just another handler — usually the same class with the compensate() method implemented.

Definitions are versioned#

A definition row is keyed by (service_slug, workflow_type, version). Inserting a new definition for an existing (service, type) auto-bumps the version. The most recent active version is what new workflows attach to.

Bumping versions is the editing motion: instead of mutating a deployed definition, you create v+1.
Old workflows keep running against the version they started on — definitions are templates, copies live on workflow rows.
Deactivating the last active version of a service + type breaks new provisioning of that service. The admin UI and CLI's deactivate command guard against doing it accidentally when subscriptions are present.

The visual designer in /admin/provisioning defaults to "clone & edit" because of this — there's no in-place edit of the step definition once it's been used.

Handler registry#

The engine looks up handler classes by short name ("CallProvisioningApi") at runtime. Three resolution sources, in order:

Setuptools entry points under group scaicontrol.provisioning.handlers — how service teams ship their own handlers. The plugin's package declares them in its pyproject.toml:
toml
1 2
[project.entry-points."scaicontrol.provisioning.handlers"] BootstrapFlowWorkspace = "scaiflow.provisioning.handlers:BootstrapFlowWorkspace"
Built-in common.py — the seven handlers ScaiControl ships out of the box (ValidateDependencies, GenerateServiceCredentials, CallProvisioningApi, CallDeprovisioningApi, StoreCredentialsInVault, HealthCheck, SendProvisioningNotification).
Fully-qualified class path — pkg.module.MyHandler. Rare; used when neither short-name path applies.

Discovery happens at scheduler startup. Collisions (two plugins registering the same short name) raise HANDLER_NAME_COLLISION with both source packages named, so it's loud, not silent.

The GET /admin/provisioning/handlers endpoint enumerates everything visible — used by the visual designer's palette + the config_schema autocomplete in the property panel.

Workflow-level lifecycle#

stateDiagram-v2 [*] --> pending pending --> running: scheduler picks up first runnable step running --> completed: all steps terminal-success running --> partially_failed: a failed step blocks downstream running --> failed: max_retries exhausted on a blocker failed --> rolling_back: admin or auto completed --> rolling_back: admin rolling_back --> rolled_back completed --> [*] rolled_back --> [*] failed --> [*]

partially_failed is a stuck state — the engine isn't trying anymore, but admin recovery actions (Retry Step, Force Complete) can move it forward.

The visual designer#

/admin/provisioning has three tabs:

Workflows — running workflow instances. Recovery actions: Retry Step, Rollback, Force Complete.
Definitions — the templates. Browse, view, clone-and-edit. The editor modal has a Visual and JSON mode that share the same model. Visual mode renders the DAG as SVG with drag-to-reposition, drag-from-port-to-port to connect, and form-rendered config + input_mapping per step.
Handlers — registry of every discoverable handler with its declared schema, expected inputs, and produced outputs.

For the cookbook take, see Tutorials → Your first provisioning recipe.

Authoring locally#

Recipes live in service repos as provisioning/provision.json + provisioning/deprovision.json. Validate them in CI without a database:

bash
scaicontrol admin workflow-defs validate provisioning/provision.json --format json

Import them into the platform DB once they're clean:

bash
scaicontrol admin workflow-defs import provisioning/provision.json \
    --service scaiflow --type provision

Full CLI reference at reference/cli.