---
title: Provisioning workflows
path: concepts/provisioning-workflows
status: published
---

# Provisioning workflows

When a tenant subscribes to a ScaiLabs service, *something has to happen on the other side* — a tenant record gets created in the service, credentials get minted, the service's health gets verified, an email goes out. ScaiControl owns that orchestration through a small workflow engine. This page covers the model.

## The four nouns

| | What it is | Lives in |
|---|---|---|
| **Workflow definition** | A JSON recipe: ordered steps with dependencies, handler refs, retries, timeouts. One row per `(service, type, version)`. Immutable once any workflow attaches. | `provisioning_workflow_definitions` |
| **Workflow** (instance) | A specific execution of a definition for a specific subscription. Has a status, a snapshot of context (tenant_id, plan, URLs, callback token), and per-step rows. | `provisioning_workflows` |
| **Step** | One node in the DAG. Per-step status, retry counters, configured handler, dependency list, mapped inputs, captured outputs. | `provisioning_workflow_steps` |
| **Handler** | The Python class that actually *does* something. Built-in (`ValidateDependencies`, `CallProvisioningApi`, …) or shipped as a plugin by a service team via setuptools entry points. | `services/provisioning/handlers/` |

Definitions are templates; workflows are runs; steps are units of execution; handlers are code.

## The DAG

A definition is a directed acyclic graph. Each step declares which other steps it `depends_on` by name. The scheduler walks the DAG forward: a step becomes runnable the moment all its dependencies are in a terminal "succeeded" state (`completed` or `skipped`). Independent branches run in parallel within the same scheduler tick.

```mermaid
flowchart TD
    validate[validate]
    gen[gen_creds]
    vault[store_vault]
    api[call_provisioning_api]
    health[health_check]
    notify[notify]

    validate --> gen
    gen --> vault
    vault --> api
    api --> health
    health --> notify
```

Cycles are rejected at validation time (`CYCLE`), so the graph is guaranteed acyclic by the time the scheduler runs.

## Step types

| Type | When to use it | Handler base class |
|---|---|---|
| `sync` | The handler completes in one call (preflight check, credential mint, vault write). | `StepHandler` |
| `async_callback` | The handler kicks off external work (POST to the service's provisioning API) and waits for the service to POST back to `/provisioning/callback` with the result. | `AsyncCallbackHandler` |
| `async_poll` | The handler kicks off external work and returns a poll URL — the engine polls it on a schedule until terminal. | `AsyncPollHandler` |

The state column tracks where each step is in its lifecycle:

```mermaid
stateDiagram-v2
    [*] --> pending
    pending --> running
    running --> completed
    running --> failed
    running --> waiting_callback
    running --> waiting_poll
    waiting_callback --> completed
    waiting_callback --> failed
    waiting_poll --> completed
    waiting_poll --> failed
    failed --> pending: retry (auto, until max_retries)
    pending --> skipped: workflow rollback / cancel
    completed --> rolled_back: workflow rollback
    completed --> [*]
    failed --> [*]
    rolled_back --> [*]
```

A step that fails with retries remaining flips back to `pending` and is picked up again after `retry_delay_seconds`. Only when `retry_count == max_retries` does it become terminally `failed`.

## Inputs and outputs

Each step's handler reads from an `inputs` dict and writes to a `result` dict. Wires are explicit:

```json
{
  "name": "call_api",
  "handler_class": "CallProvisioningApi",
  "depends_on": ["gen_creds"],
  "input_mapping": {
    "client_id":     "gen_creds.client_id",
    "client_secret": "gen_creds.client_secret",
    "tenant_label":  "context.tenant_id"
  }
}
```

Source expressions:

- `<step_name>.<output_key>` — pull from an upstream step's `result`. The step must be in `depends_on` (the validator enforces this; the visual designer auto-creates the edge when you pick a source step).
- `context.<key>` — pull from the workflow's frozen context (`tenant_id`, `partner_id`, `service_slug`, `plan_slug`, `service_base_url`, etc. — see [the context spec](#workflow-context) below).

What outputs a handler produces is declared in the handler's `produces_outputs` class attribute, surfaced via [the handler registry](#handler-registry).

### Workflow context

Set once at workflow start, frozen for the lifetime of the run:

| Key | Source |
|---|---|
| `tenant_id`, `partner_id`, `subscription_id`, `service_slug` | Subscription row + service registry |
| `plan_slug`, `plan_tier`, `plan_id` | Plan associated with the subscription |
| `service_base_url`, `provisioning_api_url`, `health_check_url` | Service registry record |
| `callback_url` | `<API_URL>/api/v1/provisioning/callback` |
| `capabilities`, `features`, `quotas` | Catalog snapshots — service capabilities, plan features, plan quotas |

## Compensation and rollback

Every step can declare a `compensate_handler` — a handler the engine calls **in reverse topological order** when the workflow is rolled back. Rollback is admin-triggered (`POST /admin/provisioning/workflows/{id}/rollback`) or automatic on a fatal `failed` workflow state.

- Steps in `pending` / `running` / `waiting_*` are marked `cancelled` (no compensation — they never had a side effect).
- Steps in `completed` get their `compensate_handler` invoked, then move to `rolled_back`.
- Steps that already `failed` are left as-is.

So if `gen_creds` ran and minted a credential, `compensate_handler: "GenerateServiceCredentials"` would revoke that credential during rollback. The compensator is just another handler — usually the same class with the `compensate()` method implemented.

## Definitions are versioned

A definition row is keyed by `(service_slug, workflow_type, version)`. Inserting a new definition for an existing `(service, type)` auto-bumps the version. The most recent **active** version is what new workflows attach to.

- Bumping versions is the editing motion: instead of mutating a deployed definition, you create v+1.
- Old workflows keep running against the version they started on — definitions are *templates*, copies live on workflow rows.
- Deactivating the last active version of a service + type breaks new provisioning of that service. The admin UI and CLI's `deactivate` command guard against doing it accidentally when subscriptions are present.

The visual designer in `/admin/provisioning` defaults to "clone & edit" because of this — there's no in-place edit of the step definition once it's been used.

## Handler registry

The engine looks up handler classes by short name (`"CallProvisioningApi"`) at runtime. Three resolution sources, in order:

1. **Setuptools entry points** under group `scaicontrol.provisioning.handlers` — how service teams ship their own handlers. The plugin's package declares them in its `pyproject.toml`:

   ```toml
   [project.entry-points."scaicontrol.provisioning.handlers"]
   BootstrapFlowWorkspace = "scaiflow.provisioning.handlers:BootstrapFlowWorkspace"
   ```

2. **Built-in `common.py`** — the seven handlers ScaiControl ships out of the box (`ValidateDependencies`, `GenerateServiceCredentials`, `CallProvisioningApi`, `CallDeprovisioningApi`, `StoreCredentialsInVault`, `HealthCheck`, `SendProvisioningNotification`).

3. **Fully-qualified class path** — `pkg.module.MyHandler`. Rare; used when neither short-name path applies.

Discovery happens at scheduler startup. Collisions (two plugins registering the same short name) raise `HANDLER_NAME_COLLISION` with both source packages named, so it's loud, not silent.

The `GET /admin/provisioning/handlers` endpoint enumerates everything visible — used by the visual designer's palette + the `config_schema` autocomplete in the property panel.

## Workflow-level lifecycle

```mermaid
stateDiagram-v2
    [*] --> pending
    pending --> running: scheduler picks up first runnable step
    running --> completed: all steps terminal-success
    running --> partially_failed: a failed step blocks downstream
    running --> failed: max_retries exhausted on a blocker
    failed --> rolling_back: admin or auto
    completed --> rolling_back: admin
    rolling_back --> rolled_back
    completed --> [*]
    rolled_back --> [*]
    failed --> [*]
```

`partially_failed` is a stuck state — the engine isn't trying anymore, but admin recovery actions (`Retry Step`, `Force Complete`) can move it forward.

## The visual designer

`/admin/provisioning` has three tabs:

- **Workflows** — running workflow instances. Recovery actions: `Retry Step`, `Rollback`, `Force Complete`.
- **Definitions** — the templates. Browse, view, clone-and-edit. The editor modal has a **Visual** and **JSON** mode that share the same model. Visual mode renders the DAG as SVG with drag-to-reposition, drag-from-port-to-port to connect, and form-rendered config + input_mapping per step.
- **Handlers** — registry of every discoverable handler with its declared schema, expected inputs, and produced outputs.

For the cookbook take, see [Tutorials → Your first provisioning recipe](../tutorials/your-first-provisioning-recipe).

## Authoring locally

Recipes live in service repos as `provisioning/provision.json` + `provisioning/deprovision.json`. Validate them in CI without a database:

```bash
scaicontrol admin workflow-defs validate provisioning/provision.json --format json
```

Import them into the platform DB once they're clean:

```bash
scaicontrol admin workflow-defs import provisioning/provision.json \
    --service scaiflow --type provision
```

Full CLI reference at [reference/cli](../reference/cli).

## See also

- [Tutorials: your first provisioning recipe](../tutorials/your-first-provisioning-recipe) — end-to-end cookbook.
- [Reference: state machines](../reference/state-machines) — the full lifecycle.
- [Reference: admin-provisioning API](../reference/api/admin-provisioning) — every workflow + definition endpoint with request/response schemas.
- [Sandbox](../sandbox) — local docker-compose stack for safe end-to-end testing.