---
title: Webhooks reference
path: reference/webhooks
status: published
---

# Webhooks reference

Operational details on ScaiControl's outbound webhook dispatcher. For the topic catalog see [Events](./events/catalog); for the conceptual overview see [Concepts: webhooks](../concepts/webhooks).

## Where the dispatcher lives

`backend/src/scaicontrol/workers/event_dispatcher.py` — runs as an `arq` cron every 30 seconds.

## Storage

Two tables:

- `webhook_subscriptions` — one row per subscriber. Managed via `/admin/webhook-subscriptions`.
- `event_outbox` — durable log of every event AND every delivery attempt. Two row generations live here:
  1. **Source rows** (`subscription_id IS NULL`) — written by `emit_event()` inside the originating transaction. One per logical event.
  2. **Delivery rows** (`subscription_id` set) — created by the dispatcher when it fans the source row out to matching subscribers. Each carries its own retry state.

## Dispatcher tick

Each 30-second tick does two passes:

**Pass 1 — fan-out.** Claim up to 50 pending source rows; for each, look up subscribers whose `topics[]` glob matches the event_type, and INSERT one delivery row per match. Flip the source row to `dispatched`. The `(subscription_id, idempotency_key)` unique constraint catches double-fan-out on crash recovery.

**Pass 2 — deliver.** Claim up to 200 pending delivery rows that are ready (`next_attempt_at <= now`); build the canonical envelope, HMAC-sign, POST to `target_url` with a 10-second timeout. Update status per the response code:

| Outcome | Status | Next |
|---|---|---|
| 2xx | `dispatched` | terminal |
| 409 | `dispatched` (idempotent ack) | terminal |
| 4xx (except 409) | `dead` | terminal |
| 5xx / timeout / network | `pending` | backoff: 1m → 5m → 30m → 2h → 12h → 24h, then `dead` |

Both passes commit their own transaction. A crash mid-tick is safe — the next tick picks up where it left off.

## Backoff schedule

```python
BACKOFF_SCHEDULE = [60, 300, 1800, 7200, 43200, 86400]  # seconds
MAX_ATTEMPTS = 6
```

After `MAX_ATTEMPTS`, the row goes `dead` and stops being retried. Dead rows are kept indefinitely for audit; clear them via a manual SQL purge if storage matters.

## Idempotency

`idempotency_key` is constructed in domain code and shared across the source row + all delivery rows for the same logical event. Subscribers should use it as their inbox dedup key.

Format: `<resource_type>:<resource_id>:<event_type>:<lifecycle_step>`. Examples:

- `subscription:sub_abc:activated:initial`
- `subscription:sub_abc:cancelled:reaper`
- `pack_subscription:pak_xyz:activated:initial`
- `tenant:tnt_xyz:billing_updated:2026-05-12T15:22:00`

Once an idempotency_key has been seen by a subscriber for the same `subscription_id`, the dispatcher's unique constraint prevents a duplicate insert if the source row is somehow re-fanned-out — so subscribers never see duplicates with the same key from the same logical delivery attempt, even on dispatcher crashes.

## Signing

```python
import hmac, hashlib
sig = "sha256=" + hmac.new(secret.encode(), raw_body_bytes, hashlib.sha256).hexdigest()
```

`raw_body_bytes` is the byte-exact body — re-serialising the parsed JSON will not produce the same signature.

The secret resolves in this order:
1. If `webhook_subscriptions.secret_vault_path` is set → look up in ScaiVault.
2. Else use the inline `secret` column.
3. If neither resolves → the delivery row goes `dead` with `error_message="no secret resolved for subscription"`.

Rotating a secret: edit the subscription via the admin UI (PATCH `secret` to the new value), confirm the subscriber accepts the new key, then delete the old one in their config. There's no overlap window built in; for zero-downtime rotation, run two subscriptions in parallel during the cutover.

## Operational notes

- **No tenant filter.** The outbox is platform-wide. Webhook subscribers see events for every tenant in the system; topic filters are the only mechanism for narrowing.
- **No rate limiting.** If you need to throttle a subscriber, do it on their side (queue inside their inbox handler and ack 200 immediately).
- **Inspection.** `SELECT * FROM event_outbox WHERE event_type LIKE 'subscription.%' ORDER BY created_at DESC LIMIT 50` gives a recent timeline. Delivery rows have `response_code` + `response_body_sample` (first 512 chars) for debugging.
- **Replays.** To re-emit a logical event, INSERT a new source row with a fresh `event_id` but the same `idempotency_key` — subscribers that have already seen the key will dedup, those that haven't will receive it. Or just trigger the original domain action again if it's idempotent.

## Adding a new topic

1. Define the payload shape in code (`services/events/builders.py`) and write a JSON Schema in `docs/integrations/scaicontrol/events/`.
2. Call `emit_event()` at the appropriate domain mutation site.
3. Add the topic to the [Catalog](./events/catalog) docs page.
4. Bump no version — adding topics is backwards-compatible.
5. The dispatcher needs no changes — it's topic-agnostic.

## Adding a subscriber

`POST /api/v1/admin/webhook-subscriptions`:

```json
{
  "name":      "scaicrm-staging",
  "target_url": "https://crm-staging.scailabs.eu/v1/webhooks/scaicontrol",
  "topics":     ["subscription.*", "pack_subscription.*", "tenant.*", "partner.*"],
  "secret":     "shared-secret-here",
  "is_active":  true
}
```

Or via the UI at `/admin/webhook-subscriptions`. Either way, deliveries start on the next dispatcher tick (within 30 seconds).