---
title: Webhook not delivered
path: troubleshooting/webhook-not-delivered
status: published
---

# Webhook not delivered

A subscriber isn't receiving (or isn't processing) a webhook you expect. Diagnose top-down: was the event emitted? was the subscriber matched? was the request sent? did the subscriber accept it?

## Step 1 — Was the source event emitted?

```sql
SELECT id, event_type, idempotency_key, status, created_at
FROM event_outbox
WHERE subscription_id IS NULL
  AND event_type = '<topic>'
ORDER BY created_at DESC
LIMIT 20;
```

`status='pending'` — fan-out hasn't run yet; wait up to 30 seconds (dispatcher tick).
`status='dispatched'` — fan-out ran; check Step 2.
No rows — the producer never emitted. Possible causes:

- The domain action that should produce this event errored out. Check backend logs around the timestamp.
- The producer call is in a code path that wasn't reached (e.g. early-return on validation).
- The event topic name doesn't match what you're searching for. Cross-reference with the [Catalog](../reference/events/catalog).

## Step 2 — Was the subscriber matched?

```sql
SELECT id, name, target_url, topics, is_active
FROM webhook_subscriptions
WHERE is_active = 1;
```

For each active subscription, check whether the `topics` array (glob patterns) would match the failing topic. For example, `["subscription.*"]` matches `subscription.activated` but NOT `pack_subscription.activated`.

Then check whether the fan-out actually created a delivery row:

```sql
SELECT eo.id, eo.subscription_id, eo.status, eo.attempt_count, eo.next_attempt_at,
       eo.response_code, eo.error_message
FROM event_outbox eo
WHERE eo.idempotency_key = '<idempotency_key_from_step_1>'
  AND eo.subscription_id IS NOT NULL;
```

If no delivery row exists, the subscriber wasn't matched. Verify:

- `is_active = 1`
- Topic patterns include the event type
- The fan-out pass actually ran (look for `dispatched` on the source row in Step 1)

## Step 3 — Did delivery succeed, fail, or stall?

From the delivery row above:

| `status` | Meaning | Next step |
|---|---|---|
| `pending`, `attempt_count = 0` | Never tried yet | Wait 30 sec. If still pending, check dispatcher is running |
| `pending`, `attempt_count > 0`, `next_attempt_at` in future | Backoff in progress | Wait. Inspect `error_message`, `response_code` |
| `dispatched` | Subscriber returned 2xx | Subscriber received it. Move to Step 4 |
| `dead` | Terminal failure | Read `error_message` and `response_code`. Common: 4xx → subscriber rejected, 6+ retries → repeated 5xx/timeout |

### Dispatcher not running

If `pending` rows are piling up:

```bash
ps aux | grep arq | grep -v grep
```

No process → start the arq worker:

```bash
arq scaicontrol.workers.main.WorkerSettings
```

Or check container logs for `event_dispatcher` cron firing every 30 seconds.

### Common response_code outcomes

| Code | What to fix |
|---|---|
| `401` | Subscriber rejected the signature. Verify the shared secret. Subscribers should HMAC the **raw body**, not a re-serialised JSON. See [Events overview](../reference/events/overview). |
| `400` | Subscriber rejected the payload. Inspect `response_body_sample` — usually a JSON schema error |
| `404` / `502` | Subscriber URL wrong, host down, or proxy misconfigured. Test with curl from inside the ScaiControl container |
| `408` / `timeout` | Subscriber too slow to ack (>10 sec). They should 200 immediately and process asynchronously. See [Events: subscriber inbox pattern](../reference/events/overview) |
| `5xx` | Genuine subscriber-side issue. Will retry per backoff |

## Step 4 — Subscriber acked 2xx but the side effect didn't happen

The dispatcher is done — anything past this is on the subscriber side. Check the subscriber's inbox:

- Did they store the row? (`event_id` should now exist in their `webhook_inbox` table)
- Is their async processor running?
- Did processing fail and they're still retrying?
- Did dedup logic reject it as a duplicate? (Replay scenario — see Step 5)

ScaiControl can't see past the 2xx. Coordinate with the subscriber's owner.

## Step 5 — Replays and dedup

If the subscriber says "I already saw this," they may be deduping by `event_id` or `idempotency_key`. Manually replaying with the same `event_id` will be rejected by their inbox UNIQUE constraint.

To force re-delivery: insert a new outbox source row with a fresh `event_id` but the same `idempotency_key`. The fan-out will deliver to all matching subscribers. Subscribers that previously processed the `idempotency_key` will return 409 (idempotent ack) and ScaiControl treats it as success.

```sql
INSERT INTO event_outbox (id, event_type, payload, idempotency_key, status, created_at)
SELECT
  UUID(),                          -- new event_id
  event_type,
  payload,
  idempotency_key,                 -- KEY: same as the original
  'pending',
  NOW()
FROM event_outbox
WHERE id = '<original_source_row_id>';
```

The next dispatcher tick will fan it out again. **Caveat:** this only re-emits to subscribers that match TODAY — if a subscriber was added since the original event, they will see this replay as a "new" event (which is usually fine; idempotency_key dedup catches it on their side anyway).

## Step 6 — A dead row needs to retry

`dead` is terminal — the dispatcher won't pick it up. To retry:

```sql
UPDATE event_outbox
SET status = 'pending', next_attempt_at = NOW(), attempt_count = 0, error_message = NULL
WHERE id = '<dead_row_id>';
```

Do this only after fixing whatever caused the failure (e.g. subscriber's URL, their secret, their bug). Otherwise it'll just die again.

## See also

- [Reference: webhooks](../reference/webhooks) — dispatcher internals
- [Reference: events/overview](../reference/events/overview) — envelope, signing, retry policy
- [Concepts: webhooks](../concepts/webhooks)