Webhook not delivered

A subscriber isn't receiving (or isn't processing) a webhook you expect. Diagnose top-down: was the event emitted? was the subscriber matched? was the request sent? did the subscriber accept it?

Step 1 — Was the source event emitted?#

1
2
3
4
5
6

SELECT id, event_type, idempotency_key, status, created_at
FROM event_outbox
WHERE subscription_id IS NULL
  AND event_type = '<topic>'
ORDER BY created_at DESC
LIMIT 20;

status='pending' — fan-out hasn't run yet; wait up to 30 seconds (dispatcher tick). status='dispatched' — fan-out ran; check Step 2. No rows — the producer never emitted. Possible causes:

• The domain action that should produce this event errored out. Check backend logs around the timestamp.
• The producer call is in a code path that wasn't reached (e.g. early-return on validation).
• The event topic name doesn't match what you're searching for. Cross-reference with the Catalog.

Step 2 — Was the subscriber matched?#

1
2
3

SELECT id, name, target_url, topics, is_active
FROM webhook_subscriptions
WHERE is_active = 1;

For each active subscription, check whether the topics array (glob patterns) would match the failing topic. For example, ["subscription.*"] matches subscription.activated but NOT pack_subscription.activated.

Then check whether the fan-out actually created a delivery row:

1
2
3
4
5

SELECT eo.id, eo.subscription_id, eo.status, eo.attempt_count, eo.next_attempt_at,
       eo.response_code, eo.error_message
FROM event_outbox eo
WHERE eo.idempotency_key = '<idempotency_key_from_step_1>'
  AND eo.subscription_id IS NOT NULL;

If no delivery row exists, the subscriber wasn't matched. Verify:

• is_active = 1
• Topic patterns include the event type
• The fan-out pass actually ran (look for dispatched on the source row in Step 1)

Step 3 — Did delivery succeed, fail, or stall?#

From the delivery row above:

Dispatcher not running#

If pending rows are piling up:

1

ps aux | grep arq | grep -v grep

No process → start the arq worker:

1

arq scaicontrol.workers.main.WorkerSettings

Or check container logs for event_dispatcher cron firing every 30 seconds.

Common response_code outcomes#

Step 4 — Subscriber acked 2xx but the side effect didn't happen#

The dispatcher is done — anything past this is on the subscriber side. Check the subscriber's inbox:

• Did they store the row? (event_id should now exist in their webhook_inbox table)
• Is their async processor running?
• Did processing fail and they're still retrying?
• Did dedup logic reject it as a duplicate? (Replay scenario — see Step 5)

ScaiControl can't see past the 2xx. Coordinate with the subscriber's owner.

Step 5 — Replays and dedup#

If the subscriber says "I already saw this," they may be deduping by event_id or idempotency_key. Manually replaying with the same event_id will be rejected by their inbox UNIQUE constraint.

To force re-delivery: insert a new outbox source row with a fresh event_id but the same idempotency_key. The fan-out will deliver to all matching subscribers. Subscribers that previously processed the idempotency_key will return 409 (idempotent ack) and ScaiControl treats it as success.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

INSERT INTO event_outbox (id, event_type, payload, idempotency_key, status, created_at)
SELECT
  UUID(),                          -- new event_id
  event_type,
  payload,
  idempotency_key,                 -- KEY: same as the original
  'pending',
  NOW()
FROM event_outbox
WHERE id = '<original_source_row_id>';

The next dispatcher tick will fan it out again. Caveat: this only re-emits to subscribers that match TODAY — if a subscriber was added since the original event, they will see this replay as a "new" event (which is usually fine; idempotency_key dedup catches it on their side anyway).

Step 6 — A dead row needs to retry#

dead is terminal — the dispatcher won't pick it up. To retry:

1
2
3

UPDATE event_outbox
SET status = 'pending', next_attempt_at = NOW(), attempt_count = 0, error_message = NULL
WHERE id = '<dead_row_id>';

Do this only after fixing whatever caused the failure (e.g. subscriber's URL, their secret, their bug). Otherwise it'll just die again.

See also#

• Reference: webhooks — dispatcher internals
• Reference: events/overview — envelope, signing, retry policy
• Concepts: webhooks