Skip to content

fix: dev-mailbox oncall 409 redelivery loop behind EventQueueAgeAlarm - #91

Merged
alexeygrigorev merged 3 commits into
mainfrom
hub/intake-4ead1426691742787732f10f593d9cfaebbafb2a2e91642eebdc7bf62543e41c
Nov 20, 2026
Merged

alexeygrigorev merged 3 commits into
mainfrom
hub/intake-4ead1426691742787732f10f593d9cfaebbafb2a2e91642eebdc7bf62543e41c

Conversation

@alexeygrigorev

Copy link
Copy Markdown
Member

Fixes the root of the Dapier runtime alerts from 19–20 Nov 2026 (Hub intake 7ca6d265).

Root cause (one loop behind 3 of the 4 alert types)

Every dev-mailbox failure on 18–20 Nov (runs 91acff66, 9143bacc, 08674c39) had the same shape:

  1. Attempt 1: the Developer webhook leg succeeded. Hub's oncall ingest stored the email, but its response timed out, so verify-delivery failed the run.
  2. On each SQS redelivery (1800s visibility timeout):
    • The completed route condition was restored from history as a whole, so route.else.post-webhook was missing from steps and read as failed. This is an engine bug.
    • The retried ingest resent format-email output recovered from DynamoDB, whose key order isn't preserved. So Hub answered 409 delivery_id_conflict for the same X-Delivery-Id, every time.
  3. Each event looped through 5 receives (about 2h) and ended in the DLQ. The failure notice fired on the 3rd receive.
  4. EventQueueAgeAlarm fires when the oldest message reaches 900s. All four alarm datapoints (911/903/905/907s at 19 Nov 00:55 and 18:40, 20 Nov 10:22 and 11:52) are about 15 min after one of these messages was first sent, or re-received for the 4th time (SQS moves the message to the back after 3 receives, which resets its age).

Changes

  • engine: when a completed condition/paths step is recovered on redelivery, re-enter its recorded branch (no predicate re-evaluation). Completed children restore their own records; unfinished children run again.
  • dev-mailbox: an ingest 409 counts as already delivered, because the key is sha256(message_id), so Hub holds the email. Added retry: {attempts: 4, backoff_seconds: 120} so a transient Hub outage retries as a fresh message instead of a 1800s redelivery that trips the queue-age alarm.
  • ci: scripts/inspect_alarms.py, run by deploy and by the dispatchable inspection. It prints alarm states and transitions, queue-age peaks, DLQ depth and ClientCheckFailed counts, read-only within the existing grants (pinned by a test).

Tests

Targeted locally: test_branch_recovery.py (new), test_dev_mailbox_workflow.py, test_backup.py, test_grok_quota_fallback.py, test_retry_policy.py, engine/worker/logic/paths/replay suites: all green. Full suite runs in CI.

After merge, the dev-mailbox YAML is published live via dapier workflows save + publish, and the failed runs are replayed.

A redelivered event restores a completed condition/paths step from run
history as a whole, so its children never re-entered the steps context
under their nested ids. dev-mailbox's verify-delivery then read
route.else.post-webhook as missing and failed every redelivery even
though the webhook leg had been delivered on the first attempt
(runs 91acff66, 9143bacc, 08674c39 on 18-20 Nov).

Re-enter the recorded branch (no predicate re-evaluation): completed
children restore their own records, unfinished ones run again, like any
unfinished top-level step on a redelivery.
…ivery

Every dev-mailbox failure on 18-20 Nov had the same shape: Hub's oncall
ingest stored the first attempt but the response timed out; each queue
redelivery then resent the format-email output recovered from run
history (DynamoDB map, key order lost), so Hub answered 409
delivery_id_conflict for the same X-Delivery-Id forever. The run looped
through five 1800s redeliveries into the DLQ, and every redelivery
tripped the 900s EventQueueAgeAlarm (all four alarms of 19-20 Nov).

verify-delivery now treats an ingest 409 as already delivered (the key is
the message id's hash, so Hub holds this email). A retry policy (4
attempts, 120s) re-enqueues a failed leg as a fresh message instead of
waiting out the visibility timeout, so a transient Hub outage no longer
ages the queue past the alarm threshold.
The deploy and the dispatchable inspection now report the event-queue
alarms and the client-check alarm the operator topic pages on: current
state, verbatim transitions over three days, oldest-message age peaks,
dead-letter depth and ClientCheckFailed counts. An alarm email only says
a threshold was crossed; these lines prove recovery and non-recurrence.
Read-only, within the existing inspection grants (pinned by a test).
@alexeygrigorev
alexeygrigorev merged commit 4eb4cdc into main Nov 20, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant