Repository navigation
ci(dlq): dispatchable event DLQ drain (deletes only named events) - #92
Merged
Merged
Conversation
Each dev-mailbox 409 loop of 18-20 Nov ended with the event in the DLQ, and EventDeadLetterAlarm has sat in ALARM since 13 Nov (4 messages now), so it pages nobody about new dead letters. The events themselves were delivered and their runs replayed green; the DLQ copies are stale. The DLQ cannot be redriven to the event queue from CI (the queue policy's DenyUntrustedPublishers boundary), so the drain lists every message with its event id - the suffix of the run id - and deletes only the event ids an operator names after replaying them. Grants live in their own stack, scoped to the dapier DLQ.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #91 (Hub intake 7ca6d265): drains the event DLQ that the dev-mailbox 409 loop filled.
EventDeadLetterAlarmhas been in ALARM since 13 Nov, so it sends no new pages. Live inspection from deploy run 44740195169: DLQ depth 4 on 20 Nov 12:06.DenyUntrustedPublishersboundary admits only Dapier roles and the inbound SNS topic. That boundary is intentionally left untouched.Adds:
scripts/drain_dlq.py: lists every message (event id = the run id's suffix, plus connector, event, subject, receives).--delete <ids>removes only the named events..github/workflows/dlq-drain.yml: workflow_dispatch with adeleteinput (empty = list only).infra/dlq-drain-access.yaml: its own stack, deployed by deploy.yml. Grantssqs:ReceiveMessage/DeleteMessage/GetQueueAttributesondapier-DeadLetterQueue-*only.tests/test_drain_dlq.py: listing, targeted delete, grants scope.