StreamShop Closing Failure Modes #
With the StreamShop happy path and Idempotency Keys in place the next step has been closing some failure modes:
Transactional Outbox #
If Redpanda is down we were returning a success response code but the message would never actually publish. This update adds an outbox table and a poller in the Order API to fix this issue.
The poller claims unpublished records from the database table. If the message can be published the published_at column is updated, if the message cannot be published the column stays as null.
Bounded Retries and a Smarter DLQ #
Rather than infinite retries in the Event Processor this commit adds an environment variable and a loop to ensure only a set number of retries occur before moving the message to the Dead Letter Queue.
This also sets the stage for replays in case the retries did not exceed a system outage.
Catalog Timeout and a Fail-Closed Breaker #
A circuit breaker has been added to the Order API when it calls out the the Catalog API
If the circuit breaker is triggered a 503 is returned to stop hanging order requests. The circuit breaker is then reset after a short recovery window.
Video Walkthrough #
I have a recorded a short video on YouTube showing the closed failure methods