- 29 Oct 2021
- 2 Minutes To Read
- Print
- DarkLight
- PDF
Circuit Breaker for Webhook Notifications
- Updated On 29 Oct 2021
- 2 Minutes To Read
- Print
- DarkLight
- PDF
The Circuit Breaker pattern in Mambu manages webhook notifications to prevent delays caused by slow processors. If an endpoint takes more than 500ms to respond to 5 out of 10 consecutive webhooks, the Circuit Breaker transitions to "Open" state, pausing notifications for 5 seconds. It then moves to "Half Open" state, allowing one successful response within 500ms to reset to "Closed" state. If the response fails, the Circuit Breaker returns to "Open" state. This pattern protects against cascading effects of delays and failures in third-party applications.
The Circuit Breaker pattern for Webhook notifications is a feature in Mambu that affects the rate at which notifications are sent if the target endpoints are responding slowly.
In Mambu, webhook notifications are queued as raised and move through the queue only after the response from the processor is received. Yet, we have noticed significant response delays in webhook sending caused by slow responses from certain processors. Thus, to encourage fast processors and/or asynchronous webhook-dependent integrations, we put in place a flow called "Circuit Breaker".
The Circuit Breaker is a software design pattern that is meant to protect one application that calls third-party applications (Mambu) from the bad cascading effects of delays and failures that may arise on the third-party side (tenant/processor).
How the Circuit Breaker works in Mambu
A timeout limit of 500ms exists, with the following flow:
If a given endpoint takes more than 500ms to respond to 5 out of 10 consecutive webhooks sent to it by Mambu, then the Circuit Breaker transitions to "Open" state
In the "Open” state, no webhooks are sent to that specific endpoint for 5 seconds
After 5 seconds, Mambu transitions the Circuit Breaker to "Half-Open" and allows sending webhooks to the endpoint again
While in "Half-Open" state, if the endpoint responds to 1 webhook successfully and doesn't exceed the 500ms timeout, then the Circuit Breaker transitions to the "Closed" state and webhooks are sent again as usual
If the endpoint fails the above validation, the Circuit Breaker moves back to "Open" (i.e. no webhooks sent for another 5 seconds), and so on.
The Mambu Workflow diagram below describes the flow mentioned above.
In a nutshell, when we identify a slow response, then we put that endpoint on "pause" (assuming that it's receiving too many notifications) for 5 seconds. Afterwards, the next response’s speed gets tested, and the sending of webhooks resumes normally only if the endpoint responds within the acceptable threshold of 500ms.
Conditions
2XX
", i.e. confirms that the webhook was received before performing next steps.