Circuit Breaker for Webhook Notifications
  • 29 Oct 2021
  • 2 Minutes To Read
  • Dark
    Light
  • PDF

Circuit Breaker for Webhook Notifications

  • Dark
    Light
  • PDF

Article Summary

The Circuit Breaker pattern for Webhook notifications is a feature in Mambu that affects the rate at which notifications are sent if the target endpoints are responding slowly.

In Mambu, webhook notifications are queued as raised and move through the queue only after the response from the processor is received. Yet, we have noticed significant response delays in webhook sending caused by slow responses from certain processors. Thus, to encourage fast processors and/or asynchronous webhook-dependent integrations, we put in place a flow called "Circuit Breaker".

The Circuit Breaker is a software design pattern that is meant to protect one application that calls third-party applications (Mambu) from the bad cascading effects of delays and failures that may arise on the third-party side (tenant/processor).

How the Circuit Breaker works in Mambu

A timeout limit of 500ms exists, with the following flow:

  • If a given endpoint takes more than 500ms to respond to 5 out of 10 consecutive webhooks sent to it by Mambu, then the Circuit Breaker transitions to "Open" state

  • In the "Open” state, no webhooks are sent to that specific endpoint for 5 seconds

  • After 5 seconds, Mambu transitions the Circuit Breaker to "Half-Open" and allows sending webhooks to the endpoint again

  • While in "Half-Open" state, if the endpoint responds to 1 webhook successfully and doesn't exceed the 500ms timeout, then the Circuit Breaker transitions to the "Closed" state and webhooks are sent again as usual

  • If the endpoint fails the above validation, the Circuit Breaker moves back to "Open" (i.e. no webhooks sent for another 5 seconds), and so on.

The Mambu Workflow diagram below describes the flow mentioned above.
Notifications-Circuit-Breaker.png

In a nutshell, when we identify a slow response, then we put that endpoint on "pause" (assuming that it's receiving too many notifications) for 5 seconds. Afterwards, the next response’s speed gets tested, and the sending of webhooks resumes normally only if the endpoint responds within the acceptable threshold of 500ms.

Retry mechanism
It is important to mention that the retry mechanism is not influenced by the circuit breaker.

Conditions

Integrations review
If you are experiencing delays, we highly recommend you review the integrations using Mambu webhooks and ensure that before processing the information received from the webhook any further, your endpoint first responds to Mambu with status code "2XX", i.e. confirms that the webhook was received before performing next steps.
Data processing
We also strongly recommend that data processing requiring multiple steps is done asynchronously, not synchronously, as this affects the performance of the integration in general - and especially in a multi-endpoint configuration, if one endpoint is waiting for another to respond and so on.

 


Was this article helpful?