Brevo

Write-up
Multiple Services impacted
Degraded performance
View the incident
Summary:

On July 11, multiple core services including Transactional Emails, SMS, WhatsApp, Onboarding, Segmentation, and Contacts were impacted due to connectivity issues following a platform-level infrastructure change. The issue stemmed from a database migration to a private network, which disrupted the ability of certain services to connect to their databases.

The operational changes were reverted, and recovery procedures were initiated.

Impact:

During the incident, users experienced the following issues:

  • Delayed sending of over 300,000 transactional emails.

  • Delays in sending WhatsApp and SMS messages.

  • Interruptions or slowdowns in services relying on real-time data: onboarding flows, segmentation updates, contact operations, and WhatsApp/Conversations features.

  • Dashboard and webhook delays due to lags in event processing.

Please note that no data loss occurred, and affected services have since resumed normal functionality.

Root Cause:

The incident was triggered by a change in infrastructure involving the migration of database services to a private network. This change:

  • Caused connectivity issues between application and the database.

  • Introduced non-retryable errors in database queries.

The situation was further complicated by a simultaneous restart of multiple database nodes, which increased recovery time and led to resource saturation.

Resolution:

To restore stability, the following steps were taken:

  • The platform-level change was reverted, restoring connectivity to services.

  • Impacted services were restarted after verifying that all database shards were functional.

Prevention:

To prevent similar incidents, we are:

  • Enhancing deployment safeguards for infrastructure-level changes.

  • Reviewing and strengthening our database migration procedures to minimize risk and improve recovery mechanisms.


We deeply regret the disruption caused by this incident. We understand the importance of timely communication and platform reliability. We are taking this event seriously and will continue strengthening our systems to prevent recurrence.

Thank you for your understanding and continued support.

Powered by