On Thursday, July 17th, between 17:00 UTC and 21:15 UTC, some users experienced delays in the delivery of outbound transactional and marketing emails. This was caused by an issue during a planned system update, which affected internal communication between critical components of our infrastructure.
The issue has since been resolved, and all services are now operating normally.
During the incident window, users may have experienced:
Delays in the sending and delivery of transactional and marketing emails
Temporary interruptions in automation workflows based on email events
All email queues have now been processed, and no data has been lost.
The incident occurred during a planned system update to improve performance. Unfortunately, we encountered a failure in the internal coordination system that manages how different parts of our platform communicate. Efforts to roll back the update did not succeed immediately, which prolonged the service disruption.
Eventually, we were able to resolve the coordination issue and restore normal service from the original system.
Our engineering team:
Identified the issue shortly after the update began
Resolved internal communication problems between systems and resumed normal service
Closely monitored the processing of delayed emails until all queues were cleared
To prevent similar incidents in the future, we are implementing the following improvements:
Improve testing and validation of rollback and disaster recovery (DR) procedures
Review and strengthen upgrade processes, ensuring better safeguards during maintenance
Address system configuration inconsistencies and ensure platform settings are aligned
We sincerely apologize for any inconvenience this caused and appreciate your patience and trust. Ensuring the reliability and resilience of our platform remains a top priority.
Thank you for your continued support.