On June 26, we experienced a disruption in our marketing email sending service, which caused scheduled campaign emails to be temporarily delayed. This issue began at 10:00 AM UTC and lasted until 10:39 AM UTC.
The disruption was traced back to a spike in memory and connection usage in our database system during a routine background task. While this activity was running, it caused the system responsible for processing marketing email jobs to stop consuming new messages, effectively delaying all campaign sends.
Marketing email campaigns were delayed for ~39 minutes. Transactional emails sending continued without interruption. Other platform features remained fully operational.
8:30 AM – A routine background task was initiated on our one of the database system.
10:00 AM – A significant increase in system resource usage (memory, CPU, connections) was observed.
10:10 AM – The system began experiencing delays, and marketing email sending was impacted.
10:30 AM – Connections reached their peak, further contributing to the blockage.
10:39 AM – Marketing email sending fully resumed.
10:50 AM – The background task was completed and system metrics began to return to normal.
We are reviewing and optimizing background task scheduling to prevent resource spikes during peak hours.
Database connection limits have been temporarily increased as a safeguard.
Additional monitoring alerts will be implemented to detect similar issues faster in the future.
We sincerely apologize for the inconvenience this may have caused. We understand the importance of timely email delivery and are taking steps to ensure improved reliability moving forward.