On July 29, 2025, we experienced a critical infrastructure-level disruption due to a network incident at our hosting provider OVH. The incident resulted in connectivity loss across multiple production servers, which caused cascading failures across several critical services.
The issue created ambiguity at the middleware layer while the network was not entirely down, partial connectivity led to services being partially reachable and behaving unpredictably. This made real-time diagnosis complex and required a coordinated response across infrastructure, application, and support teams.
Clients may have experienced outages, errors, and broken flows in the following areas:
Email Delivery (Transactional & Marketing): Delay in delivery.
SMS Delivery (Transactional & Marketing): Delay in delivery.
WhatsApp Messaging: Delay in transactional traffic delivered.
Tracking: Events failed or were delayed.
Contacts Application: Create/Update/Delete operations failed.
Redirection Links: Broken or unresponsive.
ma-tracker service: Fully impacted.
The source of the disruption was traced to a network failure within OVH, which affected internal communication across services.
Middleware layers received inconsistent connectivity, creating confusion in service behavior and complicating diagnosis.
The partial network availability led to some services appearing operational while others failed silently or experienced heavy delays.
Incident Bridge Activated: Our SRE and platform teams, coordinated in real-time.
Root Cause Identified: Ticket opened with OVH for the incident.
Service-by-Service Restoration: Systems recovered in stages, with automatic retries and manual interventions.
Post-Recovery Monitoring: Teams opted to keep the incident open for extended observation to ensure full service stability before formal closure.
We sincerely apologize for the disruption and appreciate your patience as we worked to resolve the issue. We understand how critical real-time communication and campaign tools are to your business, and we remain fully committed to delivering the reliability and transparency you expect from us.