Partial Outage across multiple systems

Post-Incident Review: Partial Service Outage

A networking hardware fault at our data centre provider caused a partial service outage affecting customers hosted on the impacted SQL Server.

Customers with live replicas were restored using their replica environments. Customers without a live replica remained unavailable until the data centre provider resolved the networking fault.

The issue was traced to a failed network interface module carrying public internet traffic. Although the upstream switch port remained online, the server stopped accepting traffic after an earlier link interruption. The provider manually disabled the affected port to stabilise connectivity and restore service.

The suspected faulty SFP module will be replaced, and our data centre provider are reviewing additional monitoring and resilience measures to improve detection and recovery from similar hardware failures in the future.

The issue has been discovered and resolved. Thank you for your patience.

We are currently experiencing performance degradation and intermittent availability across several systems. Some features may be slow or unavailable. Our team is investigating the root cause.