Major outage affecting all applications on EU + NA Data Centres

Root cause
The network interruption caused by the configuration change lasted approximately 30 seconds — within the normal, expected range for this type of network change. However, our cluster’s heartbeat threshold (the maximum time nodes may lose contact with each other before being considered failed) was set to 10 seconds, well below that duration. Once the interruption exceeded this threshold, the two hosts were considered to have lost cluster membership. As a built-in safety mechanism to prevent data inconsistency, the cluster automatically shut down all virtual machines on the affected hosts rather than resuming them once connectivity returned — this is why recovery required each service to be restarted, rather than resuming immediately once the network issue itself was resolved.

Resolution and remediation:

• The Hyper-V cluster heartbeat threshold has been increased, so that brief network interruptions of this length no longer cause hosts to lose cluster membership.
• Network configuration changes of this type (switch topology / Spanning Tree Protocol changes) are now classified as critical operations.

We apologize for the disruption and any inconvenience this caused.

In parallel, we are launching an investigating to thoroughly investigate the root cause to ensure we apply all necessary remedial and mitigating patches and fixes to ensure this does not reoccur.

A post-mortem will be posted under this incident in the near future to give you all more details about what went wrong and how we addressed the underlying issues.

We have just recovered from a major downtime event that took all of our applications offline for clients hosted on the EU and NA Data Centres. We sincerely apologise for the impact our clients faced during this period, and we are working flat-out to monitor for any further instability.