We are seeing connectivity to one of our switches in Frankfurt. We are investigating.

Wed, 21 Jun 2017 11:30 AM UTC: We are still working on this. We have tried various ways to restore connectivity, including reverting the changes made during the recent maintenance. We are working through some options to get connectivity restored as quickly as we can.

Wed, 21 Jun 2017 14:26 PM UTC: We are aware there is still packet loss to some servers. We are working to resolve that.

Wed, 21 Jun 2017 15:06 PM UTC: Shortly after the last update we made a change that seems to have resolved the packet loss issue. We are seeing normal levels of packet loss and throughput again. We will continue to monitor it closely. A full outage report will be posted once we have gathered all the details.

Thu, 22 Jun 2017 03:10 AM UTC: An outage report is below.

Around 0830UTC we were alerted to ports on our core flapping. This was seen to be caused by a member connection to one of the top of rack switches (ToR). The link was disabled to force traffic over the redundant link, but that link started flapping as well.

We had datacenter staff check the fibre connection, and try a replacement link as well, with no change. At around 0930 the switch became unresponsive and was power cycled. Once it came back we disabled all uplinks and began the transition to alternative (copper) uplinks. This resolved the issue with that ToR switch.

Our monitoring continued to report some some packet loss.

Further investigation found another ToR switch had a similar issue. We involved senior datacenter staff to help. Shutting down one of the redundant links would only cause the remaining link to flap more, similar to the other switch. The issue was also aggravated by the switch spontaneously rebooting itself a few times around 1200UTC.

Our emergency spare switch was the same make and model as the two with problems, so we were not confident replacing the device would resolve the issue. Instead we decided to migrate all traffic away from the fibre uplinks of the second ToR as as well. The replacement uplinks were in place by 0200 and by 0230 we were bringing them up.

Traffic levels returned to normal shortly after. The network remains stable.

Our team are reviewing logs and configurations to see if we can find the root cause. It is likely remedial work will be required, a separate maintenance notice will be scheduled should it be required