We are seeing connectivity to one of our switches in Frankfurt. We are investigating.

Wed, 21 Jun 2017 11:30 AM UTC: We are still working on this. We have tried various ways to restore connectivity, including reverting the changes made during the recent maintenance. We are working through some options to get connectivity restored as quickly as we can.

Wed, 21 Jun 2017 14:26 PM UTC: We are aware there is still packet loss to some servers. We are working to resolve that.

Wed, 21 Jun 2017 15:06 PM UTC: Shortly after the last update we made a change that seems to have resolved the packet loss issue. We are seeing normal levels of packet loss and throughput again. We will continue to monitor it closely. A full outage report will be posted once we have gathered all the details.

Thu, 22 Jun 2017 03:10 AM UTC: An outage report is below.

Around 0830UTC we were alerted to ports on our core flapping. This was seen to be caused by a member connection to one of the top of rack switches (ToR). The link was disabled to force traffic over the redundant link, but that link started flapping as well.

We had datacenter staff check the fibre connection, and try a replacement link as well, with no change. At around 0930 the switch became unresponsive and was power cycled. Once it came back we disabled all uplinks and began the transition to alternative (copper) uplinks. This resolved the issue with that ToR switch.

Our monitoring continued to report some some packet loss.

Further investigation found another ToR switch had a similar issue. We involved senior datacenter staff to help. Shutting down one of the redundant links would only cause the remaining link to flap more, similar to the other switch. The issue was also aggravated by the switch spontaneously rebooting itself a few times around 1200UTC.

Our emergency spare switch was the same make and model as the two with problems, so we were not confident replacing the device would resolve the issue. Instead we decided to migrate all traffic away from the fibre uplinks of the second ToR as as well. The replacement uplinks were in place by 0200 and by 0230 we were bringing them up.

Traffic levels returned to normal shortly after. The network remains stable.

Our team are reviewing logs and configurations to see if we can find the root cause. It is likely remedial work will be required, a separate maintenance notice will be scheduled should it be required

Host server maintenance (will restart VMs).

RimuHosting has received a Xen Security Advisory (XSA) that requires us to perform updates to our Xen host servers. To do this we need to stop the VMs running on these hosts, update the host servers and restart the host servers and their VMs.

These security advisories impact only customers running VMs on our Xen-based host servers.  They do not impact any customers with 'bare metal' dedicated servers.

Your VM will be unavailable during the update.  We expect the updates to take 45 minutes (typically) to 2 hours (less commonly) per host.

Since the update requires a VM restart, now would be a good time to check your setups to ensure that your server boots up as expected.  e.g. that expected services run.  e.g. that SSL certificates do not require manual pass phrase entries.

Scheduling

You can view the scheduled upgrade times at https://rimuhosting.com/cp/hsupgrades.jsp

Customers on dedicated servers can edit the scheduled time at https://rimuhosting.com/cp/vps/hsupgrade.jsp

The XSAs will be publicly released by the Xen project team on Jun 20. We must complete this maintenance before then.

Where possible we attempt to schedule updates at times that are off peak at the host server's location.

Customers on dedicated hosts can schedule updates for a time that suits (provided it is prior to the XSA release date).  Due the logistical demands of the update we are unable to accommodate customer preferences for times on shared hosts.  Customers on shared hosts can move to dedicated VM hosts should they need to control maintenance window timing.

Please subscribe to this notice to stay updated.