Update at 01:45 AM UTC: B3 switch stack needed to be restarted.  We are looking at provisioning a new core networking stack to replace it over the next few months.

1:28 am, 26 July 2024 UTC: Issue has been resolved, service restored.

We have detected an issue with networking.  Remote hands has been dispatched to the data center to troubleshoot and resolve.

Wed 24 Jul: Datacentre has provide initial RCA for outage with finalized one been ready on or before the 31 July

An issue was found with a utility transformer. The datacenter switched to generator load for that feed to allow for repairs to be done. The generator later had a belt failure, which caused it to shut down. Dependent UPS units took the load but discharged fully before the generator could be restored. A general outage occurred on related circuits. Power was restored once the generator was repaired. After the power was restored some RimuHosting switches were left unresponsive. RimuHosting moved servers, and replaced switches to ensure that servers were accessible.

Thurs 18 Jul: Update from the data center:

TierPoint experienced a utility transformer failure that required a full replacement of a 3750kVA transformer and had two service entrances on generator power from last Wednesday, July 10 through July 15 at 9pm. During this time, we did experience additional component failures that impacted service to a subset of customers. With the event now complete, TierPoint will begin work on the RCA today and plan to have it complete and distributed to impacted customers by the end of this week (July 19th).

Wed, 16 Jul 2024 21:30 PM UTC: Site reports transformer has been replaced and all feeds are now fully connected. Site te is once again fully redundant and all connections are stable

Thu, 11 Jul 2024 22:07 PM UTC: Site remains stable and all services are normal. Affected feeds are still running from generator. The failed transformer is expected to be repaired by mid-Friday Dallas time after which point utility power will be restored. Meantime we are continuing to work on resolving power redunancy for some affected gear.

Thu, 11 Jul 2024 08:38 AM UTC: Systems seem stable for now, we will continue to monitor. Further updates will be made only if there is new news. We will be working on an incident report which will be available to customers by request next week.

Thu, 11 Jul 2024 06:30 AM UTC: Second switch replacement done. We are working on a couple of standout servers to fix hardware issuse, but otherwise services should be back online. Let us know if you are still seeing an issue

Thu, 11 Jul 2024 05:23 AM UTC: First switch replacement done, some servers are back already, validating. Switch replacement for 4.1.1029 underway

Thu, 11 Jul 2024 05:10 AM UTC: switch replacement for 4.1.6034 is configured and is being swapped in now.

Thu, 11 Jul 2024 04:34 AM UTC: Some customers have an affected server () in their /etc/resolv.conf.  This may be impacting DNS looks from your VM.  This script will remove that name server in the event it is not working:
wget -O fixdns http://72.249.185.185/fixdns
bash fixdns --removebad

Thu, 11 Jul 2024 04:10 AM UTC: Selected server moves are done and those particular servers are responding. Replacement switches being configured to resolve access issues for the remaining services.

Thu, 11 Jul 2024 03:30 AM UTC: Remote hands has almost completed moving the selected servers, they are just running network cabling for those now.

Thu, 11 Jul 2024 02:32 AM UTC: We are waiting on remote hands to progress a queue of work needed to restore services. Their workload is very high right now, we are working with them to keep things moving. A further update will be provided in the next hour or if we get news sooner.

Thu, 11 Jul 2024 01:18 AM UTC: To restore services, we are preparing replacement switches for two cabinets (4.1.6034 and 4.1.1029). And moving servers from 4.1.6035 to an alternate cabinet.

Thu, 11 Jul 2024 00:30 AM UTC: Several switches have failed after the power issue. We are working on a plan to replace those from our spares and may also move some affected servers to working cabinets.

Wed, 10 Jul 2024 23:57 PM UTC: We are continuing to work with remote hands to resolve issues with several unresponsive switches. We have identified spare devices in the event replacments are needed.

Wed, 10 Jul 2024 23:02 PM UTC: Generator power and UPS feeds have been restored. We are continuing to work with remote hands to check impacted services

Wed, 10 Jul 2024 21:56: The fault has been located on generator three, and is actively being repaired. HVAC's have been re-fed and are online. More updates will be provided as they become available.

Wed 10 Jul 2024 21:45: Dallas data center is also having a generator issue on site. They have electricians currently working at site to resolve the issue. Downtime is expected until 15:45 CST

Wed, 10 Jul 2024 20:29 PM UTC: We have been advised there was an issue with a transformer. Power was failed over to permit maintenance, but several cabinets have become inaccessible despite redundant feeds to those. We are working with remote hands to resolve.

We are seeing packet loss in Dallas in a couple of cabbinets and are investigating that

Fri, 12 Jul 2024 05:01 AM UTC: This has been completed

Wed, 10 Jul 2024 23:36 PM UTC: This work is underway, we are seeing loss of power redundancy as expected but no other issues reported.

There is scheduled UPS maintenance being run at ourr Brisbane site. No impact to avilability is expected, however servers may loose power redundancy while the work is underway.

Detail: Facility staff and approved contractors will be isolating UPS 1B followed later by UPS 1A to allow for the installation of a new submain and controller. This project will allow the site to increase load capacity.

When, Brisbane time:
UPS 1B: THURSDAY, 11 JUL 07:00 - THURSDAY, 11 JUL 20:10
UPS 1A: FRIDAY, 12 JUL 07:00 - FRIDAY, 12 JUL 23:00