host102.rimuhosting.com is showing some unusual output in the kernel logs, and customer VPS management there is currently offline.

We are preping a replacement server with a more recent version of Xen and LVM. Once that is ready we will schedule a migration of the all VPSs.

Update 0050 UTC: Replacement host server is ready. We will start moving customers to the new machine around 0230am UTC.

Update 0230 UTC: Starting on the migration now.

Summary

There was a power outage affecting some servers (about 20%) at the Dallas data center.  The power came back on quickly, but one switch did not work correctly after the issue causing one cabinet of servers to be inaccessible.  That switch/cabinet issue is now resolved.

We are working on collecting more information about what happened and will post that information here as soon as we have it.

We note that Rackspace (also in Dallas) had a power outage yesterday.  This outage appears completely unrelated.  Other than the fact that providing reliable power to buildings filled with power hungry servers is a technically challenging task.

We sincerely apologize for any inconvenience caused to affected customers in Dallas.  We work hard to provide a reliable service and part of that is to be very careful choosing the data centers we host our servers with.  We note that this is the first power issue we have experienced in the 5 years we have been at this data center.  We are confident that they are taking the outage seriously and will do all they can to avoid issues in the future.

Initial data center report

The UPS failed.  Right now all parts are here for the UPS and they are installed.  They will be bringing it up shortly.  The UPS vendors feel everything is fine, but when you switch load back to the UPS you always have a slight, very slight, chance that there will be an issue.

That UPS has worked flawlessly for 6 years and had all maintenance performed as recommended by the vendor.  The UPS failed.  And the UPS failure caused power interruption for about 10 minutes, then we had to turn PDUs back on. This far exceeds the expectation for all UPS manufacturers.  To totally avoid the chance that it can happen again is to get redundant feeds for all power feeds off different UPSs and PDUs. We already have different Generator and Services at the entrance.

Detail

We've detected a networking issue in the Dallas datacenter.  Multiple servers are affected and we're investigating the problem right now.  Updates are forthcoming.

@Update @1420UTC: It seems to be an electrical problem at the Dallas data center. We're expecting more updates soon.

@Update @1433UTC: We are starting to see some of the host servers in Dallas booting up.

@Update @1455UTC: The power is completely back up at the data center. The technicians are cleaning up after the networking issues left behind.

@Update @1710UTC: It appears there is one cabinet/switch that is not accessible.  We believe it is a switch configuration issue.  And we are working on resolving that.

@Update @1845UTC: The switch issue is resolved.  There are a handful of serves still not pinging and we are investigating these individually.

Affected host servers

The following servers were affected (does not included dedicated servers, just VPS host servers)

host32.rimuhosting.com
host102.rimuhosting.com
host103.rimuhosting.com
host104.rimuhosting.com
host109.rimuhosting.com
host111.rimuhosting.com
host112.rimuhosting.com
host113.rimuhosting.com
host114.rimuhosting.com
host117.rimuhosting.com
host118.rimuhosting.com
host121.rimuhosting.com
host125.rimuhosting.com
host130.rimuhosting.com
host131.rimuhosting.com
host132.rimuhosting.com
host133.rimuhosting.com
host134.rimuhosting.com
host173.rimuhosting.com
host186.rimuhosting.com
host193.rimuhosting.com
host531.rimuhosting.com
host534.rimuhosting.com
host556.rimuhosting.com
host557.rimuhosting.com
host716.rimuhosting.com
host721.rimuhosting.com

We are seeing heavy packet loss to host596. We are currently investigating this and will post updates here.

UPDATE: It turned out to be a DDOS attack on the network. It should be under control and your VPSs should all be accessible now.

Our monitoring has detected that some machines behind a particular switch are not responding. We are investigating.

Update: The switch was having trouble negotiating a trunk, causing loss of connectivity to the machines attached to it. This has been fixed and all machines are pinging.

host44.rimuhosting.com restarted itself.  We are investigating.

Update: Server is up and VPSs are all running. Seeing a disk repair completed on startup and raid array resyncing. We will continue to monitor the server health.

host103.rimuhosting.com is showing a device issue on it backup drive. We have scheduled a restart for this at 615am UTC to clear the issue.

Update 0635 UTC: Server is back up and VPSs are restarting.
Update 0650 UTC: All VPSs are back up and responding normally. We will continue to monitor the server.

host122.rimuhosting.com is not responding normally.  We are investigating and may have it restarted.

Update: The PSU appears to have failed in Host122. We are having it swapped out.
Update: It took longer than expected to get the PSU swapped (we had trouble finding a suitable spare). The swap has now been completed and VPSs are coming back up.
Update: All VPSs are back up and running.