Summary

There was a power outage affecting some servers (about 20%) at the Dallas data center.  The power came back on quickly, but one switch did not work correctly after the issue causing one cabinet of servers to be inaccessible.  That switch/cabinet issue is now resolved.

We are working on collecting more information about what happened and will post that information here as soon as we have it.

We note that Rackspace (also in Dallas) had a power outage yesterday.  This outage appears completely unrelated.  Other than the fact that providing reliable power to buildings filled with power hungry servers is a technically challenging task.

We sincerely apologize for any inconvenience caused to affected customers in Dallas.  We work hard to provide a reliable service and part of that is to be very careful choosing the data centers we host our servers with.  We note that this is the first power issue we have experienced in the 5 years we have been at this data center.  We are confident that they are taking the outage seriously and will do all they can to avoid issues in the future.

Initial data center report

The UPS failed.  Right now all parts are here for the UPS and they are installed.  They will be bringing it up shortly.  The UPS vendors feel everything is fine, but when you switch load back to the UPS you always have a slight, very slight, chance that there will be an issue.

That UPS has worked flawlessly for 6 years and had all maintenance performed as recommended by the vendor.  The UPS failed.  And the UPS failure caused power interruption for about 10 minutes, then we had to turn PDUs back on. This far exceeds the expectation for all UPS manufacturers.  To totally avoid the chance that it can happen again is to get redundant feeds for all power feeds off different UPSs and PDUs. We already have different Generator and Services at the entrance.

Detail

We've detected a networking issue in the Dallas datacenter.  Multiple servers are affected and we're investigating the problem right now.  Updates are forthcoming.

@Update @1420UTC: It seems to be an electrical problem at the Dallas data center. We're expecting more updates soon.

@Update @1433UTC: We are starting to see some of the host servers in Dallas booting up.

@Update @1455UTC: The power is completely back up at the data center. The technicians are cleaning up after the networking issues left behind.

@Update @1710UTC: It appears there is one cabinet/switch that is not accessible.  We believe it is a switch configuration issue.  And we are working on resolving that.

@Update @1845UTC: The switch issue is resolved.  There are a handful of serves still not pinging and we are investigating these individually.

Affected host servers

The following servers were affected (does not included dedicated servers, just VPS host servers)

host32.rimuhosting.com
host102.rimuhosting.com
host103.rimuhosting.com
host104.rimuhosting.com
host109.rimuhosting.com
host111.rimuhosting.com
host112.rimuhosting.com
host113.rimuhosting.com
host114.rimuhosting.com
host117.rimuhosting.com
host118.rimuhosting.com
host121.rimuhosting.com
host125.rimuhosting.com
host130.rimuhosting.com
host131.rimuhosting.com
host132.rimuhosting.com
host133.rimuhosting.com
host134.rimuhosting.com
host173.rimuhosting.com
host186.rimuhosting.com
host193.rimuhosting.com
host531.rimuhosting.com
host534.rimuhosting.com
host556.rimuhosting.com
host557.rimuhosting.com
host716.rimuhosting.com
host721.rimuhosting.com