Home > Control panel > Operations notices > Power issue affecting a cabinet in Dallas

Related Links

Notice Links:

Notice

Power issue affecting a cabinet in Dallas

PostedMon, 8 Aug 2011 12:44 PM UTC
Mon, 8 Aug 2011 07:44 AM CDT
Last UpdateWed, 10 Aug 2011 06:51 AM UTC (695 weeks ago)
Wed, 10 Aug 2011 01:51 AM CDT
StatusClosed
Affected Data CenterDallas (TX)

Current Status: There is an issue with a power distribution unit affecting one of our cabinets in Dallas, which has caused one of our switches and several servers to lose power.  Power has been restored but further repairs are needed which will cause some brief downtime, sometime between 8/09/2011 23:59hrs until 8/10/2011 0600 CST:

Maintenance should be completed by: http://timeanddate.com/s/23f7

We will continue to post updates below:
===

Multiple servers in the Dallas data center not reachable. We are currently looking into it. We'll post updates on the issue here.

UPDATE: Looks like it's a power problem with one of the cabinets. The data center staff is looking into it.

UPDATE from our data center: We experienced a localized issue on one of our PDU's – PDU 25, panel 5.  We are working with on site electricians and vendors to make emergency corrective actions. As more information becomes available we will keep you informed.

UPDATE: Some servers are now coming back up. The electricians on site are continually working to bring all of them back up.

UPDATE: All servers should be back up now. We'll look into this further and post details here.

Update 2140 UTC: a PDU supplying power to the affected cabinet lost phase C on one of its main breakers, causing the outage. Emergency maintenance has been scheduled to replace that component as it is no longer trusted. We will post scheduling details for affected customers on that shortly.

Update 2205 UTC: The same servers affected by the original outage will loose power during the following window. 8/09/2011 23:59hrs – 8/10/2011 0600 hrs CST. We are presently investigating solutions to reduce that downtime.

Update 0040 UTC: The maintenance window is for worst case, all thing going to plan the downtime should be significantly less. The main switch in that cabinet will be affected by the outage, that will be swapped to the other circuit right away to restore connectivity to servers still powered up. And so that the maintenance coincides.

Update 1700 UTC: Confirming maintenance is starting in a few minutes. We are preemptively stopping some VPS hosts to insure data integrity.

Update 1800 UTC: Normal power has been restored. We are working to bring servers back online.

Update 1830 UTC: All service should be back up and running. If you are still seeing an issue please pop in a ticket with details and we can take a look.

#

Keep You Updated?

Log in to subscribe to changes to this notice.

Set your contact details for future notifications.