Order VPS Hosting
Order a VPS, Semi- dedicated or Dedicated server in Dallas, London or Australia.

Get Assistance
Ask our support team about your hosting requirements.


Host where the staff takes pride in making customers happy

I just had a client's server go down during an upgrade of a bunch of packages with yum. After the upgrade of about 150 packages, I decided to reboot just to make sure it survived. It didn't come back.

Before I could get my clients account info to send you a support ticket, you guys had identified the problem, sent me an SMS to my phone to let me know you were working on the issue. Before I could get my laptop open and connected to ping it, you had the server back up.

I wish I could have given you a clue as to the nature of the problem, but you beat me to it!

- David (problems happen, we work to respond pro-actively when they do) (#55/333)
Home > Support > Notices > New York Datacenter Issue (September 26)

Related Links

Notice Links:

Notice

New York Datacenter Issue (September 26)

PostedThu, 25 Sep 2008 22:44 PM UTC
Last UpdateThu, 9 Oct 2008 19:55 PM UTC (693 weeks ago)
StatusClosed

Summary: There was a power outage at the New York datacenter.  All servers should now be back up and running and any issues resolved (else let us know).

Original updates are below:

Update @ Friday, September 26, 2008 2:44:38: It appears that the NY facility has lost power. We are waiting on updates from staff there. Once power is restored we will get machines and VPSs back up and running ASAP.  We do not have an ETA on that just yet.

Update: Servers are starting to come back online, and others are being powered up in succession to prevent power circuits from overloading. We are working to insure that servers come up safely.

Update: The datacenter staff report they are working bring up some of their firewalls that are currently down, as well as the remaining machines that are still down.

Update @ Friday, September 26, 2008 6:24:55 AM UTC: Some private networks are not working at this time. One of the switches used for private networks has failed and there are no spares on hand. Datacenter staff will let us know when a replacement has arrived, however they have no ETA at this time.

Update Friday, September 26, 2008 10:53:39 AM UTC: Most machines are back up and running after a routing issue was resolved. There are still a few machines down (some with hardware issues) that we are working through.

Update Friday, September 26, 2008 12:34:30 PM UTC: Datacenter staff located another routing issue. After resolving this more machines are back online. Most machines that are still down have been checked and have hardware faults that we are continuing to troubleshoot.

Update Friday, September 26, 2008 5:34:39 PM UTC: All machines are back up and running.

Update Sep 30: The data center report:

Late Thursday evening, we lost utility power due to weather-related conditions (lightning strike according to one report), which caused the entire Brooklyn facility to switch to generator.

Once the fuel in the day tank was exhausted, because of a malfunction in the gauge, the fuel pump did not kick on automatically, and transfer fuel from the storage tank.

This caused the generator to shut down, even though there was enough fuel in the storage tank to last several days.

We have addressed this situation, and are supplementing the alert function of this gauge and transfer pump to ensure this does not occur again.

We did transfer over to generator power several times during the days pursuant to last Thursday as we had intermittent outages of utility power, and the generator has been functioning properly.

We sincerely regret the inconvenience caused to you and your customers, and assure you that we are working to avail more robust facilities.

#

Keep You Updated?

Log in to subscribe to changes to this notice.

Set your contact details for future notifications.