There appears to be an issue at the New York datacenter causing loss of connectivity. This affects servers with an IP address in the 66.199.*.* and 72.9.*.* range.

We are investigating.

Update: Datacenter staff report it is a power issue. They are working to restore systems.

Update: Around half of the machines at the New York datacenter are back online. We are working to get the rest back up.

Update @0117 UTC: the data center report that power is back for all the servers.  But the power issue has affected a number of switches and firewalls.  They are replacing those now.  They did not give us an eta, but they are working on it and trying to resolve things as quickly as they can.

Update @219 UTC: Problem still ongoing.  As switches are replaced we are seeing servers coming back up.  i.e. some progress is being made.

Update @325 UTC: Most machines are now back up and running. We are working with the datacenter to get the remaining machines online.

Update @600UTC: All but 10 machines are back online. The majority of these have bootloader or other issues that require further attention to get back up and running. We are continuing to work on these.

Update Feb 16 @926UTC: All machines are back up and running. We will be following up with the datacenter to find out what the exact cause of the issue was, and what fixes will be implemented to prevent further outages.

Update: We received the following brief update from the data center.  We are still awaiting a detailed report:

There were issues outside the data center (some kind of flooding).  The failover generator had not started.  RimuHosting is not sure why not.  There were 'other problems' (RimuHosting is waiting to find out what those were).  The data center is installing an extra APC power array to help avoid this issue in the future.

Update: Feb 26: The data center provided us with the following update from FiberMedia, the facilities management company that provides the power to the data center building.  They report:

On Saturday 2/9/2007 the FiberMedia data center experienced a power loss. Some of our customers were impacted by the outage.

It has been determined that there was a loss in voltage from our primary feeds to the UPS systems that we have in place. This voltage issue caused the UPS system to go off-line and since there was not a loss of power being delivered to the facility the generator did not activate.

We have modified the thresholds on the UPS systems to avoid such a disruption in the future.

On Thursday 2/13/2008 the FiberMedia data center as well as the entire Industry City complex (13 buildings) experienced a power loss due to a direct power loss to the feeders providing power to the sub-station. This loss occurred in conjunction with a UPS fan malfunction on the UPS unit causing it to go into bypass mode. Our generator did not operate due to a manifold issue that caused the self presevation mode to activate and therefore it would not start at the time of the UPS bypass.

We have had a FiberMedia MEP Engineer on site working closely with the building engineers and APC UPS engineers analyzing the power loss and to ensure power to the data center. Corrective measure have been taken and completed including the replacement of the UPS fan units, a complete invasive scope review of the generator and the replacement of two after-coolers in the heat exchange system within the manifold.

FiberMedia has also increased the Preventative Maintenance programs on all units within the center to ensure power to our clientele.

Addressing customer concerns is our top priority. In the days since the incident occurred, we identified and corrected UPS system issues and are taking steps to prevent these types of problems from happening again.

FiberMedia is ALSO in the process of:

1 Adding 1000kVA UPS systems to the facility along with 2 additional PDUs (3/31/2008 completion date).

2 Adding an additional 16kVA UPS system to help balancing the output current (2/18/2008).

3 Provisioning a dual backup generator set design (Completed).

4 Modifying UPS voltage thresholds (2/15/2008).

5 Industry City has added redundancy to their substation feeders (2 feeders to 4 feeders- completed).

While we do routinely PM our systems and did have APC review our UPS systems; we have modified our MEP systems to better support our clients today and in the future. We will continue to update the facility with the best and latest systems and technology to support your requirements.

We apologize for any inconvenience that these outages may have caused and will ensure that we will provide the best possible service for your systems and business.