MTK
02-23-2010, 09:38 AM
Damn that was a rough couple of days, but we're back up now. Here's what happened:
Dear WestHost Clients,
On Saturday, February 20th at approximately 2:20 p.m. MDT, we experienced some hardware failure as a result of an annual fire system inspection at our Data Center (DC). An inadvertent release of Inergen was triggered in the DC environment. The release was triggered by an actuator that was not removed as required in the DC's pre-test checklist. As such, the entire DC was impacted at some level, however not all servers were affected. This error was not the result of a mistake by WestHost employees, our hardware or systems. The mistake was triggered by an error from a vendor. With that said, we take responsibility and are doing everything we can to restore all systems completely.
There were multiple servers impacted by this outage; some of the servers have had damaged hardware replaced while other servers need to be restored from backups. A large majority of the servers that were affected have since had all services restored and are back online. We still have a few remaining servers that are experiencing problems. We are doing everything we can to restore those remaining servers as quickly as possible.
In the future for updates if the site goes down, follow us on Facebook (http://www.facebook.com/home.php?#/profile.php?ref=profile&id=100000658935212) or on this blog page (http://redskinswarpath.blogspot.com/).
Dear WestHost Clients,
On Saturday, February 20th at approximately 2:20 p.m. MDT, we experienced some hardware failure as a result of an annual fire system inspection at our Data Center (DC). An inadvertent release of Inergen was triggered in the DC environment. The release was triggered by an actuator that was not removed as required in the DC's pre-test checklist. As such, the entire DC was impacted at some level, however not all servers were affected. This error was not the result of a mistake by WestHost employees, our hardware or systems. The mistake was triggered by an error from a vendor. With that said, we take responsibility and are doing everything we can to restore all systems completely.
There were multiple servers impacted by this outage; some of the servers have had damaged hardware replaced while other servers need to be restored from backups. A large majority of the servers that were affected have since had all services restored and are back online. We still have a few remaining servers that are experiencing problems. We are doing everything we can to restore those remaining servers as quickly as possible.
In the future for updates if the site goes down, follow us on Facebook (http://www.facebook.com/home.php?#/profile.php?ref=profile&id=100000658935212) or on this blog page (http://redskinswarpath.blogspot.com/).