About The Major Site Outage Yesterday
Our web hosting company’s data center experienced two power outages beginning at approximately 4:00 p.m. on Thursday, November 20, 2008. At approximately 4:00 p.m. MST, the data center made a switch to a generator. The data center was able to transition to generator power but they experienced several minutes of downtime where the battery backup system did not fill in and the internal power grid went down during the transition. The battery backup system did not fill in because one of its batteries exploded.
As of 8:00 p.m. Thursday evening, Idaho Power Company was still dealing with several outages across multiple grids. The Bodybuilding.com team worked to bring everything back online as quickly as possible but due due to the power being cut while all services were running it effected most of our systems.
Many of the systems (databases) had to be validated, repaired and tested as well as many of the file systems scanned which to many hours. Our store was the first back up at approximately 6:45 p.m. and all other systems (such as the SuperSite, forums, BodySpace, etc.) following that with the last becoming available at 2:45 a.m., Friday, November 21, 2008. At 9:00 a.m. MST this morning, the data center was going to switch from generator to conditioned utility power. As a precaution to keep our system from taking another power hit, we stopped all critical services for approximately 20 minutes during the transition.
Steps are being taken to make sure it doesn’t happen again.
P.S. it really sucked. 
November 21, 2008 - 5:53 pm MST at 5:53 pm
Glad everything is back up and running!
November 21, 2008 - 6:04 pm MST at 6:04 pm
yes it did suck
i was going crazy without bodyspace!
November 21, 2008 - 6:19 pm MST at 6:19 pm
Sorry to hear about that and thanks for keeping everyone informed. As someone who worked in IT I understand the incredible stress and pressure you guys were under. I love Bodyspace and you all do a fantastic job of providing an awesome website!
Belinda
November 23, 2008 - 6:24 am MST at 6:24 am
Things are still broken, though. I posted a new Body Blog entry last night and though I clicked "save," it didn’t… it’s not there. ;( Also, I updated my measurement stats last night and while those "took" the screen says that they were updated on the 19th, not the 22nd. Visitor counter to my body blog has been broken since late August. Body points haven’t worked since last spring some time. What gives, guys?
December 17, 2008 - 5:30 pm MST at 5:30 pm
As a person who works with computers and there back up system. I am curious why the battery blow up? Was a there a lot dmaged from the battery?