Extended Outage

Long story short, an entire street of data centers in Atlanta lost power and every host, and thereby every hosting company and every hosted site, went out. The outage was long enough that UPS' couldn't have sustained them but could've bought them time to do graceful shutdowns, had they had any, and of course any major host normally has actual backup generators to run indefinitely during outage, but apparently our latest had nothing whatsoever, so we went down hard, for a good six hours, and when we came up the database was understandably upset and had to be talked back down off a ledge before it'd run.

Once the database was restored we were still dead though because the ipv6 bridge between our web server and db server stopped functioning. Somewhere in the provider's network is a piece of equipment someone forgot to turn back on I think, so that had to be hunted down and traced through to determine whether it was something that died in our configs or an actual network issue as it was, and then we had to reroute everything back to ipv4 until it can be corrected.

Now we're back up and semi running while I try to find any gaping holes in our data other than whatever was occurring at the exact minute we died.

Oh I know that feel. My power went off yesterday as well.

Good timing! I was refreshing this site a few times and I'm glad is back up :) Waited a couple of minutes when it was down.

oh, so that's why I tried to access the site since 6 hours ago and failing miserably.
Glad the site's back up!

Time to get back on my addiction to this site U_u

Quote:

---

Originally Posted by ayumilove

Good timing! I was refreshing this site a few times and I'm glad is back up :) Waited a couple of minutes when it was down.

---

Quote:

---

Originally Posted by SoR0XaS

oh, so that's why I tried to access the site since 6 hours ago and failing miserably.
Glad the site's back up!

---

protip : to check if a site's down use www.isup.me

Quote:

---

Originally Posted by Five Second Pose

protip : to check if a site's down use www.isup.me

---

I prefer http://isitup.org/

A better explanation of how exactly this managed to happen for those who were as baffled as I;

Quote:

---

The cause of the loss of utility power is still being investigated. Approximately 35 minutes after the datacenter switched to generator power the generator failed, and subsequently the UPS battery backup power was exhausted about 15 minutes later. Approximately 2 hours after this occured, the generator was repaired and services were mostly online overnight until around 11 am local time, when the generator experienced another failure. Approximately 1 hour later utility power was restored to part of the datacenter, including the part that we are in, and we have been online since that time

---

And as a nice relief;

Quote:

---

As this is the most severe outage our Atlanta location has experienced in the last 3 years since we launched services here, we have decided to credit all clients in Atlanta with 1 free month of service, which should be reflected right now in your service next due date. Even though we did not cause this outage we reconize that any failure of this magnitude regardless of fault is unacceptible and we take full responsibility for the outage you have experienced yesterday evening and again this morning.

---

That'll definitely help us with the recent issues we've had with advertising outages and whatnot on top of this.

It's refreshing to see customers treated as customers

Quote:

---

Approximately 35 minutes after the datacenter switched to generator power the generator failed, and subsequently the UPS battery backup power was exhausted about 15 minutes later. Approximately 2 hours after this occured, the generator was repaired and services were mostly online overnight until around 11 am local time, when the generator experienced another failure.

---

I hope they're shopping for a new generator.