As many of you had noticed, Southperry died approximately 5:15-5:45 AM CST or so and was completely inaccessible for most of the day.
This was due to the disk space running out on the server for an unknown reason. Whenever the disk fills up on the server, the webserver will refuse to answer all requests until the disk is sufficiently freed up (since it needs to be able to write the logfile and with the disk being full, it can't do that).
Unfortunately, this condition also locked the admins out of the remote terminal (since it needs disk space to log as well), which prevented us from clearing the disk out ourselves, and forced us to contact the host for assistance, which at that hour wasn't possible until around 8 AM CST or so, and since I had to go to work shortly thereafter, I was not available to restart anything once the issue was repaired.
I returned home from work shortly before 4 PM CST this afternoon, and found the server had been repaired and only needed to have its services restarted (web, IRC [terra.southperry.net], etc), so I promptly did just that.
So. If my count's correct...Southperry was down for about 11 hours...which I know to a lot of you is truly unacceptable and I do apologize for not being around to get this back up...but I can't simply call off going to work for this, I'd get fired if I did that.
Sorry you guys had to wait that long for me to get things going again. I'll be working on a few countermeasures in the background to ensure this doesn't happen again. Thanks for your patience and welcome back!
BDRyan
Southperry.net Systems Admin
Bookmarks