Impending Downtime

[Eos]

We are going to have to schedule another maintenance outage to try once again to figure out what's gone wrong with our database since moving to this new host. I know many of you are episodically getting error messages from it flaking out and it's reached a new height of defective this week, so sometime between Thursday Night and Monday Morning SP will be locking down to try a few of the alternatives I've been researching in hopes that one of them will address whatever the underlying issue is.

If none of these do it pretty much narrows the issue down to our host being fundamentally broken, and that comes with a slew of complications of it's own we'll have to address.

I'll try to narrow in on the exact "when" later this week, but this is your advance warning that it's coming.



Maintenance will be late tonight, Friday, 6-21 / Saturday 6-22 morning. Approximate window is between 4 am to 6 am.

In PDT terms that'd be 2-4, eastern 5-7.

[Jedward]

I know this was inevitable, but it's unfortunate that this has to happen right around Unleashed.

Oh well, will just have to deal with it when it happens.

[ShinkuDragon]

but where will we complain about how our dreams were shattered by KMS/GMS?

on a serious note, i haven't had any issues with southperry actually.

[MetaSeraphim]

>downtime

>happens right after Maple goes down for Unleashed

>everyone dies

[Justin]

As you said, it's recently gotten far more noticeable/worse. Usually it was only once a week, if even that, but it's happened a good 4 times today alone. Here's hoping your efforts work out in the end.

[Heidi]

Weird, I haven't had any problems at all with Southperry

[Flonne]

I've been having issues with ads not showing up, is this one of the problems of which you speak? Even disabling or even uninstalling all of the addons for disabling ads does not work.

[TrollHunter]

I've been having issues with ads not showing up, is this one of the problems of which you speak? Even disabling or even uninstalling all of the addons for disabling ads does not work.

Same problem here. It's quite sad honestly.

But hope you all can find everything that's wrong and get it all back up to snuff. Wish you the best of luck.

[Eos]

I've been having issues with ads not showing up, is this one of the problems of which you speak? Even disabling or even uninstalling all of the addons for disabling ads does not work.

Not even in the ballpark. Ads are provided by third, sometimes fifth, parties, and I have almost no control over them. Certainly nothing I can fix by taking the site offline.

This is to address issues with the database software that drives the forums and other site features.

[KhainiWest]

Karma points compensation, or a nude day.

[Flonne]

Not even in the ballpark. Ads are provided by third, sometimes fifth, parties, and I have almost no control over them. Certainly nothing I can fix by taking the site offline.

This is to address issues with the database software that drives the forums and other site features.

Well, it's not the ads themselves that aren't showing, rather, the frame they are supposed to be in is always the starving chinchilla. The mid-page ads still show.

[Eos]

Well, it's not the ads themselves that aren't showing, rather, the frame they are supposed to be in is always the starving chinchilla. The mid-page ads still show.

The chinchilla has it's own distinct frame, it specifically can't overlay the ads if the ads are in fact present. In a worst case scenario of a false positive you'd see both, as some people have demonstrated in the thread about them.

[Eos]

Note the horrible down time we had this morning was not in fact related to the next horrible downtime we need to have, apparently this was our first time being hit by significant volumes of traffic on this new host and the webserver exploded into bloody little chunks.

I rearranged things to give it better space and we'll see how it does now.

[Zerard]

apparently this was our first time being hit by significant volumes of traffic on this new host and the webserver exploded into bloody little chunks.

So it was the spike in traffic.

[dubonzi]

Only had issues with the forum on the 19th before/during GMS maintenance, since then, only a few long load times but nothing major.

[Eos]

For those of you who have seen nothing and curious what this is even about, this is the error logs of the past 24 hours, from the database dropping connections nonstop. Historically these are non-existent, since moving to this host, this has become the typical nightmare scenario;



That's ~5 mb a day of this;

Database error in vBulletin 4.1.10:
MySQL Error   : Lost connection to MySQL server during query

[Niernen]

apparently this was our first time being hit by significant volumes of traffic on this new host and the webserver exploded into bloody little chunks.

Who's blood?

Ah well, at least the hours are not too bad, I will probably be asleep anyways (i hope....)

[Eos]

This is complete.
The first software switch failed, but plan B successfully "took" and installed. No idea yet if it actually fixed the issues though, this problem has been notorious so far for appearing to go away right after a maintenance and then slowing creeping back in.

Also worth noting that due to having moved to new, untested, software we may experience some hiccups while identifying necessary tweaks to make it run optimally.

[Eos]

Huh. Well that answers that;
The new software is performing remarkably well from a CPU & RAM perspective. as in using 1/6th of what it was previously to accomplish the same thing.

It did not, however, stop throwing the bizarre errors, and I suspect there is either something physically wrong with the host it's on, something majorly corrupted in the VM OS install, or the divine wrath of a moderately irked deity has descended upon us. Having already performed an exorcism and sacrificed a chocolate bunny and bathed in the purest mountain dew to cleanse of evil influences, the next cleanest approach is to nuke the DB VM back to a clean install and rebuild it from ground zero to see if it can be reclaimed or must be consigned to a slow cruel death.

90% of the time in that process is in creating backups of the DB and moving them somewhere safe, followed by moving them back in after the fact. It will take me a while to sort the logistics on that out so in the mean time, assume another down time will be coming, eventually.

[MetaSeraphim]

or the divine wrath of a moderately irked deity has descended upon us

That would be my bad, sorry. I meant to send them to take down another forum, I'll call them off right away.