2011-10-11 Outage / Instability

[Eos]

This is the official thread to discuss the current status of the servers, channels and general Nexon connectivity.



Red Blinking = All logins down, or master login down.
Red = More than 50% of the normal channels are down but logins are fine.
Yellow = Less than 50% of channels are down and logins are fine.
Green = All good.


Latency is now indicated in the report, including max and average.
Channels that are up but experiencing a higher than normal latency will be indicated in Yellow, but poor connection speed will not impact the status results indicator.

Note the contents of this thread will routinely be moved to the Archived Items (http://www.southperry.net/forumdisplay.php?f=9048) forum after various outages and discussion.

[Chameleonic]

Is it possible to see how many people are viewing the GMS Status page?

[Eos]

No.

[Chameleonic]

:f5:

New forum please.:f3:

[Heffalump]

I think it's certainly possible, but Eos doesn't want to do it. It'd be a drain on the speed of the site. This is why SP and SW are so much faster to load than Basil is.

[Eos]

I think it's certainly possible, but Eos doesn't want to do it. It'd be a drain on the speed of the site. This is why SP and SW are so much faster to load than Basil is.

Although right now I fear we may be nearing capacity and in need of an upgrade soon if this sort of traffic is going to keep up.

[Heffalump]

Although right now I fear we may be nearing capacity and in need of an upgrade soon if this sort of traffic is going to keep up.

True. The ferret notices are coming up much more often than they used to. I would have thought that they'd have slackened off with school starting, but apparently not, especially if Legend is coming soon.
EDIT: holy shit we have 1k again. FUN FUN FUN LOOKING FORWARD TO THE WEEKEND.

[Eos]

There we go. That should significantly improve the hit we take from the status page - People accessing it from the vanity URLs will actually hit a static copy of it and should never actually be parsed by the server or database.

(This reinforces my previous "no" btw; You can't see a count of people when the server is actively trying to not deal with as many of them as possible)

[byakugan]

Uncheduled SC coming our way?

[ChosenFruit]

Yup yup, seems like it.

Also, didn't know there was a "Maplestory is currently offline, check back later" window =o

[SaptaZapta]

Bye bye buyback?
Or is anything else going on?

[Heffalump]

Uncheduled SC coming our way?

Unscheduled SC already came our way.
Fun fun fun fun looking forward to the weekend.

[TheTempest]

Just when I was about to use 13 Beauty Coupons. I wonder what the USC is for.

[Stereo]

The anthropocentric model suggests that Nexon took down the servers because I was doing too well in Monster Park.

[Killed]

Bye bye buyback?
Or is anything else going on?

There is no "rebuy dupe". People believe anything they hear and spread it like wild fire.

[Fowski]

The Status is awesome and so useful. Thank you :heart:
Also, the hell just happened? :f6: Unscheduled Maintenance without even announcing it 10 mins before? Way to go Nex0rz.
I hope it's for good, but sigh, two uSC already. It's impossible to keep a good FM spot :f3:

[Satellite]

We can only wonder why Notify ignored this (Maybe cause this wasn't on frontpage yet?):


Dear Maplers,

We will be performing a unscheduled server maintenance beginning on September 30, 2011 to address an issue with Commitment Ceremonies. The maintenance will last approximately three hours. During this period, all game servers will be unavailable.

Thank you for your patience, and we apologize for the inconvenience.

-Time-
Pacific: 10:00 AM - 1:00 PM Friday, September 30, 2011
Eastern: 1:00 PM - 4:00 PM Friday, September 30, 2011

-What-
The following will be unavailable:
All MapleStory game servers

-Fixes-

Commitment Ceremonies will be able to be completed.


- The Maple Team -

[CarrionCrow]

3 hours to fix a feature that is mostly worthless? And here I was on a roll with my new character I'm training and all.

[Fowski]

3 hours to fix a feature that is mostly worthless? And here I was on a roll with my new character I'm training and all.
I think there's something we don't know about it. Something that requires them to hurry up. :monocle:
ring dupe exploit!!1

If not, well, it's Nexon :excellent:

[Satellite]

3 hours to fix a feature that is mostly worthless? And here I was on a roll with my new character I'm training and all.
I know man, me too.

Couldn't resist making a female char simply cause it's so easy to get a sxc look on them.


I think there's something we don't know about it. Something that requires them to hurry up. ring dupe!1

If not, well, it's Nexon :3
Dupe itself is already such a devastating hack, that if Nexon America would be a good company they would (know how to) fix it ASAP.

[iamflip]

3 hours to fix a feature that is mostly worthless? And here I was on a roll with my new character I'm training and all.

I paid 2k Maple Points for a broken feature. I deserve some kind of compensation! :f3:

But really, I do hope they fix it so that me and GF can finally get our rings and gifts and crap. lol

[Fowski]

I paid 2k Maple Points for a broken feature. I deserve some kind of compensation! :f3:

But really, I do hope they fix it so that me and GF can finally get our rings and gifts and crap. lol
What was the problem? Good I didn't buy the ticket yet.

[Eos]

We can only wonder why Notify ignored this (Maybe cause this wasn't on frontpage yet?):

Notify only checks every 5 minutes.

[ChosenFruit]

I hope one day they remember people can't see the future and warn us before doing a maintenance like they've done before =/ twice in two days with no warning is pretty bothersome.

[iamflip]

What was the problem? Good I didn't buy the ticket yet.


So... me and GF had been married way before the Amorian revamp. When the patch went live, we were eligible to do both 30 and 100 day commitments.

We do the 30 day one and that came and went... but we didn't get anything for it. No gifts, no rings. Then I saw that Jacob supplies the rings, so ok then.

I unequip my ring and ask him to exchange it. He doesn't recognize that I have a wedding ring in inventory. |:

So really... no gifts and no new ring. Epic fail. :f3:

EDIT: Bug confirmed by Nexon. (http://forum.nexon.net/MapleStory/forums/thread/8421432.aspx)

,

[Fowski]

Ugh, then I have a problem. Cuz I boomed my ring. :ohno:

[iForbin]

i wonder if they are going to fix potential and cubing. that would be nice

[ShinkuDragon]

the hell? 3 logins down for a sec and 6 channels in arcania?

[ChronosXIII]

ARGH!

Was at ~20 kills away from finishing for the day.

No notice, so I assume they just, "tripped on the wire."

[Blades4hire]

I crash randomly, and now the master login server is down?
The eff?

[Justin]

Even the website is down... Nexon got hit hard by SOMETHING....

[Fenne]

Yerp. Someone must be giving CPR to the hamster right about now :/

[Onion Knight]

Server stability -1.

[ChronosXIII]

And just like that, everything is all peachy.

Lolwut?

[Sorien]

And now it's green again. That was quite odd, I must say. :f6:

[Dark Link]

I got hit too. Was doing Monster Park with a couple buddies and I asploded.

[ShinkuDragon]

Even the website is down... Nexon got hit hard by SOMETHING....

it's called coffee.

[SaptaZapta]

Even the website is down... Nexon got hit hard by SOMETHING....

More likely it was their internet gateway hiccuping.

[Exidous]

Maybe the internet trolls tried to knock the real embodiment of evil corporate greed off the internet today.

[Panacea]

And it's out again.

[Locked]

Just crashed.

[ChosenFruit]

Poop D=< I was having a good conversation about Doctor Who!

[Blades4hire]

And...It's back
I smell dupes.

[Luxeraph]

I'll put my money on dupers, guess they got patched on how they usually did it and now are tackling harder.

[Chilly]

Probably unrelated, but did any of the rest of your servers have those advertisers with their stores of hacked items start with the Megaphones shortly after the first downing?

[Dark Link]

I'd be real surprised if it was dupers causing this. I noticed that the first time the login servers went down, Scania started slowly going down as well. Ch2, then 3, 1, and 4. Afterward they all went back up because the login servers came back up.

If this is really the work of dupers, we're royally screwed lol.
A hack so powerful it can actually knock the login servers down?

[Luxeraph]

Well it did rollback a few seconds (it was noticeable), I guess its some sort of DDOS attack to the login.

[ShinkuDragon]

I'd be real surprised if it was dupers causing this. I noticed that the first time the login servers went down, Scania started slowly going down as well. Ch2, then 3, 1, and 4. Afterward they all went back up because the login servers came back up.

If this is really the work of dupers, we're royally screwed lol.
A hack so powerful it can actually knock the login servers down?

i remember a server-wide crash before the first rows of dupes at the very beginning, i REALLY hope it wasn't them back then, but some crashes they caused, that comes to my mind the ones before a bunch of them got banned, were real server crashes, with ambition reset and everything

[happylight]

Server was really laggy for a couple hours before the crash. You can almost predict when it'll crash that way.

[iamflip]

It's been happening since last night lol.

http://flashfiez.files.wordpress.com/2011/07/dwayne-johnson-0.jpg

I smell compensation for this.

[SaptaZapta]

And there it goes again...

Most servers still showing green but haven't been updated in 5 minutes. I am guessing they're all down, it's just taking the script a long time to connection-timeout on them all.

[Chameleonic]

So much for improved reliability and stability after the 12 hour maintenance to improve those things...

[Foxino]

Site is down, Log in is down. Doesn't seem hack related to me, If it is that is some overpowered hax.

[modular]

So much for improved reliability and stability after the 12 hour maintenance to improve those things...

oh come on, do you really believe its about reliability and stability when channels go down 1 by 1 in single servers, and then start in other servers, and 1 channel is left up?

im surprised nexon hasnt acknowledged these hackers yet, usually anything that cuts into server uptime gets taken care of quickly

[Onion Knight]

Even that Ch20 login server crashed, whatever that does. Is that for testing purposes?

[danielcatu]

green

[Sorien]

Even that Ch20 login server crashed, whatever that does. Is that for testing purposes?

That's the Master Login Server. If that one goes down, everything is down.

[Celan]

Even that Ch20 login server crashed, whatever that does. Is that for testing purposes?

I belive it's the main sever.
AKA without it nothin' will work.

[Foxino]

and down it goes again!

[Liteness]

well, i hope those dupers are duping something good and not just crashing for the lols

[Chilly]

I'll hold off on the PQs until the evening then, I don't need my Kentas and Escapes being counted for without completion in a D/C (similar to my multiple uncompleted MPs yesterday).

[Kaffei4Lunch]

jdofijasdiofjaisdif
So frustrating lol

[SaptaZapta]

I still think it's a network issue, some router or gateway somewhere crashing.
The slowness of the stat updating when going down, as opposed the whole page going green at once when it returns, also points in that direction, as it takes a lot longer to get a "timeout" than a "no listener on this port".
(Not that I know anything about how Eos' program works)

[Justin]

If I recall correctly, Eos' script itself only checks on timed intervals. So I don't think that's a very reliable guess.

[Eos]

I still think it's a network issue, some router or gateway somewhere crashing.
The slowness of the stat updating when going down, as opposed the whole page going green at once when it returns, also points in that direction, as it takes a lot longer to get a "timeout" than a "no listener on this port".
(Not that I know anything about how Eos' program works)

Something like that.

You can now hover over the individual channels to see the latency in microseconds. It's about 20 seconds when they're not responding, .07 of a second when they are or when they're truly down.

[Dark Link]

So much for trying to set up shop and sell some stuff while I'm gone all day.

[Liteness]

more and more red dots! it will be interesting to see what nexon has to say or will it be another "server instability problem"

[Mazz]

Playing server crash lottery with some friends on Skype, this is quite entertaining.

--

fix it pliz i want 2 von leon

[happylight]

The servers themselves aren't going down, just the connections to them are. There's no rollback/duping as far as I can tell.

[Dark Link]

All login servers just went down again lol.

[Kaffei4Lunch]

Funny how there was that huge server update before Ascension... and now servers are going down left and right.
Or if the servers themselves aren't going down, then idk..... Either way it's bad lol

[Chameleonic]

This is fun...

[modular]

The servers themselves aren't going down, just the connections to them are. There's no rollback/duping as far as I can tell.

are you not watching the red dots?

[Eos]

are you not watching the red dots?

You're not following the distinction between the servers being unreachable and being down.

Your computer hasn't crashed just because it's offline.

[Chameleonic]

None of the profession traits have been reset so that means the servers never went down, right?

[Dark Link]

inb4 DDoS attack.
As someone mentioned earlier.

[happylight]

are you not watching the red dots?

Red dot only means server cannot be reached, not that it has crashed.

[Justin]

inb4 DDoS attack.
As someone mentioned earlier.

Wouldn't surprise me at all. They keep coming up and going back down within 5-10 minutes, this last one not even taking 2 minutes.

[ScottyxD]

Looks like there's a problem with Scania and the main log in servers.

[Green4Ever]

This is pathetic... How has Nexon not realized what's going on? LF> Unescheduled Server Maintenance already. :f7:

[Dark Link]

Looks like there's a problem with Scania and the main log in servers.

Scania follows suit right after the login servers bite the dust.


Wouldn't surprise me at all. They keep coming up and going back down within 5-10 minutes, this last one not even taking 2 minutes.

At first I didn't put too much attention into it and figured it was just connection issues, but after seeing this, it feels like it's being controlled by outside sources.

[L2X]

seems to be back on

[Eos]

We've had 12,000 hits to the status page so far today... So it's not just you :f2:

[Chameleonic]

seems to be back on

Wait another few minutes...

[happylight]

And... down again. Network issue for sure. Servers don't start back up again so quickly after crashing.

[Eos]

And... down again. Network issue for sure. Servers don't start back up again so quickly after crashing.

Mine have < 30 second reboots tyvm.

[ScottyxD]

seems to be back onWait another few minutes...
Indeed. Waited a few minutes and MapleStory seems to be back on.. for now.

[happylight]

Mine have < 30 second reboots tyvm.

I would say go work at Nexon but I don't think I'd wish that on anyone.

[Eos]

I would say go work at Nexon but I don't think I'd wish that on anyone.

they couldn't afford me and they're bright enough to hire me to consult.

[SaptaZapta]

None of the profession traits have been reset so that means the servers never went down, right?

Not quite. The way I understand it, it only means the database server didn't go down.
Some information is only kept in its memory, and not written to disk. Personality trait daily limits is an example. So is party/expedition info.
When the database server goes down, this information is lost.
Also, when the database server goes down, it takes quite a while to bring it back up again. (Read the Dev Blog on "what we do at maintenances")

Just recently Nexon had a half-hour maintenance, where the game servers did go down, but database was apparently left up, as parties were not cleared and trait limits were not reset.

However, in this case, it's pretty clear the game servers remain up, it's just Nexon's network going out.

[happylight]

they couldn't afford me and they're bright enough to hire me to consult.

I meant that at this point their code/infrastructure is probably a gigantic ball of overcooked spaghetti that I don't think I'd work there no matter how much they pay me.

[Chameleonic]

Whatever is going down with Scania it is stuffing up the rest of the servers.

[Eos]

Untangling that sort of thing is my daily life.

At this point other people's code is like a magic-eye book to me.

[Justin]

Bleh screw trying to wait this crap out, just gonna go play PS3 or something....

[Eos]

hahahaha

The mabi-wiki has this to say right now;

All Nexon Games and Nexon Website are experiencing Latency issues at this time

[Dark Link]

Bleh screw trying to wait this crap out, just gonna go play PS3 or something....

No kidding. Apparently I picked a good day to go hang out with my Job Core friend. I won't be surprised if this is still going on even when I return later tonight.

[L2X]

Hope this server 'Latency' issue gets resolved by the time i get home.

[Green4Ever]

Other channels in other servers are starting to go down now.
I'm just trying to get my 30 Ergoths for the day and then cap out Ambition so when they finally do have an Unscheduled Server Maintenance I can get double Ambition for the day.

[SaptaZapta]

Whatever is going down with Scania it is stuffing up the rest of the servers.

Nothing is specifically wrong with Scania.
It's just the first server Eos' program checks, and with every channel taking 20ms to turn red, it takes a long time to get to the next server.
Usually the network problem clears up (temporarily) before Scania is done turning red, and then the whole page goes green instantly, but that is an optical illusion. If any other server were first to be checked, it would appear to be the problematic one.

[heronz]

the DL from another website is that someone/kid is crashing the servers make 10b meso per crash.

[Eos]

the DL from another website is that someone/kid is crashing the servers make 10b meso per crash.

Let's not bother spreading silly rumors.
http://www.southperry.net/showthread.php?t=49759

[Foxino]

they couldn't afford me and they're bright enough to hire me to consult.

Profit from cubes > Hire Eos > Working Game/Servers > Profit?

I could be their accountant too with these ideas :f3:

[Heffalump]

Let's not bother spreading silly rumors.
http://www.southperry.net/showthread.php?t=49759

Login issues (which one would think would be in the login servers, no?) =/= channel crashes last time I checked.
Because it's the individual channels that are crashing right now.
It was just khani 8 and now it's yellonde 4 and chaos 13.

[IMNOTSAM]

Well as fun as lagging and losing 50% exp (i tried to LHC and kept dying from lag~i am one determined sob) I think i'm going to stop playing for today
Get you shit straight nexon