Hello Maplers,
First of all, Happy New Year. On behalf of everyone at Nexon, I wish all of you a fulfilling 2014. Also, I hope everyone in the Northeastern US is doing ok and staying out of the cold.
I’m writing to address the service interruptions this week. The purpose of this message is to provide insight into what happened, what we did, and the rationale for our response. It is difficult for us to communicate details with you during these incidents, as we are heads down in addressing the issues. I often compare a part of our work to that of working in an emergency room. Everything is happening live and everything has to go right (which is not always the case).
This week we had several challenging incidents pop up at once which was quite debilitating for our teams. The first incident was a duping hack made public on the night of New Year’s Eve. The second compounding incident was a DDoS attack that impacted many Nexon game services on January 2.
I was personally notified of a duping exploit going public on New Year’s Eve. I’m grateful to have been tipped off via Instagram and Twitter (Thanks to the tippers! You know who you are). Staff were called back in on New Year’s Eve/Day to begin working on identifying the issue and worked with our developers on the fix.
The hack was identified as a packet editing hack that utilized hired merchants to duplicate items. As the hack had gone public, we quickly removed hired merchant sales from the Cash Shop and put a notice up in-game alerting players of the live issue. Players reached out for more information. Unfortunately, it is not possible to give a play by play while resolving issues like this. We have to focus and we do not want to tip abusers of our response.
Several questions have since come up… Why didn’t we shut the game down? Why didn’t we do an immediate rollback? Is the economy damaged beyond repair?
Regarding the reaction time, it takes a team to tackle issues and gathering our team takes time (especially on New Year’s Eve and Day). As we all know, New Year’s Eve and Day are times of gathering for friends and families all over the world. We still call people in, but it is difficult on everyone. Our employees do deserve their time to celebrate the New Year with their loved ones. I’m very proud and grateful that they do sacrifice some of life’s special moments to take care of the service. They did come in and worked well into the night during their holiday to fix the hack. Personally, I just had a baby this week (yay!) and I’ve been working overtime as well. My daughter still doesn’t have a name… sorry “baby”! I know our silence can create some misunderstanding, but we are genuine about our care of MapleStory and our community. We hope to make that clear and gain your trust in 2014.
Regarding the rollback, we ran our analyses and determined that a rollback was not the best way to resolve the impact. Rollbacks are a clean way to combat issues with the economy, but they do impact players negatively by deleting their progress. We use it as a last resort. Unfortunately, with this duping incident, we found that the original dupers had privately impacted the service under the radar with low levels of duplications well before December 31. Most likely, they went public with the hack to mask their own efforts and/or it leaked.
Rollbacks during duping incidents are essentially a means to make the economy right. In this case, we made the hard call to catch dupers and gather/delete duplicated items manually. We did not want to have players suffer more loss of progress and we felt we could eliminate the majority of the impact. It takes longer and requires us to run a temporary trade block on suspected accounts, so that we can stop the spread, conduct our investigations and clean up. We do this knowing that our nets will catch some good dolphins in the pursuit of some bad tuna. This does not mean we are willing to sacrifice the good with the bad. It has always been our intention to lift the restrictions on our legit players. The priority was to stop the spread. We are doing our best to work swiftly to remove restrictions on our legit players. If you were blocked by mistake, please rest assured that our CS team are currently investigating and restrictions will be lifted shortly. We greatly appreciate your patience and sacrifice for the community.
We have already eliminated a lot of economic impact with our investigation efforts. As this hack involved the purchase of hired merchant, we could more easily identify the abusing accounts. Also, the cash requirement limited the number of public hackers. In total, we have banned hundreds of direct abusers and removed hundreds of thousands of items. Without exposing the actual data, I can say that the availability of all items of concern are now below levels recorded on December 30 (before the hack went public). Catching dupers also helps us to catch banks and related accounts which also reduces item levels in the economy. We hope this gives some confidence in our efforts. The economy is not destroyed. We are diligently repairing it.
With all of the above going on and staff working around the clock, we were hit by an unrelated DDoS incident that put some serious strain on our staff who now had to deal with two very challenging live incidents. I'm grateful to the team and their families for all their efforts and sacrifice this week. It really was a Herculean effort.
On a side note, we ask that players do not fall into the lure of public dupes and too good to be true Free Market sales during duping incidents. We can’t expose how we do it, but we log a LOT of data and eventually we will catch hackers, exploiters, and those taking advantage of the situation. Please do yourself and the community a favor and do not participate in hacks and exploits. When hacks go public, your participation is simply adding to the chaos and you are being used to hide. The community impact is service instability and reduction in resources that could be used for creating a better service. The personal impact is that hackers and exploiters will lose their account, their connection to their friends, and all their invested time and money.
We are open to the possibility that we have missed some exploiters (for now). We will do our best to continue our pursuit to find them and eventually we will catch the majority of them. I've seen some comments about abusers running wild. A 24 hour rollback would not have kept all of these items off the market. If you are certain of abusing players that we have missed, please report them and evidence to our CS department via the ticketing system and we will do our best to take the appropriate action.
Well, this is a great way to start of 2014 =) On the bright side, we are making communication and transparency a priority this year. On behalf of everyone at Nexon America and our Nexon Korea development team, we thank you for your patience and support. We apologize for any inconvenience this may have caused you. In the coming week, we will continue analyzing our data to enact more bans and lift the trade ban on legit accounts. In the coming year, we will boost our security to reduce the number of these incidents. MapleStory is a very special game with a very special community. We will do our utmost to bring you the MapleStory you deserve.
Have a great weekend everyone.
Cheers!
Stove
Message to the MapleStory community from Nexon CEO
Message to the MapleStory community from Nexon CEO
Originally Posted by Nexon
Reply With Quote
about this post
.
Bookmarks