[GMS Dev] Bugs
Hello again Maplers! It’s Eurydice with another MapleStory dev blog!
First, I’d like to thank everyone who posted feedback in the forums and on Facebook about the first dev blog. It’s been a big help and I wanted you to know that I’m reading your comments to find what you’d like to hear about. The response so far has been pretty clear; you want to hear about bugs and hacking, so let’s get to it!
Search and Destroy
Let’s get this straight — no one likes bugs. For developers though, they’re more than an annoyance. They make us go crazy. With hundreds of thousands of lines of code and new content coming from multiple locations though, bugs do happen. In fact, they actually happen a lot more than players ever realize. The first version of a new update that we see is often completely unplayable. We’ll go through five to eight full versions of an update before we get to the one that is released to players. In that time anywhere from 250 to 400 bugs are discovered and fixed.
Fixing bugs before a game update goes live is the easy part. Our QA team doesn’t mind when we patch the game three times a day, or when we have to bring the servers down unexpectedly so we can make a change. When the update goes live we can’t be so flexible. We’ll even do a round of testing after the game update is complete but before allowing players to login just to be sure that all essential functions are working properly. This process adds time to the game update, but allows us to catch serious issues before they affect players.
Just recently, we added another layer of protection by introducing the Tespia public test server so that players can participate in the bug hunting process too. Even with all our testing though, issues do slip by. Some are serious and some are trivial but they all have to be dealt with eventually. So what does happen to bugs that make it into the live game?
Bugs in the Wild
After a game update goes live, our QA team continues to test it on the live servers, but there are only a handful of them and hundreds of thousands of you, so bugs are usually found by players first. We are constantly reading the forums (and not just the official ones!) and customer support tickets to discover what bugs players are experiencing. The thing is, seeing your bug posted in the forums or your ticket is only the first step. Once we’re aware of it, we need to find it and fix it. This can be a difficult and painstaking process.
Step one of a bug hunt is forwarding all the information we have on the bug to our QA team. These are dedicated testers who then try to reproduce the bug and come up with detailed steps to explain exactly how it happens. Once the bug is reproducible and fully documented by the QA team, we can begin finding the root cause and start testing fixes. This is actually an area where you can help. If you’re posting information about bugs in the forums or in a ticket, be sure to include as much information as you possibly can about how the bug happened. The more information we have, the faster the bug fixing process goes. It always saddens me when players post that they believe that Nexon doesn’t read the forums. In fact we read them all the time and find them incredibly valuable. Please keep the bug reports coming!
Three Different Solutions
Unfortunately just reading the forums and your tickets and knowing about the bugs isn’t the same thing as fixing them. First, fixing each bug is different and finding the root cause for some problems can take a long time and a lot of trial and error. Even when we have a solution though, that doesn’t mean that the fix will be deployed right away. The big determining factor is how that fix will be deployed.
Game fixes come in three varieties: changes to the server files, changes to the database, and changes to the client. Server files can be fixed with a short maintenance, so if a bug is really serious we might have an unscheduled server maintenance to apply a server file fix. Database fixes require a longer server maintenance, so it’s not as common for us to fix database issues with an unscheduled server maintenance. Client bugs require an entire patch to fix, so they generally have to wait for the next game update. Even though they take the longest to fix, client bugs are not necessarily the most serious. Typos, graphic errors, and sound problems all require a patch to fix.
Of course deploying any of these fixes means taking that game down, something nobody wants to do. That’s why we only have unscheduled game updates for something extremely serious, not just an embarrassing error (hello, Khadia!). Our goal is to keep down time to a minimum. Bugs get fixed every day of the week, but usually we deploy the fix during the next scheduled server maintenance. Our servers also require a certain amount of maintenance on a weekly basis, and we try to consolidate all our bug fixes, name change requests, and world transfers into this time frame so we only have to bring the game down once.
I hope this gives you some insight into the way we handle bugs. Before I leave though, I want to address the topic players ask about more than any other — hacking. Here’s the bottom line on hacking; I am never going to do a blog post on hacking. The details of how we detect, investigate, and block hacks are too sensitive and cannot be shared without posing a security risk. All I can really say is addressing hacking is always a top priority for us and we work on it every day.
Thanks for listening and thanks for playing MapleStory! Be sure to leave comments on the forums with your request for the next topic as well as general feedback. And please try not to give Hime too hard a time on the forums!
Until next time, Happy Mapling!
The MapleStory Team