Xbox Undercover:

Who’s watching the Xbox Live service?

You know that the Xbox Live service is available 24/7 in 19 countries. But did you ever think about who’s on watch while you’re playing (and even when you’re not)? Here at Xbox headquarters we have a place called the Xbox Operations Center. Filling this room are banks of work stations, Xbox consoles hooked up to Xbox Live, giant plasma screens showing service data, and a humongous digital clock displaying the time in Korea/Japan, London, U.S. East Coast, and U.S. West Coast. The main function of the XOC (pronounced “zock”), according to XOC manager Markus Schweig, is to monitor the health of the service, identify any issue before it impacts gamers, and drive resolutions. Eleven operations specialists run the show, and two are on shift in the XOC at all times. These guys are ready for anything.

What happens if the power goes out?
A couple months ago a fierce windstorm knocked out the electricity at Xbox headquarters. Not to worry—all key systems in the XOC are on backup power.

What happens if the telephone switch in the building goes out?
On Markus’ desk sits a big red phone. It’s not a hotline to the Kremlin; it’s a direct line to the phone company, from which he can get an outside line even if all the other phones are down.

What if the number of gamers on the service suddenly drops?
Could be any number of reasons. The blackout on the East Coast last summer is a good example. The Super Bowl and major TV events are also responsible for a bunch of you shutting down the Xbox and tuning into TV, but in those cases the drop tends to be more gradual. If your local ISP is having issues, gamers may not be able to get online, causing a smaller drop in usage.

What if something goes wrong with the Xbox Liveservice?
The XOC uses Microsoft technology to keep an eye on XboxLive. In the event of a problem with the service, the servers, or any of the components, XOC staffers receive an alert and a description of the trouble. If multiple alerts occur, the XOC guys work to discover the root cause and take immediate steps to remedy the problem. If the guys on shift can’t figure it out, there is always someone higher up in the chain of command with a pager. If it’s an issue that affects Xbox Live subscribers, the team posts a message to Xbox.com.

What if something goes nutty in a data center in Japan or London?
The XOC is equipped with a system that allows operations specialists to connect to the far-flung data centers as if they were in the same room. If a hardware issue pops up, local contacts are brought in.

Late Night in the XOC
TriXie recently spent a Friday night in the Xbox Operations Center. After hanging out with Major Nelson for Casual Gamers Night, she headed down to the XOC around 10:00 p.m. Already hard at work were Operations Lead Michael Repperger and Operations Analyst Shawn Becker. Mike has been working the XOC for about a year and a half. Before coming to Xbox he worked for NATO. “If you're wondering how hardcore Xbox Live security is, it's better than NATO's,” he says. Shawn’s been on the job for just three weeks.

Following is TriXie’s report on one night in the life of the Xbox Operations Center.

10:00 p.m. PST

Shawn: Noting TriXie’s arrival in the XOC log and watching for “events” on all the servers.
Mike: Monitoring how hard Xbox Livesubscribers are pounding on the statistics servers.
TriXie: Asking questions and playing Bejeweled on msn.zone.com.

10:30 p.m.

Shawn: Checking for events on the non-productions networks (the networks that developers use to test games).
Mike: Cycling through a tool that monitors hardware performance and writing a report on attempts to hit the Xbox Live service in countries that haven’t yet “goneLive,” mostly Taiwan, Singapore, and Hong Kong.
TriXie: Eating Fritos from the vending machine.

11:00 p.m.

Shawn: In the back room checking on a non-production network.
Mike: Investigating an issue in a non-production network.
Major Nelson: Stopping in to say hi.
TriXie: Posting on the Xbox forums.

11:30 p.m.

Shawn: Still rockin’ the non-production networks. Some game developers are testing a game that begins with “H” and ends with “2”.
Mike: Reading the Xbox Undercover article about the Xbox Secret Weapon on Xbox.com, facilitating an IM conversation between two developers, and making sure that our automated security processes are working as expected.
TriXie: Taking online quizzes. Discovers she is the lip gloss flavor “Way-Out Watermelon” and that her inner Greek goddess is Aphrodite.

Midnight

Here comes the graveyard shift: enter Tatsuaki Ryu and Quinn Bracken. Turns out these two are both fluent in Japanese. It’s 5:00 p.m. Saturday in Japan—prime time for the data center there. Mike speaks German, which is why he’s a swing shift guy. The 5:00 p.m. to 1:00 p.m. shift happens to coincide with prime XboxLive time in Europe. If there’s ever a problem in the data centers in Europe or Japan, the XOC’s got someone able to communicate with the locals.

Shawn: Noting Tatsuaki and Quinn’s arrival in the XOC. Getting ready to play Unreal 2.
Mike: Investigating new hardware compatibility for upcoming Xbox Live features.
TriXie: Forcing XOC staff to take online quizzes. It turns out that the swing shift dudes are both “Vanilla Swirl” lip gloss while the graveyard guys are more “Perfectly Peppermint.”

12:30 a.m.

Shawn: Checking a non-production network, trying to beat Tatsuaki’s time on the Rubik’s Cube, and monitoring the production network and statistics.
Mike: Writing a hardware compatibility matrix and telling a story about drawing a picture of the Space Shuttle’s fuel schematics on the chalkboard in his sixth grade classroom the day the Challenger blew up.
Tatsuaki: Fiddling with a Rubik’s Cube and checking his e-mail. (Legend has it he can solve the Cube in less than 20 minutes.)
Quinn: Checking his e-mail.
TriXie: Discovering that her Superhero Style is Catwoman. Strangely enough, so is Shawn’s. Mike is more of a Wonder Woman and both Quinn and Tatsuaki favor Xena Warrior Princess.

While we were kind of hoping for some sort of major incident just so we could use the big red phone or summon the SWAT team, it was an uneventful evening—as most are in the XOC. What can we say, XboxLive is a rock-solid service.

Comments about this article?