Crowd Testing Services I: Technical Performance

Play It Safe by Testing

I love rock climbing.

I love climbing, even though it frightens me. There's something thrilling about trusting your life to a bit of rope and a few metal pieces. I do it because I trust that the tools that I use have been built to withstand heavy weights, prolonged use, and long falls.

Take, for example, a carabiner. These small metal links need to meet some very specific criteria in terms of applied force they can withstand. Those criteria are much higher than what a normal user will ever need. Further, they are tested against all sorts of misuse. What if a carabiner is clipped backwards? What if it’s cross-loaded? What if the gate gets stuck on the rope, what if it's used for ice climbing, what if it’s thrown on rocks 200 times before it's used… An untested product could break and cause serious harm to the user, and consequently a lot of problems for the manufacturer.

I love video games.

See where I'm going with this?

Sure, at least in terms of physical harm, there's less exposure for a games studio if the game releases and the infrastructure breaks immediately. But the consequences can be financially devastating. Months, if not years in development, wiped away because the volume of players during the launch window prevents those very players from discovering your game. A couple days pass, those players move on to something else, and are unlikely to give your title a second chance. This is a costly mistake.

So, once you have tested your game in all sorts of controlled settings (playtesting, usability, functional QA, compliance QA…), you need to make sure that you give a few sessions to your game "in the wild". I'd say "one session", but I've yet to see a title go through a round of load testing and survive unscathed. Like clockwork, the cycle of using real-world scenarios is anticipation, surprise, and horror, and a few days or weeks later… another load test session with an improved product.

What Can You Learn from Real-World Performance Testing?

The list is surprisingly long. Let me name a few:

• Technical preparation assessment
• Stress limits
• Hardware compatibility
• Speed and difference in lag across geographies
• UX/UI feedback
• Localization QA from in-country / native players
• User feedback reports

On one hand, real-world performance testing will tell you how technically ready your infrastructure is. Until now, you have served your game to small number of developers and testers. Today, you verify how your servers behave with 300 (or 3000) players logging on simultaneously. How will your authentication servers react to those simultaneous requests? Will the content be served properly? Will the multiplayer code give every user a healthy lobby? How badly will the title lag?

Benefits don't only come from concurrency. How quickly will someone in your home country, versus a player across the world, get in-game is highly important information. Will they both be able to play in the same session? Will they have an equal experience?

And do not underestimate the power of stress. Even without talking about servers, functional issues will appear, left and right. You will just have to scoop up all those bugs that did not appear during your in-lab QA months.

What about compatibility? Give your title to 200 Android users and the breadth of devices you'll cover will be a real cross-section of your future target market. 500 PC users will allow you to test just about every CPU, every chipset, every RAM model…

And lastly, it can be an incredible source of pre-release user feedback; more on this in my next article.

A Diverse Crowd with Discrete Demographics

With more than 60,000 gamers located in nearly every country in the world, our ever-growing Lionbridge Games spans all demographics. Because we track dozens of data points, we can offer engagement from broad swaths of users or from a very specific demographic slice. Geolocation down to city level, hardware brand and model per component, OS versions… you name it, we have it.

Whether you need to assess your game based on platform use, internet provider or genre preference, we can rapidly deploy a hand-picked group of gamers to match your needs and answer the questions that matter to you, any day of the week.

Test Everything, Everywhere

Overwhelmed? When you contact Lionbridge Games, we'll help you decide what matters most at the development stage you're in. Here are some of the services we offer:

• Load testing (to assess infrastructure sufficiency even during user peaks)
• Stress testing
• Multiplayer testing (specifically, lobby and multiplayer testing)
• Regional connectivity (across locations and ISPs as well as 3G/4G/5G vs WiFi)
• Compatibility testing for any hardware

Need something that is not on the list? Just contact us and we will build a custom solution just for you.

You’d never climb with an untested carabiner, right? Our crowd may not throw your game against a rockface, but they still know how to push it to its limits so it’s ready for the toughest critics of all: your players.