Obtaining Feedback from Gamers - How we do it

So I got some responses to my last post about Understanding the Angry Gamer. Aside from the expected results, one recurring note that stood out was that sometimes developers are in the wrong, that we have made mistakes, and that we should listen. I totally agree in that we developers are sometimes wrong, that we make mistakes, and that we should listen. However, that brings up a rather important question: “Who should we be listening to?”

The answer, of course, is everyone. So today, in a special Labor Day weekend edition of AAGD, we can examine how developers get and parse feedback.

One thing that developers always want is feedback. We want to learn what went right, what went wrong, and how we can make things better. However, it’s important to keep the big picture in mind - you always want to try to make changes that would maximize overall happiness and engagement among all players, and not just the ones who are the loudest. That means that we can’t just do what the very loud minority wants - we need to be sure that it’s true to the vision of the game we want to make, and for the entire playerbase.

So how do we get that information? We clearly can’t go from house to house, asking people. We clearly can’t just ask our friends and families; that’s way too small a sample size and also too self-selecting. We can’t ask random people off the street either, since there’s no guarantee that they’ve even played the game. And we can’t really just ask the forum goers for the very same reason we can’t ask our friends and families - it’s way too small a sample size, and they way too self-selecting. So how do we gather this information?

Our best method on gathering feedback today is actually telemetry data. Have you ever seen those stat sheets that get posted every now and then by game developers? Fact sheets like this:

We collect this data from your gameplay. Not just you either, but the millions of players who played the game. Did you ever notice how so many games have a “Connecting to server” and “Checking for updates” message when you start the game up, even when they are single player? That’s your game letting our servers know that you started it up. How there’s a little checkbox in the options that comes on by default that says “upload non-personal information” or something like it? That’s your consent to tell us anonymous gameplay data about how you play. When you’re playing and still connected, we’re getting a steady drip of information from you. And we can get a lot of it - terabytes and terabytes of data.

Examples of things we can learn:

• How many characters you’ve created
• Any specific character creation details
• How long you spend, on average, in the character creator
• How many times you’ve started the game
• How many times you’ve saved the game
• How many times you’ve reloaded the game
• How much time you spend in any particular area
• Where in those areas you spend that time
• Which NPCs you interact with in those areas
• How much time you spend standing still
• How many times you’ve finished the game.
• Which abilities you use most/least
• Which weapon you use most/least
• Your in-game decisions, like whether to kill or spare this NPC, or who you prefer to have in your party
• Which menus you spend your time on the most
• Where you die each time.
• Which quests you completed. How long did each take to complete? Which ones you abandoned. Which you ignored.

Basically, our telemetry data allows us to answer the question “What did the player do?”. If we made it possible to do in the game, we can probably track it. And the data we get are pretty close to incontrovertible - in order to get a 99% confidence level to within a 1% margin of error on a population of 5 million players, you’d need a sample size of ~16,600 players randomly selected from those 5 million. We get literally millions of unique data points to pull from, so our data crunching gives us 99% accuracy within less than 0.1% error margin. And we can qualify it even further if we want - if we wanted to know how many players cheated on their romance between Mass Effect 1 and 2 and were also paragons above 50% but below 100%, we could find that number.

All these data allow us to model the information in interesting ways. For example - this is a map called Anchor 9 from Halo Reach that is played in multiplayer:

It’s in greyscale because this is actually a heatmap. The brighter a particular spot is, the greater the number of player deaths that have occurred in that spot. This is actually very useful for level design tuning - you can see where the players are dying and where they aren’t, and that means you can try to adjust your map to tweak that sort of gameplay in those areas. By moving power ups or adjusting the geometry slightly, level designers can rebalance the map. Perhaps they want more deaths in the darker areas because they are underutilized. Maybe it is intentional. This is one way of checking our results and iterating on it for the future.

Another use for this telemetry data is in how and where to dispense hints. In Telltale’s point-and-click adventure games, sometimes players will get stuck on a particular puzzle and won’t know what to do. Telltale’s designers anticipated this, however, and put in a reactive elaborate hint system - depending on your difficulty setting, your chances of receiving a hint from a companion character goes up for each second you stand in one place or click on screen without clicking on an interactable object - two examples of frustration. They track that data and use it for developing future puzzles.

So does that mean we should completely ignore the cries of the very invested gamers? Of course not. The crucial problem with telemetry is that it doesn’t answer the “Why”. We as developers have to start doing the interpretation, and this is where the mistakes usually come in. To help avoid this, we try to listen to all of our avenues of feedback - telemetry data, hard core fans, observations during playtests, market research surveys, and so on, and so forth. What we are looking for is corroborating data - if what the hardcore fans are saying doesn’t match the rest of the data, it probably isn’t as big a deal as the angry gamers are making it out to be. If we see anomalies in the data, we can check out what the invested fans are saying - why are they upset? Then we can cross-check that against the surveys, market research, and financial data to make better decisions. 

I think it’s important for the really engaged fans to remember that we are listening. We do care what you have to say. But though we do listen to you, it doesn’t mean that we are going to do what you want. We have millions of people who want different things, and it’s important to remember that we value them too. It’s ok to disagree with the developers. But I just wanted to show that we have access to a lot more information that we have to consider when we make our decisions, far more than what you see in the forums, comment sections, and youtube videos.