Measuring player activity in text games

Last time, we learned how to measure player retention and explored various ways to keep players coming back.

Today, we’re going to take a closer look at player activity metrics, including how to ask the right questions and different approaches you can use to collect information that can help you improve your multi-user dungeon (MUD) or other text-based game.

I’ll also touch on important issues such as:

• common pitfalls in data collection
• best practices
• transparency
• data privacy and security

Before we jump in, though, let’s quickly define what is meant by player activity. That way, we can be sure we’re all on the same page.

Other forms of player activity include:

• killing monsters
• crafting weapons
• participating in PvP arena matches
• sending messages through the chat channel
• turning in quests, etc.

With effort, these events can be measured to better understand how players are actively spending their time in your game.

If you’re running a roleplaying game, for example, you might want to know how much time your players actively spend in RP versus on single-player tasks.

But you can also look at passive player activity, such as how long players spend idle, waiting on in-game timers, or flagged as away from the keyboard (AFK).

Choosing which questions to ask

Your first step, before collecting any data, is to start thinking about the problems or questions you’re interested in addressing.

The questions you ask will help you determine a) what kind of data you’ll need to collect, and b) how you’ll go about collecting it.

For example, let’s say you’re concerned that players don’t have enough to do after a few months of play, and you’re worried they might not be very engaged.

Some questions you could ask to get to the bottom of this include:

1. What does the player retention curve look like over a 3-month period? How does it compare to similar games?
2. How much time do players spend idle/AFK after 3 months of play compared to during their first week of play?
3. What activities are players engaging in after 3 months of play compared to during their first week of play?
4. Are players getting bored of the game after a few months of play?

While these all have to do with the initial problem/concern, notice how questions 1, 2, and 3 focus on what players are doing, but question 4, which really gets to the heart of the matter, is about what players think.

The first three questions can be answered by collecting and analyzing player activity data, but to answer the fourth question directly, you’d need to survey your playerbase and get their feedback.

Asking players vs. looking at the activity data

At this point, you might be wondering, “If I can just ask my players whether they’re getting bored, why would I go out of my way to collect activity data?”

This is a great question, and to be honest, you might decide that player feedback is all you want or need to run the kind of game you envision.

One answer is that unless you have a way to contact players outside the game, you can’t survey the ones who already left because they ran out of things to do.

This is your weak spot.

Another, perhaps less important answer is that not everyone will respond to your survey even if they receive it, and the data you do get will probably be biased in some way.

Ultimately, there will be situations in which you’ll want to get feedback from players (which is itself a form of data collection), and there will be situations in which you’ll want to look at what players are actually doing and how they’re spending their time.

Each of these sources of information will provide you with different pieces of the puzzle.

Choosing which metrics to focus on

Let’s say you’ve decided to pursue a problem or issue related to player activity, and you’ve settled on some overarching questions you’d like to ask. What’s next?

Your next step is to hone in on the metrics that will help you answer those questions, so you can plan out your data collection strategy.

Often, there is more than one way to approach a problem, and some approaches may require more effort than others.

In the example above, I provided 4 different approaches for getting at whether players are running out of things to do after a few months of play.

Each of those approaches required some form of information, be it player activity data or player feedback.

For each of your questions, you should ask yourself:

1. What’s a meaningful measurement, and
2. can I reasonably track it?

Meaningful in this case simply refers to whether the metric can help you better understand the thing you care about.

In addition to viewing your own time as a valuable resource, you may want to keep in mind things like server load, disk space, privacy concerns, and so on while answering the question of what’s “reasonable.”

Research scenario: Are players running out of things to do?

To put this in context, let’s return to our example from the previous section. Let’s say that the overarching question we want to ask is:

• Are players leaving the game after a few months, and if so, is it because they’re running out of things to do?

Let’s say you already have a way to track retention, so you know when players are leaving. This answers the first half of your question. But let’s say you don’t yet have a way to report on which activities players are engaging in, how often, or for how long.

What should you do? What’s a meaningful measure in this case? One that you can reasonably track?

For practical reasons, you may decide to use the 80/20 rule here. Where can you put in 20% of your effort to get 80% of the results?

Do you really need to put in the extra dev cycles to track individual activities? Your time is a valuable resource, after all.

In fact, you may decide that using your existing retention data and supplementing it with player feedback is good enough to answer the question.

If so, your next steps might be to 1) create a survey to collect player feedback, and 2) distribute it among your playerbase. To address your weak spot, you might even post the link to your Discord in order to get feedback from players who stopped logging in but are still lurking in the community.

Once you know what’s causing your players to leave, you can then devise a strategy to keep them around longer.

Collecting player activity data

In my guide to player retention, I listed account creation date, last login date, amount of time spent logged in, and amount of time spent actively playing as potential metrics to look at for each player.

It’s worth noting that these are all individual-level metrics that can be used to calculate descriptive statistics like averages, medians, and ranges.

For example, if you collect data on the amount of time spent questing, you can calculate each player’s average amount of time spent questing, as well as the overall average for the entire playerbase.

You could then compare the amount of time spent questing to the amount of time engaged in some other player activity, such as PvP or crafting, to see where your players are spending most of their time.

In short: descriptive statistics are a powerful yet simple way to analyze data.

However, you can run into some problems if you don’t save the right types of information during data collection.

Let’s look at a couple of examples.

Issue #1: Keeping count

If you’ve never done quantitative research before, counts can be tricky depending on what you’re counting.

Let’s say you want to track overall levels of player activity, and you define an “active player” as someone who spends at least 1 hour a day logged into the game.

At the end of each day, you count the number of players who were logged in for at least 1 hour and save that in a table. On Tuesday, there were 32 active players, on Wednesday there were 34 active players, and so on.

At the end of the week, you now know how many players were active each day according to your original definition, but what if you change your mind after a few months and decide that 1 hour isn’t enough to differentiate who’s active and who’s not? Maybe players need to be logged in for 3 hours to really be considered “active”?

Well, if you didn’t save the amount of time spent logged in and you don’t have a way to calculate that metric from login/logout times, you’re out of luck.

You simply don’t know if some of those players who were logged in for at least 1 hour were also logged in for 3 or more. Maybe some of them were and some of them weren’t.

If you do know the number of hours, however, you have options. You can group those data however you like.

For example, you can decide that under 2 hours is low activity, 3-5 hours is moderate, and over 5 hours is high activity. You can report on all of these groups or change your definitions at any time because you kept the original data.

The lesson here is to avoid storing counts based on definitions that can change and instead store the data used to get those counts, where possible.

Issue #2: Survey answer choices

When surveying players, you may find that you want to gather demographic information such as their gender and age. After all, demographic data can help provide valuable context and reveal important patterns among your playerbase.

Players can be hesitant to give out this information, however, and may even quit your survey early if they find the questions too personal or the survey too long.

For these reasons, you’ll want to be very intentional with your questions.

Meaning, don’t ask for personal information because you think it might be interesting or useful later. Be respectful of players and their time.

Keep in mind that you’ll have to balance player needs with your own needs when it comes to the type and quality of the data you collect.

Collecting high-quality survey data

For example, say you want to know about players’ ages so you can calculate the average for your entire playerbase. Or perhaps you want to find out whether younger players in your game are more active than older players in some way.

Let’s say you want to ask for the respondent’s age as part of your survey. Not everyone wants to give out their exact age, so you decide to let players select from a set of age ranges (very common among surveys).

You supply the answer choices as follows:

• 17 or younger
• 18-30
• 31-40
• 41-50
• 51+

Great! You’ve now created a simple question with only 5 answer choices. That wasn’t too hard, right?

But now you’ve got a different problem: your data are defined by these buckets.

What if you decide to look more closely at the experiences of minors? Too bad. You lumped everyone between ages 18-30 together, so now you’re limited in that way.

The lesson here is that you’re going to want to be thoughtful about how you set up the questions, including how many, how personal, and how long.

What both of these lessons have in common is that they demonstrate the importance of thinking ahead and collecting the right data.

You can always aggregate and group your individual-level data later as you see fit, but you can never work backward from high-level data without losing something in the process.

Analyzing patterns in player activity

Once you’ve got a good system in place for measuring player activity, you can look for interesting patterns in the data, including differences between groups of players or their characters.

For example, you could compare activity and characteristics between:

• spellcasters vs. fighters
• different races or professions
• PvP opt-in vs. no PvP
• crafters vs. combat characters

Pretty much anything you can think of, as long as you have a way to measure it!

Let’s look at one more possible research scenario, then we’ll move on to other issues having to do with data collection and player activity.

Research scenario: Are all the races equally fun to play?

Let’s say you’re concerned that not all of your game’s races are equally fun to play.

You know that your game currently has very few dwarves and a lot of elves, and you suspect there may be several reasons for that.

You look at the data and find that new players are creating dwarven characters, but overall, the retention rate among dwarves is much lower compared to the other races.

This tells you that it’s not an issue of players just not wanting to play dwarves. Rather, there’s something about playing a dwarf that loses players after a time.

You might then ask your playerbase to confirm whether this is true based on their own experiences and, if so, ask for their feedback and suggestions on how to make dwarves more enjoyable to play.

Does sample size matter?

By now you may be wondering whether the size of your sample (or population) matters. After all, it’s a common concern in research, even in the mobile gaming industry.

So does sample size matter in this case? No, not really.

The larger your sample size, the more confident you can be about any patterns you find, but you’re not trying to write a research paper for a peer-reviewed journal.

Unless you’re planning to formally test a hypothesis or calculate statistical significance, there’s no need to get hung up on sample size.

Essentially, what you’re doing is researching your own playerbase in order to better understand your players and their needs. And for that, you don’t need statistical significance, just a thoughtful approach and an open mind.

Should you tell players you’re collecting activity data?

If your game is available to the general public, then yes, you should.

Even if players don’t sign up for your game using their real names or email addresses, it’s better to be transparent about the information you’re collecting and why you’re collecting it.

When it comes to MUDs, most players likely accept that certain information is being collected about their activity (for example, IP address and latest login date/time), so you’re not likely to shock too many people by storing a few more data points. It’s also true that most games don’t offer secure/encrypted connections.

However, that doesn’t mean that privacy and data security aren’t important to people in the MUD community.

If you’re going to collect and store data on individual players, I suggest having a privacy policy and terms of service in place. Use these documents to educate players and lay out your ground rules.

One of the benefits of having a privacy policy is that it can help you build trust.

Due to issues discussed in my post on ERP and consent, I also strongly recommend these documents if your game features mature adult themes.

You can find free templates for your privacy policy and terms of service online. A template can provide you with a starting point, which you can then modify to suit your game.

Make the information you collect benefit players

Lastly, I recommend making some of the information you collect available to players in a way that benefits them.

For example, give players a command they can enter that will print out their own activity report. You could even print a comparison of their activity to the average activity levels across the entire playerbase during those same time periods.

Here’s an example activity output from an RP game:

In the last seven days:
=============================================================================
      Lupin is RP active with 5 hours and was online 31 hours.
      There were 38 unique characters online of 280 total in the database.
      Of those online, 24 actively role-played with an average of 11 RP hours.
      A total of 275 hours of RP were logged.
      The Top Ten RP hours ranged from 47 hours to 8 hours.
      Minimal (3), Low (10), Average (4), High (1) and Excellent (4)
=============================================================================

By being honest, transparent, and sharing information with players rather than withholding it, you can foster a healthier community around your game.

Depending on what information you collect and how you store it, you may also want to consider anonymizing, hashing, or de-identifying the records – not just to be safe but to show players that you really do take their privacy seriously.

Additional ideas and considerations

Finally, here are some additional thoughts to help with your data collection, engagement, and retention efforts:

• It may be worth recording instances in which players start but do not finish character generation. Otherwise, you won’t know how many players aren’t even making it past that first interaction with your game.
• If your game allows multiple characters per player, you may want to track metrics for both player accounts and characters.
• If your game flags people as AFK or idle, you may want to track the amount of time spent AFK versus not.

These are all things you can do with code, but you may want to go a step further and actually prompt players for information, either in the game or outside of it.

Prompting players for feedback

For example, here are some ways you could prompt players for feedback at different stages of the player life cycle:

• If a player enters the game’s “quit” command within N minutes of creating a character, try prompting them for why they’re quitting so soon. When they answer, save their response, thank them, then log them out.
• N weeks after a player signs up for your game, send them a link to a feedback form to ask them for their newbie feedback while it’s still fresh in their mind.
  • You’ll probably want to automate this, which you can do using your in-game message board, email, or Discord integration, depending on the info you collect at account creation.
• If an account is less than N months old, and a player stops logging in for X weeks, send them an email with a link to an anonymous feedback form asking them why they stopped playing and what it would take to get them to return.

If you decide to collect email addresses, keep in mind that they count as personally identifiable information (PII).

While the ideas above won’t directly improve your player activity metrics, they can help you collect valuable clues about why players don’t stick around or engage, which you can then act on as part of your player retention or engagement strategy.

I hope this week’s post provided you with some food for thought. Good luck, and happy MUDing!