news details

 
A key step in creating your Xbox LIVE® Community Game is playtesting. This is a time to share your game with other Creators to figure out what works, and perhaps more importantly, what doesn't work.

We had questions about this process and so we sat down with Frank Savage, the XNA® team’s Zen master on making and breaking games, to learn a bit more about the playtesting process.


In your own words – what is playtesting?

Playtesting a game comes in two flavors. The first is a comprehensive functional pass over the entire game to make sure that the game handles different hardware configurations and generally doesn’t crash. The second is a comprehensive playability pass over the entire game to determine if the game is well-balanced, if the player can both win and lose, and if it is fun to play.

What makes a good playtest?

The key to great playtesting is making it comprehensive. You want to play as much of the game as you possibly can, in as many ways as possible.

For example, during the functional pass:

• try plugging different memory units into the console.
• try the game on different TVs and different monitors with different AV cables.
• try the game both wired and wireless to make sure multiplayer works with both settings.
• try different controllers to make sure the game doesn’t crash, and that it supports the controllers that make sense for the game.
• bring up the Xbox 360® guide and try changing the game’s soundtrack to make sure that option works.

There are dozens of simple tests you can perform to make sure the game can handle a wide variety of different configurations.

The playability pass is typically much harder because it is primarily a subjective test of the game’s playability and fun factors. Your humble opinion matters much more on this pass since you are providing feedback on aspects of the game other than pure functionality. This part isn’t about hard and fast rules. It’s about playing the game to see if you have fun, if the gameplay is too hard, if you can play all the way through, and if you can save the game successfully.

Both of these kinds of playtesting are important for different reasons. The functional test ensures that the game works and doesn’t crash. The playability test ensures that the game is fun and is winnable with a reasonable amount of effort.

What can you expect from a playtest?

You can expect a lot of extra and occasionally unnecessary information from playtesters. This doesn’t mean that their feedback isn’t important but it does mean that you will get feedback on things that may not matter to you as the game developer. Many playtesters will report issues with colors in the interface ("It’s not blue enough!"), problems with the landscaping ("NO one would plant that kind of shrub next to that kind of tree, it just isn’t done!"), and problems with the weather (“It’s winter and that plant is clearly deciduous, but it has leaves!”).

The game developer will often make a critical mistake here and disregard all of the feedback from playtest rather than sift through all the comments looking for feedback on things that really affect the playability of the game. I’ve made this mistake myself. Developers should look through all the comments to determine the importance of each problem. If they are unsure, they should ask the playtester! Nothing makes playtesters feel better than to know they are being listened to.

What are some things developers should ask playtesters to do?

Give the playtesters specific things to test. This will cut down on the fluff, and it will help ensure that you get a good pass on your game. Ask some playtesters to play to lose as well as to win. Ask some playtesters to focus on specific levels, or ask them to test 5.1 speaker playback with the game so you can make sure it works, or ask them to play on different TVs, ranging from very small standard definition sets to big screen 1080p monsters. Ask them if the sound levels are OK throughout the game. Ask if level three seems too hard or too easy to them. Make sure they have definite targets to hit.

You can submit a game to playtest as many times as you want but you only get one chance to make a good first impression with players. Make sure that first four minutes is a great experience by asking the playtesters to check out demo mode.

As a tester, what should you be looking for? What sort of feedback is the most helpful?

As a functional tester, your job is to break the game. Try everything you can think of to make it crash. Unplug the controller in the middle of gameplay, try the game with and without memory units, try multiplayer with both wired and wireless connections. Do crazy things during the game like switch weapons over and over again for a minute or so, or leave the game running all night and see if it’s still running in the morning. Look for places to break the game’s logic.

For example, if you have to cross a bridge, can you destroy it before you get there? If you can, do you automatically lose, or are you stuck at the bridge with no way to continue play, forcing you to exit the game? What happens if you blow up the bridge while you are standing on it? Can you kill an important non-player character that you have to talk to later in the game? Can you throw away a key item that is required to finish some part of the game? When you’re focusing on playability, test whether the game follows its own rules. For example, if the game is a digital version of an existing board or card game, can you do things the rules wouldn’t allow? Does the game treat its own rule sets consistently? Can the AI do things that the rules don't allow?

You should also try to ensure that the game is fun. Give the game developer lots of specific feedback on why it is or isn’t fun. Suggestions are also helpful. Things like, the game would be 1000x more fun if you simply used the A button to fire instead of the trigger. The more specific feedback and suggestions you can give to the developer, the more the game can be improved. The developer will be looking for patterns in the feedback. Things like, everyone reports that level three of the game is too hard. The more data you give the developer, the better the game will get!

How do you know when a game passes?

What you are testing for is a good game play experience where you can actually play all the way through the game, you can win or lose the game, and you can have fun playing it. Note that both must be true!

A game is never actually done because there are always things to improve, and new ideas or skills that the developer can use to improve the game. There may still be issues with a game that the developer can’t or won’t fix, but they shouldn’t get in the way of playing the game or of having fun. These two are related because a game can be a tremendous amount of fun to play but takes ten minutes to save. Players may give up because they don’t want to wait that long for each save, or they may stop saving frequently, and get frustrated because they die too often.

The problem doesn’t even have to be something this big. Games can also give players a “death by a thousand paper cuts” experience when something they have to do over and over again is just painful to accomplish or takes too long. Imagine that reloading your gun involved flipping down to reload, bringing up a menu with all of your weapons on it, selecting which gun to reload, and then finally reloading. Hitting the X button would probably be a better and faster way to do this, since the player probably just wants to reload the gun currently in use. Often playtesters will find these kinds of things, and the developer will ignore the feedback.

Remember that as the developer, you will play your game more than anyone else does until it releases. What seems easy or simple to you, may not be simple for the playtesters!

How often should you playtest your game? Should you wait until you're done, or playtest each part individually?

You should always have at least one playtest for your game, and the best time to playtest is when you think you are done. You almost certainly won’t be done because the playtesters will find issues with the game.

How many additional playtests you have depends entirely on how much effort you are willing to put into making your game the best it can be, and on how many issues the playtesters find. However, at some point you will experience diminishing returns from playtesting. For example, if after the third playtest, all of the reported issues are truly minor,you could spend another week or two fixing them, but you should balance that against the very real possibility of breaking something else in the game with those changes.

If you decide to do another rev, you might find that the fourth playtest version was much worse than the third version! If you ever find yourself in this situation, go back to third version and call it a day!

Again, as the developer you will have to decide if it’s worth the risk of destabilizing the game to fix something minor. If you have a successful playtest and the coverage for the game seems complete from the testers, it’s probably time to send it to Peer Review! If it isn’t broken, don’t fix it!

Got any good playtest stories?

I have dozens of great playtest stories from the trenches that could probably fill a book. I’ll give you my favorite two.

The first was from the Wing Commander era, and is a great example of an issue that didn’t really matter. In Wing Commander 1, when you returned to the Tiger’s Claw after a mission, the carrier was sometimes upside down, depending on your orientation when you were coming back. This was to be expected since the game was 3D and it was easy to get your ship into that state by simply rolling. A very persistent playtester at Origin kept filing bugs on this until the game shipped.

So, in Wing Commander 3, I purposely brought the carrier in upside down for several missions in the hopes of getting this bug again. Alas, no one complained about it. This is a great example of an issue that doesn’t really matter. You could land on the carrier regardless of its orientation in both games.
 
The second story is from the Strike Commander era, and is a great example of how to approach an issue when you think your game is done. We were running through the last set of tests before shipping the game to the duplicators, when I got a call that the cluster bombs in the training mission on the F-16 were exploding the instant you dropped them, which of course destroyed your plane. Since we had dropped hundreds, if not thousands, of cluster bombs during every other mission in the game, this bug was perplexing. I went into work and watched someone reproduce the bug and I noticed that they were flying over water in the training mission when they dropped them. This struck me as important because I had never seen the bombs dropped over water before. There was simply no reason to drop them over water as there were no targets in the water that a cluster bomb would damage severely.

I went and dug into this issue, only to discover that I had indeed put a bug in the code almost eight months earlier! The cluster bombs were supposed to detonate in the air and scatter their munitions over a wide area. Instead of code instructing the game to “detonate when you reach this altitude or go below it”, what I had written was code to “detonate if the terrain is ever lower than this altitude”! Since the terrain was always higher than the minimum altitude specified in the code in every other part of the game except over water, the bombs usually dropped fine and detonated when they hit the ground. This was not the desired effect, but we had tweaked the entire game to play well with the cluster bombs hitting the ground instead of going off in the air. No one knew the bug had been there for the last eight months.

Making the code “right” by changing it to act as I had intended would almost certainly have made the cluster bombs behave differently everywhere. This might have made them more effective or less effective, but only a complete play through would have been able to make sure this feature wasn’t broken, and we were scheduled to ship the game the next day!

We decided to remove the line of code that made the bombs detonate over water, instantly destroying your plane, and left the rest of the code the way it was. It wasn’t “accurate” but we knew we had a good game with the cluster bombs behaving the way they did over land. This also fixed any possible issues from a player inadvertently dropping the bombs over water and blowing themselves up! In this case, the right thing to do was to fix the small thing that was clearly broken, and leave the rest of the bomb’s behavior alone.

Many thanks to Frank for taking time out of his busy day to help answer some questions and provide pro-tips on putting your game through Playtesting!

What are you waiting for? Submit your game now!