"How do you test puzzles with players? Do you test the final puzzles or more just the concepts of puzzles?"
Thank you to Adam Stichter on twitter for this great question.
Like anything else, designers have different methods and processes for testing their puzzles and games. Before making The Tale of Ord my playtesting experience was limited to live escape rooms, so the process started out a little rocky and has grown more refined as I continued to design for tabletop gaming and making online puzzles as well. A lot of this relates to previous posts I’ve written, so I will include links that expand on certain points.
Step 1: The “Self-Administered” Test
At this point I've come up with an idea for a puzzle and drawn it out (usually by hand, but sometimes digitally if there is more than just a little text.) It might look something like this:
Then I sit down and pretend I am the player looking at it with fresh eyes. Obviously it's impossible to escape the things you already know about the puzzle, but doing this before giving it to another person might help you find a mistake or a gap in cluing. The above example is pretty simple--your goal is to figure out how much each item is worth by assigning numbers from 1 through 6. If you had a multi-step puzzle though, you would want to move through every step and make sure each one has enough signposting to logically get you to the next step.
Step 2: Solvability Test
Once the puzzle has been checked for mistakes, I hand it off to someone just to test the mechanic, and again check for errors. (I usually try to ask people who I know are not going to test the entire game so there are no spoilers later on.) The purpose of this step is to see if the concept makes sense and if there is enough information to solve it. In simple cases like this weight puzzle, it might be finding a mathematical error, an alternate solution, or maybe that you only require three out of the four examples to solve the puzzle. When my first tester found a second way of solving the above puzzle I had to rethink which examples were being presented so that the other possibility was eliminated.
Once I redid the drawing with four new scales I went back to step 1 and tried to test it myself, checking this time not only that there was enough information to find the correct solution, but for any other possible answers. When it seemed correct, I gave it to the same tester to double-check my work before passing it on to someone else.
The main purpose of the first two testing steps is to confirm that the puzzle is logical and fair. In short, the reasoning behind the solution makes sense and there is no trickery or guesswork involved. You can read more about what I mean by these terms here.
Step 3: Mockup/Clarity Test (WARNING: Slight Emerald Flame spoiler below)
If I know the puzzle generally works, the next step is to create a mockup. In this phase I test not only for solvability but also for difficulty, clarity, and solving time. The mockup usually goes through several iterations (being used for steps 3 and 4), so I like to make these digitally for easier editing. I try to make these as close as possible visually to what I expect the final product to look like, so that the player is as close as possible to final experience and I basically just need to follow a template when I'm making the final artwork. Here's what the mockup looked like after a few iterative steps:
As you can see, there are 4 main changes:
- The items have become shapes. This was done to better fit the connected puzzles in the game.
- The numbers on top have been listed out with commas rather than written as a range of 1-6. This was done to make it clear that there was only going to be one of each number, as opposed to leaving the possibility open that two shapes could be the same weight.
- Rather than being laid out in a line, the shapes are scattered in such a way that makes it clear they're not meant to be taken in any specific order.
- The scale examples themselves have changed to a configuration that is solvable with the exact amount of information provided and has only one possible answer.
I'm getting a little ahead here though, as these changes often come about as a result of steps 3 and 4. I usually test the first mockup just a couple of times to make sure it works before putting it aside until I'm ready to test the entire game (or chapter). As long as it works, there is still plenty of time for refinement in step 4, which is the most thorough of the testing phases as it combines all the puzzles into the context in which they'll be played.
Another thing to note is that steps 2-3 are not always possible to do with every puzzle. If the puzzle contains physical components or is impossible to complete without several other parts of the game (like a meta puzzle for example) then it usually goes straight from step 1 to step 4, and as a result those puzzles typically go through the most iteration in step 4. On the flip side, if the puzzle is a stand-alone (not part of a larger game) or is made for social media, etc. then there is no step 4 and the puzzle goes straight to step 5.
Step 4: In-Game Test
Once I have put together an entire chapter of a game, I write out the hints page so that playtesters can experience the game as they would under normal circumstances. (I have found that if people are testing without a self-service hint system, they are more hesitant to ask for hint even when they need one.) At this point I'll hand off the game to my first playtesters and make changes after each play-through. If you're doing blind playtesting you'll want to make sure that your testers are taking detailed notes about their thought process and where they get stuck. I find it really helpful to watch groups (as long as they're willing) because it's easier to catch details that might otherwise not get relayed. Without in-person meetings, I have found playtesting over video chat to still be more helpful than doing it blind. In some ways it's even better than doing it in person because it's easier for the players to ignore your presence and act naturally.
In this phase I always iterate between each playtest, even if it's a really small change, because sometimes what seems like a small change could end up leading someone down the wrong path instead of making the puzzle clearer. Remember that details matter when it comes to cluing.
Step 5: Art Test
Once a puzzle seems polished enough, I make the "final" artwork. I'm using quotes when I say "final" because this phase can still include changes, just keep in mind that changes will be harder with final art. This is the stage to double check the execution and make sure everything fits, reads, and solves just the way it's supposed to, down to nit-picky details like the letter 'u' looking like a 'v' somewhere.
Here's what the final art looks like for the puzzle shown above. In this case the puzzle didn't change from the last mockup version, so I was able to just use the same graphic as a guide to draw it by hand.
Step 6: Final Final Test
Now that everything has been thoroughly checked and you really think you're done, it's time for final testing before you actually send your files to the printer. Always always always make sure to test the finished files. I usually have someone proofread all the text first and then do a couple of blind tests so the players are not influenced by my presence whatsoever. If any errors, however small, are revealed at this stage, I fix those errors and do another test, until the finalized version is exactly how it should be.
So that's the process in a nutshell! What is your puzzle testing process like?