Prototype testing — the complete guide
So, you’ve worked relentlessly on creating your new product or adding the coolest new feature and are impatient to hand it off to development. But are you actually confident that what you’ve designed solves a need, is going to be usable, and have you considered all the flows the users might go through to achieve their goals?
To feel more confident in your design, consider building fake façades, imitations of products that will capture the experience without going through the extensive process it takes to build the real thing just yet. Prototype tests do just that.
Why do it?
✅ To validate the ideas
✅ To minimize costs
✅ To uncover opportunities
✅ To get unbiased feedback
✅ To discover minor bugs
How to do it?
Well, that’s what this article is about!
In my experience the easiest way to dissect a prototype testing experiment is by separating it into three phases:
- Script writing
- Preparing files
- Moderating
On their own, each phase has its best practices and popular pitfalls, but together they form the experience the test participant will have from the session and each will directly impact the results you obtain.
Script writing
1. Test for what’s important, focusing on a few high priority hypotheses
Create a hierarchy of priorities of hypotheses you need to validate foremost before moving forward with product decisions. Especially if you have a complicated product, de-emphasize the actions you’re not prioritizing to test within that research session and focus on topics that will enable you to tick the big questions off the list.
When considering the priorities, remember to get the input of other stakeholders, since there may be hypotheses that the Product Manager, UX Designer, or Engineering Manager may have that you didn’t consider previously but which are critical for overall product development.
Focus on a maximum of 2–3 topics for the research during each session. Jumping from one topic to another will break the storytelling process of the user and might result in missing information that the interviewee would otherwise give outside of just answering the questions or performing tasks.
Also, be careful not to fall for the loophole of having few but very vague goals. It’s one thing to say, “I want to test the prototype,” and completely different to say, “I want to explore how the users navigate through the hospital signup flow when they are in an emergency.” Unfortunately, you won’t be getting far with the first approach.
2. User test VS Usability test
Once you have your goals in mind, it’s important to decide what exactly you’re trying to achieve — gathering qualitative feedback or a list of usability issues and ideas. Most prototype testing sessions I’ve observed have focused on the usability of the prototype, trying to see if the product is usable or not, how participants interact with it, where they get stuck, etc.
Qualitative user research sessions on the other hand would be focusing on the need of the users in the first place — what are the user’s expectations, how do they perceive the product, when do they imagine they might need it.
While both test formats are completely legitimate and useful, it’s important to decide what you want to focus on. Is it to validate the need in the first place or to see if the users can add items to the cart and proceed through checkout?
3. Start with a mindset or scenario and utilize the Jobs To Be Done Framework
Participants must have a certain level of contextual understanding before they jump into testing your product. When will they feel the need for your product? Where will they be? Are there any external or internal influences to be expected? If you’re familiar with the Jobs To Be Done Framework, this could be the perfect time to use a Job — what is the thing the user is trying to achieve in their life when using your product?
4. Ask participants about their expectations
This can be done on a more general or local level. Explore what the participants expect to happen after they click a button or when/how they expect to interact with the product. Questions on expectations are crucial, especially towards the beginning of the conversation before the mental model of the participant adjusts to the product you’ve presented them with.
Some great follow-up questions to a task the participant has performed are to probe, “does this meet your expectations?” and “what did you expect instead?”
5. Dry run
Most of the time, prototypes aren’t as complete with all of the actions users might consider doing during the test. The best way to minimize wasted time re-running interviews and usability sessions with your precious users is to perform a dry run (also known as a pilot test) with other people before the session. The pilot tester can be a colleague that hasn’t worked on your product or even a family member if your product category permits it. The important part is that you keep it as realistic as possible, using the same script as you plan to with the user.
Preparing Files
6. Pick the right fidelity level
The depth of the insights you gather from your prototype test is in direct correlation with the fidelity level of your prototype.
Low fidelity / Paper prototypes
Excellent for: Initial architecture and idea validation
Advantages: Quick, easy, no commitment, easy to modify
Disadvantages: Somewhat unrealistic experience, little visual presentation
Best used for: Small usability tests and initial product feedback
Wireframe or mid-level fidelity prototypes
Excellent for: Getting more product-focused feedback
Advantages: Building blocks can be reused and adapted quickly
Disadvantages: Dummy texts and wireframe components may provide an unrealistic experience
Best used for: Multi-page prototypes with different flows and user subtasks
High fidelity or complex prototypes (including developed product tests)
Excellent for: End-to-end user flow experience feedback that can include feedback on elements of UI
Advantages: Realistic, easy to test
Disadvantages: Hard to modify, drastic changes can be too late to implement
Remember, when building a prototype not EVERYTHING needs to be functional, but rather just enough to have a smooth testing experience to generate data points.
7. Use realistic, non-distracting content / scenarios
Form fields with Batman’s personal information and adding ‘lorem ipsums’ are easy options, but will keep the scenarios unrealistic, taking the concentration off from the product. Avoid these extremes whenever possible and test with content that will help users focus on the task at hand rather than on the distracting made-up content.
8. Don’t forget prototype-specific test factors
Most prototyping tools have a feature that highlights the clickable area whenever the user clicks anywhere on the screen. Obviously, this is not going to be the case when the product goes live and the hints may guide the users where to click to proceed, so you might consider disabling the highlights for a more realistic test.
Will the users be under any time constraints when using your product in real life? Account for that as well or you risk validating a solution that will be inconvenient in the real usage environment.
An example of this may be prototype testing ticket selling machines at train stations. These users are most likely under stress since their train is about to leave, there’s a huge queue behind them, and they’re not performing this type of action daily, so it is important to account for this during testing.
9. Test in the right environment
What is the context in which the users will be interacting with your product? The ideal place would be the actual environment in which the product will be used. Unless you expect the real use to be in a calm environment with no distractions, avoid testing at usability labs as much as possible and opt for more close-to-reality options, like testing at participant’s home, a busy cafe, or even in traffic (ensuring safety first, of course).
Another tip would be to test outside of your company premises, as the excitement of visiting a cool office or the view of a team looking busy working on the product may lead the tester to highlight positive feedback only.
Moderating
10. Informing users
Prototypes are different than final products, obviously. You know it, the Product Manager knows it, but the user might not be as familiar with the concept. It’s best to inform the user that:
- This is a prototype test so some of the functions will be disabled during the session.
- You want them to stick to the scenario, however, provide verbal feedback about anything else that they’d expect to see in the live product.
- In case you have the highlight feature enabled for the prototype, the clickable areas that the user will see will be highlighted. This is not part of the product itself but the testing tool, so remind them it won’t be highlighted when they use the product in real life.
11. Encourage participants to think aloud and share “It would be great if…”
Like with any qualitative research, the point of prototype testing isn’t just to receive answers to the specific questions you ask or tasks you give, but also to learn about the thought process of the testers, their expectations, what mismatches they experience, and more. Ask users to think out loud as they navigate the product and encourage them to share additional feedback and contribute ideas.
12. Reverse the questions addressed to you as a moderator
During any moderated prototype testing session, you will hear questions like, “can I do X with this?”, “why did this page reload?”, “what should I do to get to the next screen?” A LOT. You will naturally be inclined to answer them (at least some), but this is highly inadvisable. To begin with, the participant will stop feeling like the one doing the testing and will feel more like they are being tested, causing them to try to answer all the questions in “the right way”.
The best response you may have to such questions addressed to you is to reverse them and ask the participant what THEY think is going on. Variations of, “what do YOU think this change means?”, “what makes you say that?” and “how would you explain what you should do next?” are here to save you at such times.
Final Thoughts
In his book, Zen and the Art of Motorcycle Maintenance, Robert Pirsig has a quote that 50 years later, stays true, no matter the field of study:
“An experiment is never a failure solely because it fails to achieve predicted results. An experiment is a failure only when it also fails adequately to test the hypothesis in question or when the data it produces don’t prove anything one way or another.”
Similarly, the point of prototype testing shouldn’t be confirming what you expect and moving forward, but testing hypotheses and gathering data that will enable you to make informed data-driven decisions. Most of the time this will include iterating, iterating, and iterating again until you get a product version that has just the right balance of everything.
