How to pick the right UX research method
There are plenty of arguments to advocate for user research: it saves money, it keeps opinions out of the door, it makes you learn fast and so on.
However when you are involved in a project or coaching people that are passionate about user research and don’t need to be convinced, but that are nevertheless in the delicate unconscious or conscious incompetence stage, what are the best strategies to help?
How do you say that, yes, the 5-people rule generally works but not for everything, without getting the puzzled frown look in people’s face?
How do you talk about data triangulation in a way that newbies understand and trust you?
Sometimes the problem is that we tend to rely on the same research methods without taking too much time to properly plan our research strategy. This might be because of the overwhelming number of research methods that make teams feel lost. Might even be that it is tempting to repeatedly use the cheapest methods or the ones that are more familiar to us.
Whatever the reason, poor research planning increases the risk of collecting poor data which will be of a little help to the project.
In this short post I want to share with you a tool I’ve been using to help me express the thinking behind the design of a research plan and describe the reasons behind the choice of certain research activities instead of others. Within our team we called this tool the research cheat sheet.
What is it?
It’s essentially a table that brings together product development phases, types of user research (generative, formative, summative), types of data (behavioral and attitudinal), types of question to answer (quantitative or qualitative) and research methods. The idea behind it is that all the connections and relationships among these concepts can easily be spotted and put into practice when planning user research.
Think of user research planning a bit like planning a trip. First thing you’d do is to think about where you are and where you want to go, on the basis of that you’ll find your path in a map. You’ll then plan what to bring, what to do and who to take with you.
For example, if you are in the early phases of your project and you are defining the user problems to focus on, you better bring some data scientists with you as you are probably going to need to do some quantitative research and need to familiarize with concepts like “statistical significance”.
Some general observations can help us familiarize with the map:
- Whatever the product phase, you’ll be much better off with a combination of quantitative and qualitative data.
- In any of the product phases, it isn’t a good idea to only collect attitudinal data as what people say they do it’s different than what they actually do. This doesn’t mean attitudes aren’t important but they definitely need to be supported by behavioral data.
- When testing a prototype (implement phase), in most of the cases, it’s not really useful to discover what people think of your idea, which should have been done before development work. At the implementation phase in fact, it might be much more useful to observe if users can use your product OK rather than ask them what they think of it. Hence collecting behavioral data should be the priority.
- Attitudinal data are always better collected quantitatively.
What can the research cheat sheet be useful for?
Establish a common language
The major problem I needed to solve was the lack of a common language between people.
Thankfully it’s more and more common to involve people from different disciplines when planning UX research. This has enormous advantages which I won’t go into details here. However the difficulty sometimes lies in the fact we don’t speak the same language.
Nevertheless, despite the different backgrounds and the different levels of maturity in user research, the fact that the development of a product goes through different phases and cycles it’s generally something everyone is very familiar with. Therefore, I thought that explicitly connecting UX research to development phases, it should make more evident how different types of user research and different types of data are needed to support the development and team decision making process.
In fact, by using the right research methods teams will benefit from the right data points and they will be able to make user-centered decisions necessary to move forward.
Setting this common ground should make it easier to move the conversations from “how many interviews should we do?”, to “What type of answers we need?”. In other words, a thorough planning process won’t just help teams get the right data, but it’ll make them proactive about important aspects of the planning phase. One example could be that by properly planning research teams are able estimate the budget needed in the next six-twelve months, rather than saying “we woke up this morning and we decided we are going to do 5 interviews starting from tomorrow”.
User research isn’t just qualitative
In our team we want to be intentional about improving the research practice, which means we need to know our strengths and our weaknesses and work on the skills we need to acquire.
The research cheat sheet has been a useful tool to highlight what research we do the most and what’s missing. This has helped to start out constructive conversations about what to improve. For example we started to talk more in depth about data triangulation and what it means for us.
We’ve decided we need to work on our quantitative research skills and proactively improve the quality of data we gather.
The implementation phase isn’t the end.
After months or years of development, optimizations, iterations and user research sessions surely that must be the end of it.
Actually, to avoid the experience rot effect it’d be good practice to schedule summative research and evaluate the experience as a whole.
An example of this type of research would be a longitudinal experience benchmark.
Again, also in this case the research cheat sheet can be useful to visualize gaps in your research strategy and help to be more proactive.
Visualize and communicate when and why research won’t give you reliable data
Our main mission as UXers, whatever job title we have, is to bridge the distance between executives and users. However, this means we must be quite sophisticated about UX research methods choice to be able to bring reliable data needed to inform decisions.
Has it ever happened to you to see someone planning to speak to 5 people to gather their thoughts and feelings on something and then use the data to draw conclusions?
It did happen to me. I found it quite difficult to reverse engineer decisions already made based on the well known rule “5 users are better than nothing”. Actually it is not true: in some cases attitudinal data collected from 5 or less users is as dangerous as no data. The problem there isn’t just the small sample size but also the fact that opinions and feelings really depend on the context and individual characteristics.
Therefore, in order for attitudinal data to be useful, they need to be connected to real behaviors in a realistic context of use. For example people might say they like your product very much and you might think you are out of job, or that they hate everything and you might think you failed everything. If you put users’ opinions into context by observing them, you’d be able to detect problems to fix and strengths to enhance.
To conclude, the research cheat sheet can help also in the case you need to make sure there is nothing off in the planning of your trip towards your data.
Thanks for reading. If you end up using the research cheat sheet I’d love to hear about your experience!
Happy research planning everyone.
