Evidence-based interviewing deep dive: The case against some common interview questions
Why you should collect specific stories about past behavior in interviews and how to do it
I’ve only interviewed one job candidate in my life, but I’ve interviewed many more customers and users. That’s why my cousin’s recent job interview experience didn’t perplex me as much as it perplexed her:
“For every question, they wanted me to share concrete examples. Every question. I struggled to come up with so many stories. Like when they asked me about a time I’d missed a deadline, I had to share a story from personal life, because I’ve just never missed a deadline at work.”
To be fair, she’d only had a one-hour notice for the interview due to a misunderstanding; she’d barely had time to put herself together, let alone prepare for the questions. No wonder the interviewing style had caught her off guard.
But when my cousin was finished with her rant, I asked her if she knew why the interviewers were so eager to hear specific examples from the past. She said she didn’t.
“If you ask someone how they tend to do X or how they’d approach situation Y — two extremely common interview questions — they’ll tell you how their ideal self, or the self they have a conscious representation of, would behave. These descriptions may, however, have little to do with how they’d actually behave.”
So I told her that social psychologists have found verbal reports, like the stuff that comes out of interviews, to be, well, a bit tricky. If you ask someone how they tend to do X or how they’d approach situation Y — two extremely common interview questions — they’ll tell you how their ideal self, or the self they have a conscious representation of, would behave. These descriptions may, however, have little to do with how they’d actually behave.
My guess is that to counteract this phenomenon, the interviewers had asked my cousin how she’d done X or approached situation Y in the past, and to name an example. That’s because past behavior is the best proxy for future behavior. Or at least better than generalizations and predictions.
This technique — fishing for specific stories about the past—is often recommended in product discovery books like the Mom Test and Continuous Discovery Habits, which is why it has become my go-to technique as a user researcher. But although these books offer some reasoning for it, they rarely cite the science I find most convincing and compelling. I’d like to fill that gap.
I’d like to tell you the longer version of what I told my cousin, an evidence-based answer to the question, “How should you formulate interview questions if you want to learn about a person’s behavior?” It’s largely based on the article Telling More Than We Can Know: Verbal Reports on Mental Processes by Richard E. Nisbett and Timothy D. Wilson, which has been cited a whooping 17 315 times.¹
I’ve organized this deep dive on evidence-based interviewing under the following four points and a fifth, and final, section on how to apply the science in practice:
- We can’t access higher-order mental processes such as how we make decisions, form preferences, or solve problems
- We often tell more than we can know
- Our answers are based on what’s most plausible or accessible in our minds
- So, to uncover valuable insights, collect stories about past behavior
- The questions you shouldn’t be asking and how to turn them into good (if not great) ones
As far-fetched as it sounds, I want to start with a story about an art project I did a few years back. It’ll help make the case against some common, but mediocre, interview questions and offer a foundation for formulating slightly (or sometimes tremendously) better ones.
We can’t access higher-order mental processes such as how we make decisions, form preferences, or solve problems
Eight years ago, when I was studying design in my bachelor’s degree, I did an art project related to memory. The project was inspired by an observation I’d made about the peculiar workings of my mind: I’d be eating lunch at the university cafeteria when all of a sudden I’d be reminded of an afternoon at high school when my friends and I joined the spring assembly tipsy, because we’d been drinking wine at the local bar during a free period (don’t worry, we were of legal age, because we’d all been on exchange for a year). Or I’d be lying on the couch reading a novel, when memories of the outdoor swimming pool of my childhood would suddenly visit me: the light blue changing rooms, the shallow swimming pool with a slide, and the smell of the crimson rubber covering the playground.
Where did these memories come from? And why did they pop up at this particular moment? It felt as if the physical world — the lunch tray at the cafeteria or the 93rd page of Normal People — acted as portals to the past. This (something akin to the Proust Phenomenon) was a mystery I wanted to solve.
So for my art project, I asked friends and acquaintances to depict how they believed their memory to work. I booked one-hour sessions with them, gave them some paper and crayons, and asked them to draw while we talked. These sessions turned into an adorable collection of sketches — boxes connected with lines, cloud constellations, a spiral — which I hung up in the end-of-the-course exhibition accompanied by my own visual interpretation.
I’m not sure what I believed to get out of this project. Perhaps I thought I’d actually learn something about the information retention feature of our brains. Or perhaps I never thought it through. Although I haven’t since become an expert on cognitive science, I’ve learnt that my initial observations on the serendipitous nature of memory were much closer to the “truth” than any drawing could ever be.
“We have little to no access to higher-order cognitive processes like memory, creativity, and decision-making.”
I’ve learnt that we have little to no access to higher-order cognitive processes like memory, creativity, and decision-making. As the cognitive scientist and psychologist George A. Miller (p. 56) put it, “It is the result of thinking, not the process of thinking, that appears spontaneously in consciousness.”² I can observe that I’m transported to the outdoor swimming pool of my childhood while reading a novel, but I can’t know how or why my mind transported me there at this particular moment. No matter how hard I try, I can only observe the product of mental processes.
Due to this, Nisbett and Wilson point out the futility of asking people to verbalize higher-order mental processes: “It would be absurd, for example, to ask a subject — — whether he stored the meanings of animal names in a hierarchical tree fashion or in some other manner.”¹ What an absurd little art project, eh?
Most people would only try to depict their memorial processes to help a naive design student pass a course. They’d understand that such a task wouldn’t result in pictures of how memory works, but pictures of how people believe their memory to work.
But what about more mundane questions and answers? Would it be equally absurd to ask someone why they skipped the salad section at a buffet or why they chose to travel to Tel Aviv for vacation?
We often tell more than we can know
In everyday life, we readily answer questions about our decisions and preferences: “I skipped the salad section because salad is boring,” “We have some family in Tel Aviv.” As Nisbett and Wilson put it (p. 232), “while people usually appear stumped when asked about perceptual or memorial processes, they are quite fluent when asked why they behaved as they did in some social situation or why they like or dislike an object or another person.”¹
What’s more, we don’t tend to question the validity of our answers to such questions. If a friend of mine explains that he liked the person he recently went on a date with because the guy seemed smart and funny, I wouldn’t question his reasoning and insist on a potential alternative explanation: “Maybe you liked him because he had dark eyes and long curly hair. You tend to be attracted to people who look like that.” (Or, perhaps I would if he has a tendency to get infatuated with people who look a certain way and to ignore other factors.)
“Verbal reports can be quite unreliable, especially when we’re explaining the influences on our behavior and decision-making as well as our preferences or hypothetical future behavior.”
Anyway, the science suggests that we should perhaps be a little more wary of people’s answers. Verbal reports can be quite unreliable, especially when we’re explaining the influences on our behavior and decision-making as well as our preferences or hypothetical future behavior. ¹ Let’s look at a few examples.
In a classic study on human reasoning, Norman Maier showed that people may be incorrect about what influenced their solving a problem.³ He asked participants to solve a puzzle which required bringing together two pieces of cord hanging from the ceiling at opposite ends of a room. The puzzle had several solutions, one of which was the trickiest to discover: it required bringing one of the cords into a swing by attaching a weight to it.
For the participants who didn’t come up with the (tricky) solution in ten minutes, Maier gave a hint: He walked across the room and nonchalantly set one of the cords in motion. Consequently, the majority of the participants figured out the solution and did so quite quickly (it took approximately 40 seconds), indicating that the hint had worked. One group a participants was given a second hint as well: Maier twirled a weight to the end of one cord and suggested that it would help find the next solution. The first hint — setting the cord in motion — was a lot more effective even though it wasn’t even labeled as help.
The study’s most interesting finding relates to how the subjects who were given hints explained how they’d arrived at the solution. When only the effective hint was given, about a third reported it as influential. But when both hints were given, the majority of the participants reported the ineffective hint as influential. In other words, they were wrong about the actual sequence of events; their reasoning was misconceived. A similar unreliability has been found in other types of verbal reports, too.
In a study on the halo effect⁴— the tendency for an evaluation in one area to influence evaluations in another in the same direction as in “She’s intelligent so she must also be diligent, dependable, and hardworking” — Nisbett and Wilson showed that people may incorrectly report on the reasons for their preferences.⁵ The subjects watched a video in which an alleged psychology instructor was interviewed and were then asked to rate him on different dimensions. When the instructor responded to the questions in a cold and rigid style, his appearance, accent, and teaching style were evaluated as more negative than when he responded in a warm and enthusiastic style. In other words, the research participants judged the warm instructor more favorably.
But when the subjects were asked about whether the speaking style had influenced their evaluations, the majority reported it as having had no effect on them. In fact, they reported the opposite effect: that their evaluations of the instructor’s attributes, such as his Belgian accent, had influenced their liking — which the study could prove not to be the case. In follow-up interviews, most subjects firmly denied that the speaking style could’ve affected their ratings.
There are many more studies which show that our explanations don’t always align with observable reality. Many of these explore how we explain the past, how we attribute causes to events, behavior, or preferences. But a similar unreliability has been detected in verbal reports about the future. Here are a few highlights:
- The planning fallacy: We underestimate how long it takes to accomplish tasks.⁶
- The durability bias: We predict emotional states to last longer than they actually do. For example, we underestimate our ability to feel happy again after a negative a event like losing a job or a partner.⁷
- The hot-cold empathy gap: We underestimate the influence of our visceral state, such as hunger, pain, and sexual drive, on future behavior.⁸
Research clearly casts doubt on our ability to accurately report why we behaved the way we did, why we like what we like, and how we might behave in the future. But since we fluently answer these questions, what are our responses based on?
Our answers are based on what’s most plausible or accessible in our minds
Research indicates that when we’re asked about our past behaviors or decisions, we may answer based on internalized causal theories, we say what’s most plausible.¹ These theories may be adopted from the surrounding culture and be either implicit (“I can’t lose weight because I lack self-discipline”) or explicit (“If you burn more calories than you consume, you lose weight”). Or they may be based on personal observations of stimulus and response (“I get hangry when it’s been over six hours since my last meal”).
When asked a question, we may also search our minds and respond based on our most accessible thoughts, feelings, and memories.⁹ If I ask someone why they chose a direct flight, they may say that they find layovers stressful and time-consuming, when in fact the airline’s website design has played a major role in their decision.
What about predictions of future behavior, preferences, or feelings?
Prominence plays a role in these as well. We easily succumb to focalism, considering an event in isolation and failing to appreciate the other things that may be occurring simultaneously.¹⁰ We may imagine being elated at a music festival, not realizing that when the day arrives, there may be a thunder storm and we may have had a fight with our partner the night before.
Even when we’ve experienced a similar event before, we often fail to acknowledge it when making predictions.⁶ When choosing whether to attend one, two, or three days at a festival, we may not think back to how much we despised the rowdy crowds and the 20-minute food truck queues the previous summer.
We also tend to overemphasize desirability as opposed to feasibility when it comes to distant future events.¹¹ We may buy tickets to three music festivals in February thinking “music plus friends plus sun equals fun” not considering the arduous practicalities involved: finding a cat sitter, booking cheap accommodation, and purchasing public transportation tickets.
As we can see, numerous biases can spice up our answers. This is OK when we’re explaining to a friend why Lawrence of Arabia is our favorite movie or how we ended up choosing between a job at a startup and a job at a corporation. But is it also OK when we’re interviewing job candidates for a product manager position or trying to build a viable streaming service?
So, to uncover valuable insights, collect stories about past behavior
The research we’ve covered so far does read a bit like a warning: Be careful what you ask about. That’s also what the authors of the seminal article Telling More Than We Can Know contend (p. 247): “- — [T]he evidence indicates it may be quite misleading for social scientists to ask their subjects about the influences on their evaluations, choices, or behavior. — — [S]uch reports, as well as predictions, may have little value except for whatever utility they may have in the study of verbal explanations per se.”
Now, most of us aren’t social scientists. But if you’re reading this article, you’re probably interested in uncovering reliable insights in job or user interviews. You want to understand how a product manager candidate goes about motivating his team or how a Gen Zer travels — not what they believe or think about these topics.*
So let me repeat the authors’ recommendation in plain(er) English: If you don’t want misleading responses, you should refrain from asking people why they behaved or chose the way they did or how they think they’ll behave or choose in the future. That’s because the responses to such questions aren’t useful unless you’re studying how people explain things.
I can hear someone crying out a boomer-style complaint as a response to this: “All these forbidden questions! Is there anything I am allowed to say?”
That’s a good question — which the authors don’t directly answer. But I’ll give it a try based on what I’ve read so far.
The most direct advice — which can still feel a bit far-fetched for our context, but bear with me — comes from the book Strangers to Ourselves: Discovering the Adaptive Unconscious by the second author Timothy D. Wilson. In the book, he recommends observing behavior as a recipe for “knowing thyself,” understanding who you are and how you feel about something.¹²
For example, if you want to understand where you lie on the extroversion-introversion scale, you can observe how you spend your time. If you regularly attend salsa parties, go bouldering in a group several times a week, and organize tea tastings for your friends (and enjoy all of these social activities), you’re probably pretty extroverted even if you identify as shy and think you enjoy spending time on your own. The behavior (and the emotions accompanying the behavior) give you a more accurate image than simply asking yourself, “Am I more of an introvert or an extrovert?”**
“In interviews, the technique means asking people to recall a past behavior instead of asking them to examine their thoughts and feelings about that behavior.”
I believe we can also apply this technique to knowing thy neighbor, a.k.a. your customers, users, and job candidates. What’s the technique exactly? It’s recalling instead of introspecting. In interviews, it means asking people to recall a past behavior instead of asking them to examine their thoughts and feelings about that behavior.
The research on verbalizing how we feel has come to similar conclusions. A paper that looked at multiple studies on emotional self-report concluded that data collected while the emotion is being experienced is the most reliable: It’s better to ask someone how they feel about salsa when they’re on the dance floor as opposed to when they’re folding laundry.¹³ But when you can’t collect data in real time, like when you’re interviewing someone, the second best option is to ask them to recall past emotions. Hypothetical questions (“How do you think you’d feel at a salsa party?”) and future-oriented questions (“How do you think you’ll feel at next week’s salsa party?”) tend to lead to less reliable answers because they tap into beliefs instead of experiences.
Obviously, biases and errors can swim into retrospective reports, too. Interviewees may, for example, remember things incorrectly or leave out the details they find embarrassing. But if you’re going to do interviews — and need to rely on verbal reports — fishing for specific stories will probably lead to the most accurate insights about behavior.
So here’s the main takeaway:
To keep interviewees from telling more than they can know, center questions around recalling instead of introspecting. Ask them to think of a specific (near) past situation, decision, or behavior and then to describe it in detail. Do this even when you’re trying to understand how someone might behave in the future.
So how can we apply this new knowledge when crafting an interview guide?
The questions you shouldn’t be asking and how to turn them into good (if not great) ones
I want to end this article with a few practical tips derived from the research we’ve covered. Below, you can find four rules of thumb on what not to ask as well as examples of turning mediocre interview questions into good (if not great) ones.
Don’t ask how someone “generally” behaves.
🔴 Don’t ask this: “I saw from your CV that you’ve run design sprints. How do you conduct them?”
🟢 Ask this instead: “I saw from your CV that you’ve run design sprints. Could you tell me about the last design sprint you did? How did you go about planning and running it?”
Don’t ask how someone might behave in a hypothetical situation.
🔴 Don’t ask this: “How would you approach a situation in which you realize that you have too much on your plate and won’t be able to finish it all before the deadline?”
🟢 Ask this instead: “Could you tell me about a past situation in which you realized that you have too much on your plate? How did you go about resolving the situation?”
Don’t ask how someone might behave in the future.
🔴 Don’t ask this: “Would you use a chat bot to find answers to questions about our product?”
🟢 Ask this instead: “Have you ever been in a situation in which you needed to seek help regarding our product?” If yes → “Think of the last time you had a question about our product. How did you go about finding an answer?” (Follow-up question: How did that work for you?)
Don’t ask why someone behaved the way they did.
🔴 Don’t ask this: “Why did you decide to buy our product?”
🟢 Ask this instead: “I’d be curious to know more about how you ended up buying our product. How did you first find out about it?” (This is only the beginning. I’d also be interested in any of the following issues: what was going on in the organization before the interest in such a product emerged, how they were working prior to purchasing the product, which other products they considered, who was involved in the decision-making process, how long the entire process took from initial idea to purchase, etc.)
Later that day when my cousin had vented about her peculiar job interview experience, I received a text from her: “I feel so embarrassed that I didn’t know this interviewing technique. Now I feel dumb. I should’ve known better.”
No, she shouldn’t have. Few people know about it. Heck, the interviewers themselves might not have known about it; they may have just googled “best job interview questions” and used the technique without understanding what it’s based on.
But yes, knowing — and understanding — the technique can be useful.
It can be useful when preparing for job interviews as a candidate or an interviewer. And it can be useful when preparing for customer and user interviews. Because as we’ve learnt, some stories are more reliable than others. Let’s leave the fantastical stories to the artists and the artists to (wanna) be.
Notes
*Of course, you may be interested in many other things than behavior in interviews. You may want to understand the expectations a product manager candidate has for the role or the beliefs a person with type-2 diabetes has about weight loss. Go ahead and ask these questions. Just keep in mind that ideas that inhabit the conscious mind, like expectations and beliefs, may not influence behavior — at least the more automatic and habitual behavior which is driven by unconscious processes.¹⁴ Consider pairing questions about thinking with questions about actual past behavior to get a broader, and hopefully more realistic, idea about the person you’re talking with.
I would also recommend complementing the interview insights with one or several of the following methods (in ascending order of effort):
- Exploring data on user behavior (i.e., product analytics)
- Collecting people’s actions/emotions in real time (i.e., daily mood check-ins)
- Conducting diary studies
- Conducting observations
**There are situations in which behavior isn’t the best marker for the “truth.” If you, for example, engage in all kinds of social activities and constantly feel drained and overwhelmed, you might not be highly extroverted despite exhibiting extroverted behavior — you may just have a misconception about how you enjoy spending time.
References
- Nisbett, R. E., & Wilson, T. D. (1977). Telling More Than We Can Know: Verbal Reports on Mental Processes. Psychological Review, 84(3), 231.
- Miller, G. A., & Buckhout, R. (1973). Psychology: The Science of Mental Life. Harper & Row. Accessed On
https://archive.org/Details/psychologyscienc00millrich/Page/58/Mode/2UP?View=Theater - Maier, N. R. (1931). Reasoning in Humans. Ii. The Solution of a Problem and Its Appearance in Consciousness. Journal of Comparative Psychology, 12(2), 181.
- Thorndike, E. L. (1920). A Constant Error in Psychological Ratings. Journal of Applied Psychology, 4(1), 25–29.
- Nisbett, R. E., & Wilson, T. D. (1977). The Halo Effect: Evidence for Unconscious Alteration of Judgments. Journal of Personality and Social Psychology, 35(4), 250.
- Buehler, R., Griffin, D., & Ross, M. (1994). Exploring The” Planning Fallacy”: Why People Underestimate Their Task Completion Times. Journal of Personality and Social Psychology, 67(3), 366.
- Gilbert, D. T., Pinel, E. C., Wilson, T. D., Blumberg, S. J., & Wheatley, T. P. (1998). Immune Neglect: A Source of Durability Bias in Affective Forecasting. Journal of Personality and Social Psychology, 75(3), 617.
- Loewenstein, G. (1996). Out of Control: Visceral Influences on Behavior. Organizational Behavior and Human Decision Processes, 65(3), 272–292.
- Wilson, T. D., Hodges, S. D., & Lafleur, S. J. (1995). Effects of Introspecting About Reasons: Inferring Attitudes From Accessible Thoughts. Journal of Personality and Social Psychology, 69(1), 16.
- Wilson, T. D., Wheatley, T., Meyers, J. M., Gilbert, D. T., & Axsom, D. (2000). Focalism: A Source of Durability Bias in Affective Forecasting. Journal of Personality and Social Psychology, 78(5), 821.
- Liberman, N., & Trope, Y. (1998). The Role of Feasibility and Desirability Considerations in Near and Distant Future Decisions: A Test of Temporal Construal Theory. Journal of Personality and Social Psychology, 75(1), 5.
- Wilson, T. D. (2002) Strangers to Ourselves: Discovering the Adaptive Unconscious. The Belknap Press of Harvard University Press.
- Robinson, M. D., & Clore, G. L. (2002). Belief and Feeling: Evidence for an Accessibility Model of Emotional Self-Report. Psychological Bulletin, 128(6), 934.
- Wilson, T. D., Lindsey, S., & Schooler, T. Y. (2000). A Model of Dual Attitudes. Psychological Review, 107(1), 101.
