The scourge of the leading question
Remembering Loftus and Palmer (1974)
I knew the insights were bullshit and built on shaky foundations, but I was in a large meeting and felt the social pull of not pointing this out. But as the presentation went on — our research partners talking through key findings and insights from some recently completed research— I couldn’t hold it in any longer.
‘Can I stop you there. Do you think that that is valid, given how the question was asked?’
Cue embarrassment, excuses, before I pointed out to the room that because the question asked was so leading — and I knew because I was there when it was asked — we had to take that insight with a lot of salt, never mind a pinch.
What a wasted opportunity. And time. And money. All because questions were asked badly, leading to results we couldn’t rely on.
Why you should care
Leading questions have huge impacts for research and can completely invalidate findings. A leading question is a question that suggests what answer is desired or leads to the desired answer.
So instead of asking someone broadly about their experience of something, a leading question would perhaps ask something along the lines of, ‘how much did you like that experience?’ or ‘what was wrong with that?’ (and there are many more insidious examples available).
We all ask leading questions from time to time, the key is to know when you’ve done it and think through what the impacts are.
If you’re doing research poorly, it will be filled with leading questions and you won’t be getting anywhere near the truth; you could end up making major decisions based on flawed data.
Asking good questions is at the heart of good UX research, and not doing it properly causes all sorts of problems.
What I want to outline (and hopefully remind people of) is a famous psychology experiment that took place in 1974 by Loftus and Palmer, as it holds the key as to why you should care about questioning technique.
What they did — experiment one
Having been concerned with the influence of misleading information in relation to eyewitness testimony, Psychologist Elizabeth Loftus set out to create an experiment that tested how subsequent information can affect an eyewitness’s account of an event.
The original memory can be influenced by doing something as simple as switching a word.
They showed participants video clips of car accidents, in random order. After each one, they were given a questionnaire which asked them a number of things, including one critical question:
“About how fast were the cars going when they smashed each other?”
One group of participants was given this question and the other four groups were given either the verb ‘collided’, ‘bumped’, ‘hit’ or ‘contacted’ in the place of the word ‘smashed’. They just replaced one verb.
Here are the results:
- Smashed — 40.8 mph average speed estimated
- Collided — 39.3mph
- Bumped — 38.1mph
- Hit — 34mph
- Contacted — 31.8mph
That seems incredibly powerful to me. Changing the word from ‘smashed’ to ‘contacted’ changed the average speed estimate from 40.8mph to 31.8mph.
That’s how powerful leading questions can be, and how powerful in general the words you use are.
What they did — experiment two
150 students were shown a short film of a car driving through the countryside followed by four seconds of a multiple traffic accident.
50 students were asked, ‘how fast were the car going when they hit each other?’, another 50, ‘how fast were the car going when they smashed each other?’, and the remaining 50 participants were not asked a question at all (the control group).
One week later they answered ten questions, one of which was a critical one randomly placed in the list:
“Did you see any broken glass? Yes or no?”
There was no broken glass on the original film.
Where the word used in the original questioning was ‘hit’, 7/50 thought there was broken glass as a result of the accident. When the word used was ‘smashed,’ 16/50 thought there was broken glass. In the control group, 6/50 thought there was broken glass.
Conclusion
The findings indicate that memory for an event that has been witnessed is highly flexible. If someone is exposed to new information during the interval between witnessing the event and recalling it, this new information may have marked effects on what they recall.
Think about that for a minute. Think about how it applies to your research and what you’re asking from people. Think about how important the words you use are when you form questions and how much power a leading question has.
Think about how innocuous it feels when reporting a finding from research to a group of people, when the way the questions are worded are empirically proven to have a massive effect on that ‘finding’. It’s incredibly important to get the questions right.
I often wonder and marvel at journalists when they ask questions. They often have several parts, go on for far too long, and are full of leading parts. It’s because they have a deadline and an agenda; they want the answer to fit into their narrative.
Don’t be a journalist.
If you want to know a bit more about the Loftus and Palmer (1974) experiment, have a look here: https://www.simplypsychology.org/loftus-palmer.html
If you liked this and you want to get it pushed directly into your brain (email), then feel free to subscribe.
If you don’t want to do that, but love my stuff an incredible amount, you can join Medium and get access to everything I’ve written, plus all other Medium content. It doesn’t cost a huge amount and it’s definitely worthwhile.
