Enhancing Research Synthesis with AI: An Experiential Report
Co-creation credit with Rachel Lawson
Once upon a deadline, we found ourselves submerged in the labyrinth of research recordings and extensive notes. Ready to embark on synthesis, we were bracing ourselves for a journey that was known to be both enlightening and dreadfully time-consuming. Eager to escape the relentless waves of extracting, grouping and filtering, we turned our sights to the golden shores, with hope twinkling in our eyes like stars on a clear night. We envisioned a saviour, a beacon of light, named AI. It was going to be all plain sailing from here.
Ok, back to reality. What really happened? After becoming surrounded by digital post-its and snippets of text generated by ChatGPT, we quickly realised that the notion of AI as an autonomous entity capable of completing large intricate tasks to a high quality while you settle down for a cup of tea is largely a myth. John Maeda’s metaphor of a pilot and a copilot1 aptly embodies the dynamic and collaborative relationship between the user and AI. In this partnership, you — the user — play the pilot, steering and shaping the course. Meanwhile, the AI, your co-pilot, aids in expanding your journey through the realms of knowledge and inquiry. It’s not about handing over full control; rather, it’s about harmonising the abilities of both entities, allowing you to delve deeper and farther into topics, even those with which you might have a good working knowledge but not the depth of expertise of someone that does it day in, day out. But a word of caution here — straying into the complex subject matter with little existing knowledge shifts the dynamic between pilot and copilot. You become the dependent copilot. But done carefully, this symbiotic interaction promotes speed and a richer, more nuanced exploration of subjects, where you hold the reins, managing the overarching narrative and assessing the merits of the copilot’s contributions.
Our test
The research we conducted was a small-scale qualitative exploration involving four participants, all senior stakeholders at the same company. We set out to understand their roles, responsibilities, aspirations for the project and any potential pain points. The outcome of this work forms an important part of the discovery phase in our rebranding process. We picked this situation for our test because of the limited number of participants and the straightforward nature of the research questions. We have run this process without AI many times before, over short durations and with great success — all adding up to a safe space to try a new technique.
In our pursuit to boost efficiency, our approach was to blend the power of AI with traditional methods, attempting to ensure a balanced, reliable outcome, especially given the live nature of our project. We instilled a significant degree of trust in the AI but also allocated time to meticulously review the results using conventional human-only methods, confirming the accuracy and reliability of the output. That was a smart move, as there was a great deal of editing and cross-reference to be done.
Step-by-Step
To conduct our research, we progressed through five distinct stages. AI was integrated into the process after we secured participant session recordings.
1. Setting the Stage
We laid the groundwork before diving into the research, focusing on clear research goals and formulating the pertinent questions.
2. During the Sessions
Each interview was recorded, with a dedicated individual taking notes of key points and observations — not trying to transcribe the conversation.
3. Post-Session Processing
We utilised Otter to transcribe the interviews, ensuring every detail was captured.
4. Diving into Initial Insights
We began by inputting the transcriptions into ChatGPT to extract preliminary insights. Initially, we used broad prompts that aimed to encompass the main themes of the interview. This overwhelms the AI. The results were overly generic. But it was worth a try. When this didn’t yield satisfactory results, we pivoted. We broke down our enquiries, introducing questions one by one. If the answers weren’t hitting the mark, we refined and rephrased our questions to delve deeper into our areas of interest. As we accumulated insights from ChatGPT, we paused every so often and asked for a bulleted summary of all the findings. This process of summarising, in most cases, worked to refine the response, distilling it all down into a concise point or two.
5. Deep Dive into Insights
Armed with all the notes, Otter summaries, insights from ChatGPT, and the original recordings, we forged ahead, synthesising and distilling the second-wave insights, painting a holistic picture of our findings. Being able to place all these elements together side by side allowed us to validate horizontally.
AI piloting
Throughout the experiment, we gathered valuable insights on how to effectively pilot the process, focusing on ways to support and nurture the AI as a copilot.
- When comparing research participants, strive for as much consistency as possible. Ensure that each stakeholder is asked the same set of questions. This uniformity aids in later synthesis using AI.
- Break up the enquiries for the AI into smaller, bite-sized pieces. Progress slowly, rather than feeding it a complete transcript with a broad request like ‘provide insights from this document’. The more granular your questions, the better. The process should feel more like a conversation with an expert researcher than a mechanical, one-size-fits-all approach.
- Expect to have to refine the questions you pose to the AI. Sometimes, ChatGPT gets stuck and needs a broader question to get going again.
- Be aware that the AI’s response style may vary between sessions. Invest time in warming up the AI with simpler questions to achieve high-quality answers before moving to more complex ones. Patience is key in this iterative process.
- Take the time to read the OpenAI Six strategies for getting better results. We found this late into our test. Having it upfront would have saved some time.
Conclusion
AI is incredibly helpful, but it’s not going to do all the heavy lifting on its own. Think of AI as a newbie on your team; they’re great with a bit of guidance, but they might not always get things right when left to their own devices. The key to successfully teaming up with AI is to find that sweet spot where your input and its capabilities complement each other effectively.
Prompts
Round 1
A failed attempt to pass a list of questions and a transcript in the hope that ChatGPT would extract all insights in one hit. The insights were generic and very low value.
Step 1. Prompt
As an expert researcher, I would like you to review an interview with the participant [Participant]. [Participant] is the head of operations, HR, and recruitment at a company called [Company Name]. We have nine research questions we would like you to focus on producing insights for.Research questions
1. What does the participant think [Company Name] is and does?
2. What does the participant think is the reason [Company Name] exists and the vision for [Company Name]?
3. How does the participant think [Company Name] fits in the market they operate in?
4. How does the participant think about [Company Name]’s competitors?
5. What are the short and long term goals for [Company Name]?
6. What facts can the participant share about the current [Company Name] hiring experience?
7. How does the participant think about the [Company Name] hiring experience
8. What facts can the participant share about the current [Company Name] New business experience?
9. How does the participant think about the [Company Name] New business experience?
Round 2
Similar content and assets, but delivered to ChatGPT slowly, one question at a time. Resulting in better insights. But still, with a lot of repetition, that needed to be filtered out by hand.
Step2. Prompt
As an expert researcher, I would like your help answering research questions for the attached interview. The interview participant is [[Participant]]. [[Participant]] is the [head of operations, HR, recruitment] at a company called [Company Name]. Could you please try to include direct quotes from the participant when providing an insight? Could you please try to include direct quotes from the participant when providing an insight?In a moment, I will ask you specific questions about the participant’s views. Do you have any questions before we start?
Follow-up questions that were delivered to the AI one at a time.
Q1. Prompt
How does the participant describe [Company Name]?Q2. Prompt
What does the participant think is the reason [Company Name] exists and the vision for [Company Name]?Q3. Prompt
What are the participant views on how [Company Name] is similar and unique compared to its competitors?Q4. Prompt
How does the participant describe [Company Name]’s ambitions for the future?Q5. Prompt
What facts can the participant share about the current [Company Name] hiring experience today?Q6. Prompt
What ambitions does the participant have [Company Name]’s hiring experience?Q7. Prompt
What facts can the participant share about the current [Company Name] new business experience?Q8. Prompt
What ambitions does the participant have [Company Name]’s new business experience?
Tools used
- Zoom, calling recording and transcription
- Otter, transcription and summary creation — second transcription created to sidestep issues in the first version
- ChatGPT, analysis and insight creations
- FigJam, sorting insights
References
John Maeda, Design + Ai report. https://designintech.report/sxsw2023/
A project conducted at Yaya Digital Consultancy, www.yaya.co
