Will ChatGPT take over the role of UX designers?

Case study for design & AI — Is there a gap for humans?

Photo by Fabio on Unsplash

With the recent hype of ChatGPT, many assume it could take over many roles within the design field, which is a worrying situation in the current job market. In this case study, I investigate to find the gap and how we, as UX Designers, could fill the gap between AI and the job market and, more importantly, how we can grow with the trend of machine learning.

Purpose:

• The first part of my project (#DUXW) was that I completed the UX content in 15 days. The focus of this project was targeting my problem-solving and content-writing skills.
• The second part was that I fed the prompts to create different results on UX content. The focus was figuring out the quality of answers from the AI.
• Within this case study, I compare the result of my work as a human mind and solutions from machine learning, aiming to find the gap and how we, as UX Designers, could fill the gap between AI and the job market, and, how we can grow with the trend of machine learning.

Problem Statment:

To kick off this case study, I focus on the problem for the designers:

“I am a UX designer who use my human brain to solve user problem and not sure if ChatGPT can solve them more accurately and better. As a result, I feel threatened.”

The reason is that I assumed — “ChatGPT can generate better content than the human mind.” So, I interacted with the users to validate my hypothesis which will be presented in the following part.

Deliverables 1: Quantitative survey with A/B testing

It aimed to validate my hypothesis about “ChatGPT can generate accurate content that users cannot differentiate the content easily.”

I practiced gamification in creating this survey, which enhanced the participation experience, fostering higher engagement. This is a very fascinating part of the case study, as this quiz was shared with different groups with/ without design knowledge, imitating the users of an app.

Here to the quiz.

Participants were asked to choose the content generated by a human in 10 questions as an A/B test.

Findings and implications—

A total of 101 responses were collected in 6 days, with positive feedback and high user engagement.

This distribution bar chart has suggested that it might not be as easy as my hypothesis.

• 55.4% of the participants scored between 6 and 8, and 29.1% has less than 5. It implied that not all users can tell AI-generated content as I assumed.
• Half of the participants answered hard(33) and very hard(17) within the scale, “how hard is it to differentiate AI- from human-generated content?”
• The frequently missed question is this one with less than a 50% correct rate. This particular solution from ChatGPT has provided more details and context in the body text to users than my solution.

Even though ChatGPT has exceeded the character limit from the prompt, which is unusual, the participants didn’t tell apart.

With the open-ended question “What is the clue(s) that gave away content that AI generates?”, the results are grouped and analyzed with an Affinity Map.

Clusters are created and named for similar comments.

Some explained that they try to work out what makes it human content, e.g. grammatical errors and colloquial sayings; but uncertainty and difficulty are also expressed during the task. In summary, participants have considered the following to Identify AI-generated content:

• level of human touch: “(human’s content)…the real kindness around the use in the process (extra info to help the USER in his behavioral tasks).” Sometimes imperfections happen with humans, and “some slight grammar mistakes”.
• emotional appeal: “it (AI-generated content) tends to be very true to the prompt, especially in the first iteration, and devoid of emotion.”
• personality: “wordy and cooler in tone”
• writing style: “plainer”
• the formality of language: “humans use street or colloquial sayings, AI is more formal and businessy.”
• level of details: “(human are more)… obvious in repetitions and detailing”

Takeaways:

Now, we understood better the users with proven data. If I have more resources for this case study, I would gather a larger pool of data and work with a data analyst for a better visualization, which could be more insightful in the future.

My hypothesis is correct, that ChatGPT can generate accurate content. Alert! We should also recognize that the prompt provided by the DUXW is very well-written and it gives a clear context and accurate direction for ChatGPT for what needs to be done. It is a major reason for the exceptionally good content.

Still, users, in certain situations, can point out if texts are generated by a language mode, which is still limited to mimicking humanity — creative thinking, tone, humor, sarcasm, etc.

The question is when AI tools can overcome these issues — at first, we should look into the technology at the moment.

Current AI Tools & Trends:

AI-generated output has played an important role in the Tech industry.

Norman Nielsen, VP of Growth Omio, talked about “Limitations and use cases of AI tools” for product teams at the E-commerce Expo Berlin 2023. That was very insightful to the industry, as AI tools can enhance our productivity and capability. I have grouped some tools below:

• Otter.ai — real-time transcription meeting notes that are shareable, searchable, accessible, and secure.
• Verbally — visible agenda, speaker timer, and optional timers per topic support punctuality when on a tight schedule.
• Bearly — Reading, writing, and content creation on one AI tool.
• Jasper — generative AI platform for businesses to create marketing content
• HyperWrite — Write faster from idea to final draft.

During the research, I came across many resources by different designers writing about AI; an article “design systems in the time of AI”, by Brad Frost pointed out that AI tools are not a replacement for designers. We still need to practice collaboration and communication skills with stakeholders. AI allows us to focus on why we create, so it is necessary to view it from the designers’ perspectives.

Deliverables 2: Qualitative interviews with 5 participants

To deepen the understanding of how designers use ChatGPT at work, I have conducted 5 qualitative interviews, with the following sets of questions:

• Do you use ChatGPT in your daily work? And how?
• Can you describe your prompt to ChatGPT for certain tasks? (please cite 1–2 example[s])
• How do you think AI technology will evolve in the future and impact UX?
• In your opinion, what are the qualities of good design that AI models may not be able to replicate?
• How do you envision the role of UX designers evolving in the context of AI-powered language models like ChatGPT?
• Are there any current restrictions for your field of work with AI?

Findings and implications —

I applied the thematic analysis to cluster data into themes that represent user needs, motivations, and behaviors.

Thematic analysis to analyze the user needs, motivations, and behaviors in how designers use ChatGPT

The goals of designers are always trying to improve the efficiency and quality of the solution.

The designers have adapted and used ChatGPT for the following tasks:

• Preparation — Sort ideas, data analytics for user research, and build a framework.
• Generation — Quickly generate articles, summaries, and outlines.
• Personalization — Personalize content for the target users.
• Proofreading— identify errors and inconsistencies in content.
• Optimization — Optimize content for search engines by generating meta descriptions and keyword-rich titles.
• Translation — Translate content into different languages.

To sum up, the designers can increase productivity, thus spending time on other tasks, e.g. training junior staff or designing a better workflow. We can focus on improving user experience and establishing inclusive designs.

Risks!

• Overreliance on data — bias and unfairness in the given result; who is training these language models?
• Ethical & legal — copyright, privacy, and ethical considerations.
• Lack of trust — due to lack of transparency in implication generation.
• Poor integration with human workflows — human intuition and empathy is lacking in setting strategies.

Opportunities!

As designers, we shall learn about prompt engineering — more tools will soon be available to generate new forms of content, e.g. interactive experiences, animations, and virtual reality experiences.

• Galileo AI — Generative AI for user interface design, empowering you to design beyond imagination with speed.
• ImaginGoogle — text-to-image diffusion model with photorealism and language understanding.

Takeaways:

Back to the hypothesis of “ChatGPT can generate better content than the human mind”, as designers, we shall accept that ChatGPT is a great tool for work, which could improve our workflow in many different areas. Now, it is important to identify what tasks can be automated.

A gap between AI tools and what the users need still exists, and humans are needed to take control — we should be the brain behind all actions.

Also, there will be new jobs appearing with controlling and leveraging the power of AI tools. For example, the implementation of data privacy and how to make the user journey of AI technologies more visible to users, etc.

Future of AI

One key difference is that —

AI does not have the intention to solve humans’ problems. Humans still need to initiate problem-solving and be strategic.

Final Thoughts and Learning:

As designers riding on the wave of AI technologies, we shall be open to learning and understanding the current trend and the potential applications of AI. After having the right tools, we can improve our productivity at our job and focus on creating a better user experience on-brand. We shall continue mastering our technical skills in user-centered design, user experience, and storytelling and our soft skills in adaptability, critical thinking, creativity, and communication.

To summarize this article, we should understand what the true competitor is. We shall not compete with other designers, nor the AI, but ourselves — how can we outgrow ourselves today for tomorrow?

References:

1. McCue, T. J. (2023, January 19). ChatGPT Success Completely Depends On Your Prompt. Forbes. https://www.forbes.com/sites/tjmccue/2023/01/19/chatgpt-success-completely-depends-on-your-prompt/
2. Schmid, S. (2022, December 8). ChatGPT: How To Write The Perfect Prompts. Neuroflash. https://neuroflash.com/chatgpt-how-to-write-the-perfect-prompts/
3. Babich, N. (2023, February 12). Using ChatGPT for UX Writing. UX Planet. https://uxplanet.org/using-chatgpt-for-ux-writing-9abdcfb29d95
4. Babich, N. (2023, February 24). 4 Biggest Issues with Modern AI Tools. UX Planet. https://medium.com/ux-planet/4-biggest-issues-with-modern-ai-tools-7467143fe8c6
5. Growth.Design. (n.d.). How Bing uses AI to Help You Find Your Perfect Image: Ep. 1. Growth.Design. https://growth.design/case-studies/new-bing-ai-ep1
6. Anany, A. (2023, January 20). Google’s Sparrow Will Kill ChatGPT. It is Microsoft Teams vs. Slack All Over Again. Entrepreneurial. https://entreprenal.com/googles-sparrow-will-kill-chatgpt-it-is-microsoft-teams-vs-slack-all-over-again-da8c5a69c58f