Case study: Quora usability testing

Overview
Competition among Q&A Social Network requires companies to make it easier for their users to use mobile applications provided to attract users. Various problems faced by users, such as users having difficulty accessing the application, users feel the application is too complicated to use, and the system does not suit the user’s needs. An application can be said to be successful if the application is able to meet the needs of the user by carrying out the functions of accuracy, reliability, and ease of use. To solve problems that are often felt by users, it is necessary to do usability testing. In this case, we choose Quora as an object in this test. This testing process includes determining the objectives and target users, recruiting users, creating a test script, conducting testing, and finally creating a test result report. In this study, the authors used 5 users from various different backgrounds.
About Quora
Quora is a Q&A Social Media Network that empowers people to share and develop world knowledge. People come to Quora to ask questions on a variety of subjects, read high-quality knowledge that is personalized and relevant to them, and share their knowledge with others. Quora is a place to share knowledge and understand the world better. The main functions of Quora are:
a. Ask questions and get useful answers
b. Learn from credible people with first-hand knowledge
c. Answer questions and share knowledge with the world
Goals and Objective

- To identify and discover the problem of Quora by users/participants.
- To observe how the user’s behavior and interaction with Quora.
- To discover what user’s experienced as opportunities for improving the design in the future.
Usability Metrics: Success Rate
Based on nngroup.com, the success rate is the percentage of tasks that users complete correctly. Simplified success rates are best used to provide a general picture of how your site supports users and how much improvement is needed to make the site really work. Why Success Rate? They are easy to collect and a very telling statistic. After all, if users can’t accomplish their target task, all else is irrelevant.
Methodology: Remote Testing
Since this article was written, the world is experiencing a Covid-19 virus pandemic. So, there are limits in interacting with other people, the same goes for this activity. Furthermore, Remote usability testing allows you to get customer insights when travel budgets are small, timeframes are tight, or test participants are hard to find.
Usability Testing Process

Define Goals and Target Users
The first thing that must be done is to define the purpose of testing and determine the target users who will perform the test.
Recruit participants
The second step is recruiting participants. Users who are recruited must be users who are in accordance with predetermined targets.
Make a test script
In creating a test script, it can be divided into 3 parts: Pre-test interview, Tasks & Scenarios, and Post-test Interview. The pre-test interview can serve as an intermezzo and help participants to be more relaxed and comfortable during the test. Furthermore, Tasks & Scenario aims to provide a hypothetical scenario that will direct the user to complete a certain task. Last, Post-test Interview aims to find out the user’s impression after using the product, what difficulties the user is experiencing, what tasks are easiest for the user, etc.
Conduct the remote test
Make sure all the equipment needed when testing is available and in good condition. Record every testing process from start to finish. When starting the test tell the user that the application is going to be tested and not them so that there are no wrong answers. Ask users to explain each step they took and why they did it. Ask them to be honest and open about their experiences.
Analyze and Create a test report
After testing is complete it’s time to create a report, note all patterns and user interactions. Combine all records into a test result report.
Roles
Moderator: Lead a usability test session with participants. Moderator will read and instruct the scenario and task scripts to the participants. Moderator should be neutral and never suggest anything to the participant.
Note Taker/Observer: Pay attention and monitoring to every comment, response, and issue the participant faced during the test session.
Time Keeper: Pay attention to the length of time participants work on each task and test session.
Participant/User
- TA (17) Final year high school student. Almost every hour of the day using social media.
- JE (20) Final year college student. Use social media on average more than 5 hours a day.
- HD (21) Final year Full-Time Student, Having an interest in IT. On average spend 60–80 minutes on social media in a day. Passive user.
- MI (21) Fresh Graduate, a Freelance worker in a technology company (Work From Home). Spent 1⁄3 day (8 hours) using social media.
- SY (23) Worker in Jakarta Smart City. A user who rarely uses social media.
Pre-Test Interview Questions
Pre-Test Interview Question is given before the scenario and task session started. Here are some questions that we asked:
- Please introduce yourself (Name, Age, Gender, and their interest)
- What is your current job?
- How long have you worked in this field?
- What are you busy with lately?
- How often do you use social media?
- What device do you use to access social media?
- In a day, how long are you surfing on social media?
- Have you ever heard of Quora?
- Have you ever used it?
- How long have you been using it?
- What do you think about the application?
- What problems did you experience when using this application?
Scenario and Task
- Now, you’re looking at a question, then you come across an interesting question to answer. How do you answer it?
- When you browse the list of questions, you see a question that is inappropriate and irrelevant to you. How do you skip a question and select support for the question?
- You’re just starting out investing in stocks. But you are confused about what stocks are suitable for beginners. Then you decide to ask questions on Quora. Show us how you can ask a question on Quora.
- You are an article writer. To fill your spare time, you would like to share a link that you have written on Quora. How do you share a link in Quora?
- Imagine you are an enthusiastic cyclist. To add insight and want to open a discussion with other cyclists, you decide to create a room whose contents will discuss the world of bicycles. Show us how you create a new Room.
- You are someone who loves stocks and investing. Then you decide to join the Stock Traders Community discussion room to increase your knowledge about stocks. Show us how you search for and follow a Room on Quora.
- You are someone who is just learning about stocks. There is a trader named William Mamudi who knows about stocks and wants to share his knowledge with you. Finally, you decide to follow him on Quora. Show us how you follow and view a person’s profile and on Quora.
- You’re reading a post on Quora, and then you come across an answer that is so smart and interesting that you decide to support the answer. Show us how you do it.
- You’re spending some time reading some of the posts on Quora. Then you find an answer that makes you confused, so you decide to ask about it in the comments column. Show us how you do it.
Post-Test Interview Question
Post-Test Interview Question is given when the scenario and task session is finished. Here are some questions that we asked:
- How do you feel while doing the task given?
- What difficulties did you find when using this application?
- What do you like about this platform?
- What do you dislike about this platform?
- As far as you know, is there a website/application similar to
this website? - In your opinion, who are the suitable people to use the website/application this?
Conduct The Remote Test
The test session was held from 21–23 October 2020. Regarding the current COVID-19 Pandemic, the test session was held using Google Meet as the medium. 10 minutes before the test session begins, participants are given a Google Meet link via Whatsapp. The time allocation required for the test session is approximately 20–30 minutes. The test session is divided into three sessions, namely: pre-test interview, scenario & task, and finally post-test interview. After the post-test interview is over, we do a photo session along with a screenshot of the google meet room and after that, the participants can leave the room (google meet).
The Result

In total, we observed 45 attempts to perform the tasks. Of those attempts, 29 were successful, 6 were partially successful and 10 were failures. For this particular site, we gave each partial success half a point.
So, the success rate was (29+(6*0.5))/45= 71% of Succes Rate.
What did We find?
Based on the test, here are some pain points that the user experienced:


“I’m confused, the button and icon are too small”

“I’m not familiar with the icon, its not recognized”
“When doing a task, you have to make questions first, it’s a bit troublesome and you can’t go straight. Requires more time.“

“For the first time a little bit confused.”

“The font color in each category is gray, not too contrast.“
Conclusion and Recommendation
Based on what we did, the result is 3 of 9 scenarios and given the task successfully carried out by all participants. Meanwhile, 4 out of 5 users have difficulty using Quora.
The difficulties they experience include buttons, icon fonts that are too small and less attractive so that they are not easily identified by users. Then the term “downvote” menu that confuses new users, the button “find Room” is not in accordance with user expectations when pressing the button, and there are several actions that can only be done after clicking the question, for example when answering and supporting down the question, which is considered quite a waste of user time.
The suggestions for implementing the recommendations that we submit are based on the problems and difficulties experienced by users during usability testing, namely redesigning buttons, icons, and fonts to fit the design principle, given different signifiers and clearer to each other.
For example, buttons, icons, and fonts are enlarged and their colors can be made striking to make them easier to identify by the user. In addition, the use of grammar that can be simplified and familiar to the user, as well as adding a “show less” menu to an answer so that users can scroll pages shorter so as to save the user time.
If you want to see the full documentation (such as pre-test and post-test question answer, and the detailed user’s finding), click here
Thank you for reading my article, any feedback, question, and opportunity email me at ranggaray.rr@gmail.com :)