Step-by-step usability testing for a language app

My team and I conducted usability testing for Mondly, a language-learning app that aims to help users learn a new language through interactive lessons and exercises. We tested the Mondly general app and Mondly AR.

Mondly home page

Here are the steps we followed;

Context Report

We first completed a context report, in which we researched the company extensively. In the report,

• we described the products: general Mondly and Mondly AR; all the features they have
• we described the target audience, including what prior knowledge they are expected to have in order to use the Mondly products
• we described the intended outcomes and the explicit and implicit learning objectives by going deeper into what Mondly was intending the accomplish
• we completed a preliminary evaluation in which we evaluated; the appropriateness of Mondly’s materials and tasks, ease of use of the interface, the language and visual design, and motivational design- how satisfying the educational experience is
• we lastly wrote recommendations for revision

Tryout Plan

The next step was to design a tryout plan, in which we defined the purpose of the usability study and created the data collection instruments.

Matrix of evaluative questions and instruments of data collection

Based on our study purpose, we aimed to create items that;

1. determine users’ past experiences using a language-learning app

2. determine users’ first impressions of Mondly to understand their perceptions and expectations before the usability testing and establish their level of expertise

3. determine users’ first impressions of Mondly AR to understand their perceptions and expectations before the usability testing and establish their level of expertise

4. determine if Mondly is easy to use

5. determine if Mondly AR is easy to use

6. explore the perspectives of users after using Mondly, and

7. identify the strengths and pain points of the Mondly system

These seven purposes resulted in a background questionnaire, a pre-test questionnaire, user testing, post-test interviews, and a post-interview questionnaire.

We used Rubin & Chisnell (2008) to create these data collection instruments. We referred to Drennan (2003) to validate them.

Pilot Test

All our data collection instruments were designed and validated. The next step was to test them. So we conducted a pilot study to try out the evaluation questions, our instruments of data collection, and the administration procedures. Based on the findings we acquired from the pilot test, we made several changes to the instruments and made other necessary revisions.

The pilot study helped us see the changes we needed to make before the actual study.

After the pilot test, we wrote down a description of our data collection process to be implemented in the real study, and we created a data analysis plan explaining how we are going to analyze the data that we collect from our real users.

Learner Tryout

Finally, it was time to conduct our real study with real users! We recruited users to participate in this study through a screening process; in order to be eligible for this study, our users must be learning a language and using a language app.

We collected both qualitative and quantitative data from the eligible users. We used Qualtrics and Google forms to collect the quantitative data and analyzed it using Excel.

We collected the qualitative data from the interviews and think-aloud protocol. We analyzed the qualitative data using the software Notably (here is an article I wrote about Notably).

We used thematic analysis to analyze the qualitative data. Thematic analysis is analyzing text to find repeating patterns and categorizing them under themes.

Here are the steps we followed while doing the thematic analysis:

Thematic analysis steps

Using Notably in the qualitative data analysis process was really the best decision since it helped us organize our data more efficiently and meaningfully.

We detected the patterns for user satisfaction and pain points while using Mondly general and Mondly AR. The themes helped us write the insights and, in the end, give recommendations for improvement.

We sent the full report to Mondly through email. They have not gotten back to us yet, but hopefully, they will find our findings useful in helping them improve their language app.

Contact me if you have any questions about this usability study! Here is my LinkedIn page.

Thanks for reading!

NOTE: We are not affiliated with Mondly.