6 things that I learned from conducting a survey
The story about the power of planning, the need for testing, why shortcuts won’t work, and when to start thinking about how to analyze the data.
How do designers draw their wireframes? I was curious so I asked. Read the “Designers and their wireframe habits” post to find out how they do it. In this post, I will share the 6 things I learned or experienced while running this survey so that you would not fall into the same pitfalls.
However, I would like to start with why I chose to run a survey. Surveys are inexpensive and a quick way to get quantitative and qualitative data remotely. My goal was to find out background information about the topic from as many people as possible and I would not be able to do that via interviews. Additionally, the bigger the sample of respondents, the better the odds are to get more diverse data and start noticing patterns.
Now that you know why I decided to go with a survey, let’s move to what I’ve learned or experienced while doing it, so your survey will go smoother than mine:
1. Prepare a plan
Prepare a good plan and write it down even if the survey seems simple and straightforward. Always make a plan. Why do I stress this? Well, in my case, I had a plan — in my head — but not well-defined on paper. I usually write my research plans at work, but for this project, I took a shortcut. The outcome was that I forgot to ask a simple question (Do you own an iPad?) and while some respondents covered it in their replies I couldn’t use those for quantitative analysis. Another aspect was analysis. Since I didn’t structure my survey well in Google Forms, I had to count some of the answers manually.
What should I have done?
- Write research questions. Those are questions that you ask yourself about the problem, general big questions, etc. Eg. What do users prefer in the tool they use?
- Write down the structure of a survey to use as a map. Divide your survey into the main aspects, themes of questions, etc. Mark where you would use quantitative and qualitative questions.
- Write questions that you want to ask. Build on top of the map and write all the questions that you find important or relevant. Just write them all. Don’t worry-you can always delete some of them…in fact, you may find yourself doing that… Why? Less is more, we don’t want people to drop off.
- Write down how you want to use the data. Based on what you decided in the previous points, think about how you can use the data you will get. Here you have to take into consideration the type of app you will use to conduct a survey. Apps have limitations, especially the free ones, so you have to find out which one would work the best for you and your survey. This step is important as it will help you to find mistakes in the logic of your survey. I didn’t do that so I had to analyze all the answers, one by one, just to be able to explore interesting correlations between the data and how it relates to the given answers. For me personally, I noticed that when I saw some data, I wanted to explore it further from different angles.
2. Consider limitations
Consider all the limitations and trade-offs of the tool. You may have a limited budget, complicated questions, or are planning to get millions of replies. Check the limitations of the tools and find the tool that works best for your survey and audience. I chose the free tool, which didn’t limit the number of participants (yay). It had some great “hidden” features, for instance, adding a question about the location of the participant. Unfortunately, I didn’t know how to do it so I skipped that one (regret). However, I found out that there is an easy way to do it after I finished my survey (hooray). If you want to try it, copy the list of countries and paste it into a dropdown-type question (easy-peasy).
Secondly, I wanted to ask people for examples of their wireframes, however, uploading to the survey was possible only from google drive, which would remove the anonymity from responses. I assumed that would make people drop off, even if the question was optional, so I decided not to risk that.
3. Test your survey
Test with friends or family to check if everything works and the survey questions don’t repeat. If you can test it with your target group, do it too. I ran the survey that helped me come up with better answers for multiple selection questions, which I then tweaked after around 30 responses to accelerate the process of taking the survey.
4. All questions required
Make all the questions that you wish to have an answer required. Why bother adding a question if you don’t find it as important? Not having all the answers will just make analyzing the data less consistent. Unless it is a sensitive question, such as a follow-up or e-mail request, these are best if kept optional as long as the answers aren’t crucial (it’s a matter of your priorities). My priority was to get as many answers as possible.
5. Helpful tools
Miro and Google sheets are your best friends. The survey is done…I have all the great answers…now it’s time to analyze and synthesize them. For quantitative answers, with graphs and calculations, I used Google Sheets. For open-ended questions, I did affinity mapping in Miro, where I created frames for every question. In those frames, I was pasting answers one by one. While adding answers, I was trying to categorize them into themes. For instance, when I saw a theme with 3 post-ins so I named and color-coded it. Color coding was not the easiest due to the number of categories but I tried to be consistent with coloring the biggest themes that were prominent across all answers.
6. Find your people
I was researching how designers work, so it was easier for me to find them. First, I reached out to the closest communities to me on Slack, however, it was not enough so I started posting my survey in Slack, Facebook, Discord, Spectrum’s UX, and product design communities. When that was not enough, I posted the survey in Figma, Sketch, and Adobe XD communities (the last one was super active so I think it is recognizable in the final results). The last resort was DMing on Linkedin…effective but time-consuming.
It was fun to conduct this survey as I learned something along the way and some of the answers were fun to read, too. I understand that communities are flooded with survey requests (it is even forbidden to post surveys in one of the Reddit communities) so I would like to thank anybody who took part in it and shared their process. This enabled me to learn from their experience as a designer which helped enrich my experience with running and analyzing a quantitative survey.
Here is the spreadsheet with quantitative data.
Here is the survey.
If you want to find out more about how to conduct research and quantitative research, check the IDF courses.
