If you are reading this, then you’ve probably just been given the job of improving a website, app, product, or maybe even a real-world service. Like with most things that are new, you are not sure where to start, and the amount of information Google is throwing back at your search queries is leaving you a little overwhelmed.
What I hope to achieve in this article is give you the bare basics of user testing so you can move forward and gain a little experience of your own, get some positive results and then progress deeper into the world of UX.
The first thing you need to do is:
Define the problem. One way of looking at this first point is by asking yourself:-
- What currently isn’t working at the level we’d like with our product? Or
- What is the most complained about aspect of our product based on customer feedback?
- What is the first touch point our clients reach and what is our desired action for them?
The answer to this question will be the focus of your first user test(s). If you have multiple answers which you find hard to prioritize, then you should flip a coin and focus on just one first. DO NOT focus on multiple problems with your early user testing because you’ll most likely not answer any of your questions if you do.
The reason why we identify a single problem and focus on it to the exclusion of everything else is because:-
- The user tests will be more targeted, and the chances of you getting a clear, actionable result will be higher.
- The tests will be shorter, thus, participants are more likely to remain engaged and provide valuable insights.
- If you are very specific about what you are testing, then you can ideally run the same test multiple times at regular intervals to ensure your product iterations are continuing to move you in the right direction.
“What if I can’t identify the problem?”
Firstly, this is not a silly question; many a smart UX professional has asked this, but it is what they do next that’s important. They’ll generally do some exploratory or discovery research (user testing will sometimes be a part of this process,) to explore how customers are using their product and, in the process, identify problem areas. Exploratory research is beyond the scope of this post but if you need a kick start try this article that aims to strengthen the discovery phase.
Who are you targeting?
Once you’ve defined the problem you are going to focus on, the next thing to consider is who should be participating in your studies. If you are a social media tool targeting university students, then you don’t require middle-aged professionals giving you feedback on your product.
Why is that?
Because context is key. People from different locations, and in different stages of their life usually view the same thing very differently. Imagine if you optimized your tool for middle-aged men and then marketed it to twenty-something women. What would happen?
Nothing positive, that’s what.
If your product is already in the wild and has some users, then you’re in luck. While it’s not guaranteed, there’s a strong chance that these people are your target market. So, recruit them to participate in your study. If your product isn’t live, or you cannot recruit real users then don’t be stingy on your recruitment. You may need to pay an incentive of $20 – $100 per participant, depending on your defined demographics, but you must ensure the right people are flowing through your tests.
Additionally, is money the best thing you can reward them with? If you are recruiting current clients, then you’ll most likely be able to offer your product or service at a discounted rate, or even free, which can be more cost-effective. For example, a $50 t-shirt is worth $50 to the participant but might only cost you $25. In this case, it’s a no-brainer so don’t offer them $50, instead offer them the $50 t-shirt.
Knowing who you are targeting also helps a lot when writing tasks, questions and generally structuring your user tests.
Are you looking for Quantitative or Qualitative Data? What do you want to learn?
Another way of saying this is, do you want people’s opinions (qualitative) or do you want to see the results of their actions (quantitative)? This also relates somewhat to the previous point about recruitment.
Qualitative, or Qual, testing often only requires around 5 participants to run through a usability test. Why only 5? Read this article about Why You Only Need To Test With 5 Users. Not everyone agrees with its conclusions, but many do. These tests often include video and audio of the participant ‘thinking aloud’ and telling you about their experience.
Quantitative, or Quant, on the other hand, requires a greater number of participants. The resulting data from these tests often comes from tasks that the participants were asked to complete, where you are interested in things like task success rates, page count, path analysis and time on task. Generally speaking you want to recruit over 50 participants for Quant testing.
You don’t necessarily need to choose one or the other, Quant or Qual, because tools like Loop11 – user testing let you run user tests collecting both forms of data. However, focussing on one at a time helps you to keep your testing centred.
Qual testing is often associated with the discovery phase of testing which we identified earlier, whereas Quant testing is excellent at validating a product’s usability.
What are your key tasks and questions?
It goes without saying, but just to be clear, your tasks and questions should aim to provide answers surrounding the focus of your testing. Answering where, why and how people are finding difficulties is vital, and this should lead to clues regarding how to improve your users’ experience.
To give a point of reference, Loop11 users will create three tasks for their usability studies on average, and each task is usually followed with one or two questions aimed at gaining further insight once the participant has moved on from the prior task.
Examples of a task may be to ask the participant to locate specific information, add an item to a shopping cart or to fill out a form. It’s relatively common to follow a task with a rating scale question that looks to identify the perceived ease with which the participant associated completing the earlier task. A second question might be open-ended, asking why the participant gave the rating.
Ideally your tasks should all have a flow and/or be related to each other so that the participant’s journey represents a “real” experience as closely as possible. If you find yourself writing tasks which are completely unrelated to one another, then perhaps you’ve fallen foul of our first recommendation and have not split your problems into separate user tests.
Frequently, UX professionals will add a few questions at the end of the survey asking anything from Net Promoter Score (NPS) to System Usability Scale (SUS) to focused demographic questions. No answers are right or wrong here, but only ask questions where you know how you’re going to use the resulting data. Don’t waste participants’ time.
Look at your test from a participant’s perspective
Once you’ve fleshed out the flow of your test, your tasks and questions are in place, run through the study as though you were a participant and time how long it takes. Even better, get someone else to do it as you observe, ideally someone who is not intimate with your product. This way you’ll be able to get an unbiased idea of how long your study is and the sort of time commitment you are asking of participants.
The longer your study becomes, the bigger your incentive/payment will need to be, or the higher your dropout rate will be as participants get frustrated with the length of the study.
There are no defined rules here but try to think of it in relation to your target customer. If you are looking at C level executives, then they’re unlikely to be motivated by an offer of $15 for 15-minutes. In this instance, shorten your study or increase your incentive.
How are you analyzing your results? Do you have benchmarks?
Thinking about your results before you go out and get them is one of the most important things you can do. It helps you craft your study, set better tasks and ask better questions.
Another important question that needs answering is – do you already have benchmarks or are you running a study to create them?
Benchmarks are the key data points that you will be measuring against in future UX studies. If you don’t have any, then your first study or two should be about identifying the relevant benchmarks. Even if you are committed to a new product design, you should really benchmark your old product, so you can have concrete data to measure your new design against.
Here is an article detailing how Médecins Sans Frontières (MSF) had great success benchmarking a new website against their old one.
User testing doesn’t need to be expensive and doesn’t have to take up a lot of your time. However, it needs to be done in a thoughtful, focused manner. It’s also something that should be done at all phases of your project. As long as your company is serving customers, then you should be speaking to and observing them to see how you can improve your offering.
So, now that you’ve dipped your toe into the waters of user experience testing – go forth and test! There is a world of insights waiting for you just around the corner.
Post written by Ben Newton from Loop 11.
