Should you be running user testing in-person or online?

There are different approaches you can take when running user testing and each comes with its own advantages and disadvantages…

Putting your design, solution or product in front of your target customers is the ideal opportunity to gather insightful feedback on how understandable, usable and engaging it is and allows you to test the critical assumptions you’ve made about what your users think, want and need. The findings from User Testing can help you improve your solution to align it with user needs and expectations, helping you to generate a higher return.

There are a few different approaches you can take to conducting user testing:

  • You can run the testing in-person, sitting down with users and observing them use your solution in a quiet space or dedicated lab facility.
  • You can run the testing remotely, using screen sharing software and a video calling suite such as Skype, FaceTime or Hangouts.
  • Or, a third option would be to utilise an online user testing platform such as UserTesting or WhatUsersDo.

 

How do you decide which option is best? Here, we consider the advantages and disadvantages of all three approaches to help you decide which one is right for your project…

 

In-Person User Testing

Sitting down with users and observing them use your solution in a quiet space or dedicated lab facility.

Advantages

  • Test incomplete designs – Often the material being tested is far from complete (after all, there are huge benefits to testing designs as early as possible). In these situations, a skilled researcher sat with a participant can set the scene, keep the test on track and explain things where needed. About two thirds of the User Testing we do is on designs or prototypes, rather than live, fully working solutions and in-person testing is ideal for this.
  • Better briefed participants – As with all communication, but particularly anything with an element of complexity, it’s much easier to brief someone and be sure they understand when you are face-to-face with them. When it comes to User Testing it is critical that participants are clear about the brief and have the chance to ask questions.
  • Better prepared participants – The value gained from user testing is impacted by how comfortable participants feel. Participants need to be in a frame of mind where they can safely and confidently provide feedback. In-person testing can provide a warm, welcoming and re-assuring environment that enables participants to be fully focused on the task in hand.
  • Recruitment – With online or remote testing you are already pre-selecting a tech-savvy participant by the very nature that they are confident using the very technology needed to undertake the research. In-person testing allows you to invite in and test with less tech-savvy participants; and these are often the users that you can learn most from.
  • The personal touch – Despite huge strides with technology, there are some things that skilled people can do best by deploying all their human senses, experience and intelligence. In-person testing allows a user researcher to deploy all of their skills, perform to their highest level and deliver their best work.
  • Generate deeper insights – In-person testing is an in-depth, two-way, flexible interaction. Of all approaches, in-person testing delivers the deepest, most reliable insights.

Disadvantages

  • Cost – Because in-person testing requires thorough planning, the right facilities, participant reimbursements, and is carried out by experienced experts, it can cost more than other approaches. However, as with many things, if you want the best and most valuable outputs, you need to be prepared to pay for them.
  • Observer effect – People change their behaviour when they know they’re being observed. This is a limitation of all human-centred research and you can reduce the effects of it through properly briefing participants and ensuring the test environment is welcoming and relaxed.

Online User Testing

Testing remotely, using screen sharing software and a video calling suite such as Skype, FaceTime or Hangouts.

Advantages

  • Still moderated – Online testing can be delivered by a skilled and experience user researcher.
  • Accessing participants anywhere – Sometimes it’s hard or impossible to reach your customers. They may live in a different country, or they may be dotted across many different locations. In this case, online user testing offers you an effective method for accessing users wherever they are, so long as they have a strong internet connection.
  • No lab required – Whereas with in-person testing you need a quiet room to run testing from, you could run online user testing from your desk.

Disadvantages

  • Tech inevitably goes wrong – You’re relying on your internet connection as well as the internet connection of your participant. Drop outs will happen. Furthermore, you’re relying on the speed and quality of your participants PC, smartphone, microphone and webcam – all of which might be slow or running out of date software.
  • Participants need to set up the tech on their side –To fully capture an online user testing session, a participant needs to set up screen sharing software and voice calling. It shouldn’t be underestimated just how many potential users would be unable to accomplish this on their own. Many users (typically of an older generation) who know how to use the internet may be unable or unhappy to install and run a new piece of software.
  • Sub-optimal testing environment – If a user is being tested remotely, it’s likely they’ll be doing this from home. Whilst you can advise they find a quiet time to do this, distractions can still happen that completely take a participant’s focus away from the testing.

Online User Testing Platforms

Utilise an online user testing platform such as UserTesting or WhatUsersDo.

Advantages

  • Convenience – An online user testing platform will typically recruit users, test users and sometimes even report the findings of the testing on your behalf. This can therefore significantly reduce the time or effort required to conduct user testing.
  • Cheap – This approach can, at least on the face of it, appear much cheaper than moderated testing.
  • Reduced ‘observer’ effect – As online user testing platforms are unmoderated with no facilitator present, users may act more naturally as they’re not being so directly observed. However, as will be discussed below, these platforms often use a system for rating how useful participants are, which actually increases the likelihood of biases.

Disadvantages

Online User Testing platforms have all of the same disadvantages as typical online user testing (as above), but also suffer from the following additional disadvantages…

  • Unmoderated – These platforms often display a set of instructions on the screen for a participant to follow but do not have a research or facilitator present. This drastically reduces how much support a facilitator or moderator can provide if a participant is stuck or unclear about the task they have been asked to carry out. Additionally, it makes ‘Think Aloud’ feel incredibly unnatural for a participant. Asking a participant to think aloud is a technique which is often used by a user researcher to help understand a participants actions, but having a participant do this alone, with no facilitator is a very forced and unnatural experience which they are unlikely to fully buy into.
  • No opportunities for follow-up – Moderated testing (in-person and online) gives you the opportunity to ask questions at the end of the session if a participant has done something unexpected. This can be incredibly useful for clarifying behaviour and understanding problems a user has faced. Because online user testing platforms are un-moderated, there is no opportunity to do this, leaving you to guess why users have struggled.
  • Professional testers – Participants who carry out testing on platforms such as this are able to complete multiple tests per day. This is made worse by the fact that many of these platforms ask companies to ‘rate’ the users who took part in their testing for how helpful they were. If participants receive good ratings, they get more opportunities to complete more tests. This creates a troubling environment for research, where participants are rewarded for saying things that they think a researcher wants to hear. This fundamentally undermines how good research should be conducted and seriously alters how comfortable and open a participant will feel.
  • Complicated, specialised software – Even more so than for typical online user testing, user testing platforms ask participants to download dedicated software for completing user testing sessions. This is often a complicated process, particularly for mobile testing where apps need to be installed via third-party developers. This drastically limits the types of participants who will be available for testing to those who are tech-savvy enough to understand how to download and install the software in the first place, meaning you may be ruling out the very people you’re wanting to test your product with.

So which one is right for me?

Running any form of user testing is better than running no user testing. However, when deciding on the best approach it’s important to recognise that you get out what you put in.

Whilst in-person user testing can be more time consuming and costly, this is balanced out by the authenticity and quality of the data that can be captured.

Online user testing is a great alternative when participants are difficult to reach. However, running this testing yourself rather than through a user testing platform can offer you some key advantages – you can talk to participants throughout the testing and also have greater control over who the users are.

If you’re curious about running user testing and are intrigued to try it for yourself, a user testing platform can be a useful tool for doing so, providing you are aware of the limitations, particularly for the types of users you are likely to end up testing.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408

TALK TO US
we would love to hear about your project