UX Testing eCommerce Websites

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

When, at the beginning of the year, we received our first opportunity to take our UX Testing process and apply it to an eCommerce website we were anxious…

Would we find anything of value? Would the client think they’d wasted their money? How hard can it be to find a product, add it to basket and complete the transaction?

We needn’t have worried. As it turns out, UX testing an eCommerce platform fills a gap that analytics and other Conversion Rate Optimisation tools just cannot cover.

We realised we were onto something as soon as we started presenting back to clients and witnessed their reactions as they saw target customers talk about their products and their brand and use their website and the website of their competitors in a way that they had just never been able to see before.

Seeing how valuable our process was, we developed it further and now have a core UX Testing service for eCommerce that has just been commissioned by one of the high street’s biggest names. This means we’ll shortly be using our testing service on a £1 billion website.

That’s the beauty of it. It doesn’t matter how small, or how large the company – the website-customer dynamic is very similar. Critically, whether it’s an SME or an international retailer, they’ve rarely witnessed customers talking and using their website in this way before and the insights generated by our analysis impact at both the tactical and strategic level.

Our approach to eCommerce UX Testing maintains many of the features of our other testing.

  • Between 5 and 10 participants per device type (desktop, tablet, mobile), very carefully recruited to meet the precise demographics and experiences that we have agreed with our client.
  • Each participant invited to one-hour, one-to-one sessions in our fully equipped UX testing suite.

But it is the design of the research that is generating such high value insights.

Through our carefully design research sessions, we uncover:

  1. What are the most important factors for a customer when making this type of purchase?
    Through an in-depth interview we assess where they have shopped previously, what they have been looking for, where have they ended up purchasing from and why.
  2. Which websites stand out?
    By using natural, open scenarios we allow users to shop online as they would at home, uncovering the websites that stand out in terms of information, imagery, findability, and service.
  3. How does the clients website perform?
    In conjunction with open tasks, we introduce customers to our client’s website to assess how it stacks up against the competition. What’s drawing customers in? What’s putting them off? What would make them use this website to purchase their items over another?
  4. How could the UX of the website be improved?
    Our sessions are purpose-built to form a comprehensive picture of what target customers are looking for, what they see as important, and how this can be reflected in a website. We can then show our clients exactly how their UX could be optimised to drive sales, retain customers, and attract new ones.

We are helping eCommerce businesses to increase their conversion rates by seeing, analysing and understanding the buying journey through the eyes of their customers.

I’d be delighted to discuss how we can help.


Phil Randall

Meet the author:

Phil Randall
Co-Founder at Userfy

3 Ways to More Impactful User Testing

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

Discovering UX problems and finding out how to fix them is a great thing, no matter when or how it happens…

Many organisations are blissfully unaware that they are losing business because their users are getting confused or frustrated and so user testing at any time puts you ahead of the game.

But, there are things you can consider that will make your user testing even more valuable…

1) Test early

User testing just before launch, having already invested considerable time and money in design and development, is a thousand times better than never user testing, but it can be a painful experience.  At this point issues are more expensive to fix, and you may not have lots of time or money left to make the improvements needed.  Early user testing, on initial designs or prototypes, means changes can be made quickly and at much lower cost and a user-centred solution can be built from the very start of the project.

2) Test your current solution before designing the new one

“There’s no point user testing the current site because we know it’s rubbish”; we hear this a lot and on the face of it, it makes sense.  However, we have found categorically that testing the current site is a brilliant way to better understand users, learn what problems they are having, and establish how the new solution could better meet their needs.  We use the current site to facilitate an in-depth discussion about the wants, attitudes and experiences of users and these always better inform the direction, structure and features of the new solution.

3) Don’t lose sight of the findings

Sometimes the outputs from user testing challenge the beliefs of members of a project team.  After the impact of the feedback presentation, and as the weeks pass, some of those findings are forgotten or watered down, or people put their own spin on them; this isn’t deliberate scuppering, it’s just human nature.  Where we stay engaged as part of a project team we are able to champion and defend the needs of the users and ensure they stay at the centre of design decisions.

Userfy’s extensive experience helping companies from multiple different sectors, whose turnover ranges from zero to £10 billion, to test and optimise their user experience has allowed us to develop and refine a highly effective, tried and tested user testing methodology.  We’d love to help you be even more successful.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy

Userfy’s journey so far – Notes from the Google Campus Event, London

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

There was a lot of interest and some great messages of support prior to my talk at the ‘Start-up to Scale-up’ event at the Google Campus in London last week. It was a great presentation to give as it allowed me to reflect on a bigger picture than usual – giving me time to consider the progress Userfy has made since we launched in April 2017, as well as the next big steps and challenges that lie ahead.

Given the interest and due to some specific requests, I have decided to share some of my notes that I used for giving the talk, so that those who were not able to attend the event in person can still get a sense for Userfy’s story so far…

Userfy is the result of a unique partnership between myself and my co-founder Phil Randall. Together, we’ve combined my expert knowledge for conducting expert user research, with Phil’s track record for developing and scaling businesses.

Over the past year and a half we’ve developed a robust, scalable method for conducting high quality research at a speed which meets the demands of todays fast-paced tech industry.

We’re using our method to repeatedly and consistently show our clients how conducting research in this manner has no cost. Instead, it’s an investment, and it repeatedly saves them time, money and resources by making sure their strategy, their marketing, their development, and their design efforts are all in line with the needs and expectations of their target customers.

To give a recent example, just last month we were commissioned to conduct some research for a very reputable travel company who were looking to redesign their website. They’d created a new prototype based on what they all thought their customers would need. We ran some in-depth research sessions with some target users where we not only tested this prototype, but compared it against the experience of their competitors websites as well.

We uncovered some critical gaps in their current approach, and also highlighted some key ways that competitors were delivering a superior experience by providing really important information for the customer as soon as they visited the website. In reality, this could be impacting this company’s sales, as customers may be more drawn to competitor websites where they feel more able to get the important information they need to see upfront, before even contemplating whether to proceed with booking a trip or not.

This piece of research was revolutionary for this business. It completely changed their approach, and made them feel far more confident and much more certain about how to give their target customers what they want, rather than what internal voices within the company think customers want.

Off the back of that research, we’ve now been commissioned to conduct similar research sessions with each key segment of their key customer demographics.

And thats really what our mission is…. User Research isn’t a luxury, a nice add-on, or a checkbox. For a company to be successful and to excel in 2018, User Research needs to be the foundation that you build everything else on top of. 

So Userfy as a company is very much in the middle of its start-up to scale-up journey. We started in April 2017, and started by really refining and developing our proposition.

Now, with help from Accelerate Places and Blenheim Chalcot we have expanded our presence into London, as well as Nottingham, to better service our growing list of clients, as many of them are London based.

We’re now currently in that delicate phase for any business, of expanding our team whilst still ensuring we maintain the same high-standard and same expert-level quality of research. This is something we have planned ahead for though, and we’ve ensured our very systematic approach to research is something we can train new researchers to do, helping us to always maintain the same standard and quality, even at scale.

To aid our scale-up journey, we have also been having conversations with potential investors and strategic partners, and we’re considering the options in front of us for how we take Userfy to the next level – whether that be through expanding our current approach, or though becoming more formally partnered with a larger company or agency and becoming more of a centralised innovation and insights hub.

For the future, as well as our research method, the past year has also been an opportunity to consider ideas for products to enhance and complement our existing service. We see massive potential for a research tool to complement what we already do, and and to increase our access to customer insights even further, so we’ll be developing this alongside our service over the next year.


Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy



Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

Over the last year, we have run in-depth, face-to-face User Testing for numerous organisations, each at different stages of their lifecycle and with different styles and cultures. Our clients have come from many sectors, such as Finance, Fitness, Retail, Hospitality, Education, Leisure and Engineering, and their turnovers have ranged from zero to £5 billion.

However, despite these differences, one common characteristic I have seen in people who commission User Testing for the first time, is that they are brave.

Why Brave?

I say brave because the first time someone invests in User Testing it is a step into the unknown. We are sometimes asked by teams that have not experienced user testing before “What will it tell us?”. And, of course, we can’t answer that. We can’t tell them precisely what they’ll discover about their product.

We can explain how User Testing will mean they deeply understand user behaviour and that User Testing will show them what is working well on their solution, what isn’t and why. We can even show them examples of how User Testing participants have suggested great new ideas. But initially commissioning the work requires a small leap of faith, coupled with a hope that the investment will deliver insights that lead to deeper user engagement, higher sales, greater loyalty and better conversion rates.

The Rewards

I had a great email from a client recently after we presented back our findings and recommendations. He had initially and understandably been unsure about commissioning User Testing, but having seen the findings, he was very happy he had…

“… I just wanted to say thanks for all your efforts on the UX review. It is very easy not to find time or budget to do such an exercise, but they are always hugely insightful, and this was no exception. It is invaluable. I am sure we will be back in touch with Userfy in the near future.”

Be Bold!

Despite the fact that I have been working in digital and software for 17 years and my Userfy co-founder, Dr Sam Howard is a very experienced User Researcher, we can never predict what will be learned from each project.

The amazing thing is, once we see users getting confused or frustrated by something in a design journey, it’s easy to understand why it’s happening and how to fix it; despite not being able to predict the problem beforehand. Only in-depth, face-to-face User Testing does this. These unexpected insights are the real beauty of User Testing, but for Heads of UX, Product Owners, Founders, Heads of Digital, Designers and Marketing Managers, it just needs that bit of boldness to go ahead and do it for the first time.

If you are considering User Testing, you won’t regret it. In fact, it could be one of the best things you’ve ever invested in.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy

Pure “Usability” testing is no longer enough

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249
The way organisations engage with their customers online and via digital channels can make or break their business. If you’re not Google, Amazon, or another company with a brand identity and value proposition that are recognised and understood across the world, you can’t afford to just do usability testing…
Here’s why:

Traditionally, when ‘UX’ was in its infancy and the concept of designing digital interfaces to be user-focussed was still a novel premise – ‘Usability’ testing was conducted. This type of testing focused purely on whether a user was able to complete a task successfully without error and was mostly used to answer questions such as “Do users know where to click to perform this action?”, “Do users know what this icon means?” and “Do users know how to return to the homepage?”.

This was immensely important research to conduct and usability testing highlighted the tremendous confusion that can be created for a user when ambiguous or unfamiliar icons, layouts and terminology are used. Over time, companies learned of the impact that a poorly designed interface could have on their customers, and began to adopt the principles of user-centred design advocates such as Jakob Nielsen – who’s famous heuristics included the important rules of maintaining consistency, creating visibility, and matching systems to real world concepts.

Whilst usability and user-centred design principles are still the fundamental backbone of UX as an approach and a philosophy, their adoption has led us to a point where interfaces are now considerably more usable.

A side by side example of how the Nottinghamshire County Council has changed to deliver a clearer and simpler UX (2005 vs 2018).

The difference between the devices and interfaces that were used when Jakob Nielsen introduced his design heuristics, compared to the systems we use today is quite astonishing. Our computers and phones are immeasurably more complex and powerful, and the tasks we use them for are vastly more diverse.

Technological progress is made at an unprecedented rate, meaning that in what seems like an incredibly small amount of time, our industry has changed and adapted to one where you would now be hard-pushed to find a website with the classic UX issues which were so prevalent just a decade ago.

It would be foolish and naive to assume that this means we’re now all total, undeniable experts at designing for users. In reality, digital interfaces have become far more prevalent and are a considerably more essential part of our day to day lives. With the basics of UX principles now more widely understood and adopted, we need to be doing far more than just ‘usability’ testing.

One reason for this is the dramatic shift in the websites and apps we all use and have access to. In the early days of UX when usability testing first emerged, users were mostly testing the websites of large, existing companies that they already knew and were already customers of – such as supermarkets, high-street retailers, airlines, and travel agents. For these companies, a website was a new asset, but it wasn’t yet an absolutely integral and critical part of their business – much of their revenue and traffic was still from offline sources.

Now, we live in a world of industry disrupters and technology start-ups where their first and potentially only asset, is a website. Almost overnight, we have all become inhabitants of a new online battleground, where a plethora of companies you may never have heard of or used before are all competing and vying for your attention, your money and your custom.

The Internet has levelled the playing field for businesses in a way we have never seen before – meaning a new start-up with nothing more than a great marketing campaign and a flashy website can steal customers from a huge multi-million pound company who’s own website doesn’t quite stir up the same levels of excitement.

Usability testing is no longer enough because we now live in a world where a user’s first experience of a company is their website. We can no longer just ask questions about whether a user understands an icon, or knows where to click. Usability is just as important as ever, but we now also need to answer far more fundamental questions such as – “Does the user understand what this company is offering?”, “What does the user expect to be able to use this website for?” and “What factors would influence the users decision to sign up to this service?”.

This is the fundamental difference between traditional usability testing and how we conduct our research and testing at Userfy. We cover a lot more ground and generate insights far more paramount to the overall success of your business. We’ve worked hard to optimise our research process to reflect the fact that we now live in a world where a company’s online presence often is their business.


You can find out more about Userfy’s research process over on our Services page.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy

Should We User Test?

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

Since we launched Userfy 12 months ago, we have spent a great deal of time talking with clients about User Testing. Every discussion enhances our understanding, as each of our customers have their own slightly different set of needs, drivers and requirements. Over time, this has led us to spot the types of projects where User Testing can add maximum value and benefit.

Our approach to testing solutions with real users explores:

  • How effectively your value proposition is communicated and how to improve it.
  • Where users experience any confusion or frustration and how to develop a frictionless user experience.
  • How well the needs of your users are being met and what could be done better.

Here are four factors that affect the decision to invest in User Testing:

  1. How Much is at Stake?

One of the biggest factors in choosing whether or not to undertake User Testing is how much is riding on the project outcome.  By this, I mean for both the organisation and the individuals in the project team.

The organisation, may be entirely or heavily-dependant on the solution in question.  If the project falls short this can have serious consequences.

Additionally, if a project under-performs, it can reflect badly on the project team. It’s not a nice feeling being involved with a project that has missed the mark. Questions get asked, team members are left with nothing to shout about on their CV, and it can affect career progression.

Where the stakes are high and getting things right is critical, the case for User Testing is at its strongest.

  1. Solution Complexity

Even the simplest digital journeys benefit from User Testing as they can still cause a user to get confused or frustrated, leading to costly consequences.  But certainly, the more that’s happening, the more that can go wrong.  We’ve user tested the full range of solutions, from a brochure site with simple features, through to a financial application that was replicating an extended and varied off-line experience.

Whilst User Testing benefits the development of all solutions, more complex ones require thorough User Testing if they have any chance of working well and delivering a user experience that’s memorable for all the right reasons.

  1. Culture

Organisations, departments and teams sit on a wide spectrum when it comes to how customer focussed their culture is. When the loudest voice in an organisation or team (often a senior manager), thinks they already know all the answers, it becomes harder to champion the value of user research.

Fortunately, this type of culture is rare, as more and more companies recognise the need to develop a strong and effective user experience in order to stay ahead of the competition, retain existing customers and generate new ones.

At Userfy we have found that User Testing can be a catalyst for creating a more user-focused culture. Some of our clients have utilised short highlights of video footage from our User Testing sessions to help get buy-in from senior figures in their organisation – to show them undeniable evidence of the struggles users are having, and to help shift their perception away from thinking they already know everything about what their customers want, think and do.

  1. Previous Experience

People with previous experience of User Testing are often far more receptive to it. If they’ve already felt the reassurance and benefit of making evidence-based design decisions, there’s no going back to the old guessing games.

It’s a great feeling when we get inbound enquiries from a potential customer who already knows how they will benefit from User Testing. Once someone has witnessed the value that comes from testing a design, prototype or live solution with real users, they see it as an essential and integral part of the process and feel exposed without it.

Final Thoughts

In my own experience, these four factors all have a bearing on whether User Testing will be a good fit for a project. Crucially, they are interrelated, and can all affect each-other. A high-stakes project where a complex interface is being developed creates circumstances where User Testing is essential, yet if the culture isn’t right, the project can be put at much greater risk than necessary, by failing to engage with users as early as possible.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy

Tools for Analysing Data from User Research

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

User research generates a lot of data. If you’re running user testing, interviews, ethnography, or anything where you’re capturing qualitative information, you’ll probably have hours of footage or recordings to revisit and analyse.

An often overlooked and undervalued skill for a user researcher is the ability to analyse and condense all of this data into something meaningful, useful, and representative of the views and behaviour of users.

The methods which can be used for analysing qualitative data can vary depending on the data and objectives of the research. Thematic Analysis is the most common approach – where a researcher sifts through the entire data set, searching for themes and patterns from which the findings of the research emerge.

So what tools can we utilise for conducting this type of analysis? Before delving into any of them, it’s important to point out that no tool will actually analyse the data for you – the responsibility to engage and understand qualitative data always lies at the door of the researcher – these tools can simply assist with sorting, coding and arranging.

Here, we look at software, apps and more traditional methods which can help you get to grips with a qualitative data set… 


1. The Heavy Duty Tool – NVivo

NVivo is widely regarded as the tool for analysing qualitative data. It’s purpose built for getting to grips with large qualitative data sets, and is packed full of features for importing, coding, connecting, visualising and representing data.

For many, this will be total overkill. It’s a fairly intimidating and overwhelming piece of software and you can usually achieve everything you need to using something much simpler.

2. The Traditional Method – Good old Post-It Notes

Personally, this will always be my preferred method. There’s something about the physical space that you can use with Post-It notes which makes the data so much easier to understand and visualise.

A clear downside to this method is the need to write out each post it note by hand. This can take hours. However, in my own personal experience this is often the best possible way to properly immerse yourself in the data. By writing out key points onto post it notes, your brain starts to join the dots and spot emerging patterns and trends as you go.

A Post-It wall can also be a fantastic way to collaborate and share your ideas with the rest of your team, allowing you to move notes and rearrange things based on your discussions.

3. The Digital Alternative – Post It Notes Plus

The Post It Notes Plus app has SO much potential, but has sadly barely been updated and improved since it’s original release in 2014.

The app allows you to create a digital post-it wall – you can either type or write out onto post-it notes and then arrange and group them on a board. This does work relatively well, particularly if you have an iPad.

However, a few things seriously let it down. Firstly, it’s buggy and prone to crashing, meaning you can end up losing your entire dataset with no way of recovering it. This happened to me on one occasion after over 5 hours of analysis and I lost everything (Yes, I’m still bitter).

Secondly, the app offers several ways of exporting your work, including to an Excel spreadsheet. However, it unfortunately exports every Post-It as an image! For a researcher this is far from ideal, as you’d really want it to be able to export everything in a textual format, so you have a usable database of your themes, quotes, comments etc.

Finally, the app doesn’t offer any form of collaboration. You can’t work with a fellow researcher on the same board, meaning its no where near as effective for collaborating on research findings as a real-world post-it wall is.


These are just 3 options, plenty of others are available but they all fall into similar categories to the tools mentioned above.

Ultimately, the tool that’s right for you may depend on how your mind visualises data and how deep you want to dive.

If your data set is so huge that post-its simply would not be feasible, a heavy duty tool such as NVivo may be the way to go. However, if time allows, my personal preference would always be to write out everything by hand and use paper post-it notes. Not only do I find this the best way to understand the data and visualise my findings, I also think it’s the best way of encouraging you to continually refine your thoughts. Whilst you can close down an application and put your research to the back of your mind, having a post-it wall physically present in your office can be a great way of keeping your data fresh in your head.

Which tool do you find works best for you?

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy

Podcast- Dr Sam Howard speaks to VUXWorld

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

Dr Sam Howard, co-founder and Director of Research for Userfy, recently sat down with Kane Simms, host of VUXWorld, to discuss how user testing can be adapted and successfully utilised when designing new systems and interfaces for voice-first devices such as the Amazon Echo and Google Home.

VUXWorld is new a podcast dedicated to voice user experience design and voice-first strategies. Sam was the first guest to appear on the podcast, featuring in episode 2 of this exciting new series.

Kane and Sam spoke about how Userfy started and the types of research Userfy specialise in, before delving into some deeper discussions about how user testing can be adapted and used in this new and exciting area of digital technology where visual screens are no longer needed.

You can listen to the podcast through the YouTube link below, or you can subscribe to the series and listen to other episodes via iTunes.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy

What Makes an Outstanding User Tester?

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

User Testing can be an art form. A performance that you repeat and replicate, over and over again. Any expert user tester has a finely tuned script in their head – a sequence of steps they rely on to welcome a participant, explain the research and put them at ease. This script develops and improves with practise and experience. Over time, an expert user tester also develops a rapid ability to assess the personality, confidence and comfort levels of participants – quickly identifying when extra time is required to put a participant at ease and ensure they feel happy and comfortable enough to be open, honest and behave naturally.

When considering what makes an outstanding user tester, it’s easy to only focus on their knowledge of research methods and overall experience working within a UX-oriented environment. However, there are some far more fundamental traits and abilities that can be the difference between an ‘ok’ user tester and one that truly excels in this type of research environment…

  1. Patient – Patience is about waiting for things to happen in their own time. In user testing, patience is vital for uncovering new insights into user behaviour. When a user gets confused or is struggling with a task, a natural response might be to help guide them back towards what you asked them to do. However, stepping back and taking a few moments to consider why a user is experiencing confusion and thinking about what isn’t clear enough in the digital solution you’re testing, can be incredibly valuable for learning where the user experience is failing.
  2. Approachable – When running user testing, it’s useful to think of your participants as customers. In the service industry, an unfriendly and hostile approach to customers can destroy a business. In user testing, it could seriously impact your findings. A user tester should convey their warm and friendly persona from the moment a participant steps through the door. This can do wonders for making a participant commit to the research and ensures they know that the fact that they have taken time out of their day to offer their feedback is both valued and appreciated. It can’t be underestimated just how much of a difference this can make to the findings you’re likely to obtain from a user testing session. If a participant senses that you’re not enthusiastic about the research, why should they give it their full attention?
  3. Observant – User testing sessions can be long, and it can be hard to stay focused when you know the website or app that’s being tested so well. However, an outstanding user tester stays observant and focused on exactly what a participant is doing. This is really important to do, so that when issues do arise, the user tester can immediately probe the participant for what is happening, what the issue is, and why. Being observant during the testing itself is also beneficial when it comes to analysing the videos and pulling out the key findings, as you will already have a good sense for when key moments have occurred.
  4. Flexible – It is an inevitability with any research project (particularly when technology is involved) that things will go wrong. Wi-Fi drop outs, unexpected system errors, and last minute changes to the interface you’re testing can all disrupt and change a testing session. Being a great improviser and being able to flexibly adapt to the situation at hand is therefore a really important skill to have when running user testing. When things go wrong, it’s important to quickly and intelligently respond in a way that means you can still obtain the maximum value possible from the session, despite the circumstances you’re now faced with.
  5. A Fantastic Listener – A user tester should always make the participant the central focus of the room. They are there to facilitate and guide conversation – particularly if a participant requires encouragement, but they should never lead conversation or speak over a participant. An outstanding user tester will not only be a great listener, but will also be able to withstand potentially awkward moments of silence long enough for a participant to be the one to continue talking and offering their opinion. If a user tester isn’t a good listener, it’s likely that the participant will sense this and as the session progresses will begin to talk less and less. If a user tester is a good listener, a participant will equally sense this as well, and as the session progresses will talk more and more openly.

On their own, these traits are not enough. An expert knowledge of research methodologies, UX principles and data analysis are just some of the other factors that are critical for conducting high-quality user testing. However, when it comes to dealing with real participants, face to face, an expert knowledge of research methods will be rendered entirely useless if you’re unable to connect and empathise with the person sat in front of you.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy

Should you be running user testing in-person or online?

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

Putting your design, solution or product in front of your target customers is the ideal opportunity to gather insightful feedback on how understandable, usable and engaging it is and allows you to test the critical assumptions you’ve made about what your users think, want and need. The findings from User Testing can help you improve your solution to align it with user needs and expectations, helping you to generate a higher return.

There are a few different approaches you can take to conducting user testing:

  • You can run the testing in-person, sitting down with users and observing them use your solution in a quiet space or dedicated lab facility.
  • You can run the testing remotely, using screen sharing software and a video calling suite such as Skype, FaceTime or Hangouts.
  • Or, a third option would be to utilise an online user testing platform such as UserTesting or WhatUsersDo.


How do you decide which option is best? Here, we consider the advantages and disadvantages of all three approaches to help you decide which one is right for your project…


In-Person User Testing

Sitting down with users and observing them use your solution in a quiet space or dedicated lab facility.


  • Test incomplete designs – Often the material being tested is far from complete (after all, there are huge benefits to testing designs as early as possible). In these situations, a skilled researcher sat with a participant can set the scene, keep the test on track and explain things where needed. About two thirds of the User Testing we do is on designs or prototypes, rather than live, fully working solutions and in-person testing is ideal for this.
  • Better briefed participants – As with all communication, but particularly anything with an element of complexity, it’s much easier to brief someone and be sure they understand when you are face-to-face with them. When it comes to User Testing it is critical that participants are clear about the brief and have the chance to ask questions.
  • Better prepared participants – The value gained from user testing is impacted by how comfortable participants feel. Participants need to be in a frame of mind where they can safely and confidently provide feedback. In-person testing can provide a warm, welcoming and re-assuring environment that enables participants to be fully focused on the task in hand.
  • Recruitment – With online or remote testing you are already pre-selecting a tech-savvy participant by the very nature that they are confident using the very technology needed to undertake the research. In-person testing allows you to invite in and test with less tech-savvy participants; and these are often the users that you can learn most from.
  • The personal touch – Despite huge strides with technology, there are some things that skilled people can do best by deploying all their human senses, experience and intelligence. In-person testing allows a user researcher to deploy all of their skills, perform to their highest level and deliver their best work.
  • Generate deeper insights – In-person testing is an in-depth, two-way, flexible interaction. Of all approaches, in-person testing delivers the deepest, most reliable insights.


  • Cost – Because in-person testing requires thorough planning, the right facilities, participant reimbursements, and is carried out by experienced experts, it can cost more than other approaches. However, as with many things, if you want the best and most valuable outputs, you need to be prepared to pay for them.
  • Observer effect – People change their behaviour when they know they’re being observed. This is a limitation of all human-centred research and you can reduce the effects of it through properly briefing participants and ensuring the test environment is welcoming and relaxed.

Online User Testing

Testing remotely, using screen sharing software and a video calling suite such as Skype, FaceTime or Hangouts.


  • Still moderated – Online testing can be delivered by a skilled and experience user researcher.
  • Accessing participants anywhere – Sometimes it’s hard or impossible to reach your customers. They may live in a different country, or they may be dotted across many different locations. In this case, online user testing offers you an effective method for accessing users wherever they are, so long as they have a strong internet connection.
  • No lab required – Whereas with in-person testing you need a quiet room to run testing from, you could run online user testing from your desk.


  • Tech inevitably goes wrong – You’re relying on your internet connection as well as the internet connection of your participant. Drop outs will happen. Furthermore, you’re relying on the speed and quality of your participants PC, smartphone, microphone and webcam – all of which might be slow or running out of date software.
  • Participants need to set up the tech on their side –To fully capture an online user testing session, a participant needs to set up screen sharing software and voice calling. It shouldn’t be underestimated just how many potential users would be unable to accomplish this on their own. Many users (typically of an older generation) who know how to use the internet may be unable or unhappy to install and run a new piece of software.
  • Sub-optimal testing environment – If a user is being tested remotely, it’s likely they’ll be doing this from home. Whilst you can advise they find a quiet time to do this, distractions can still happen that completely take a participant’s focus away from the testing.

Online User Testing Platforms

Utilise an online user testing platform such as UserTesting or WhatUsersDo.


  • Convenience – An online user testing platform will typically recruit users, test users and sometimes even report the findings of the testing on your behalf. This can therefore significantly reduce the time or effort required to conduct user testing.
  • Cheap – This approach can, at least on the face of it, appear much cheaper than moderated testing.
  • Reduced ‘observer’ effect – As online user testing platforms are unmoderated with no facilitator present, users may act more naturally as they’re not being so directly observed. However, as will be discussed below, these platforms often use a system for rating how useful participants are, which actually increases the likelihood of biases.


Online User Testing platforms have all of the same disadvantages as typical online user testing (as above), but also suffer from the following additional disadvantages…

  • Unmoderated – These platforms often display a set of instructions on the screen for a participant to follow but do not have a research or facilitator present. This drastically reduces how much support a facilitator or moderator can provide if a participant is stuck or unclear about the task they have been asked to carry out. Additionally, it makes ‘Think Aloud’ feel incredibly unnatural for a participant. Asking a participant to think aloud is a technique which is often used by a user researcher to help understand a participants actions, but having a participant do this alone, with no facilitator is a very forced and unnatural experience which they are unlikely to fully buy into.
  • No opportunities for follow-up – Moderated testing (in-person and online) gives you the opportunity to ask questions at the end of the session if a participant has done something unexpected. This can be incredibly useful for clarifying behaviour and understanding problems a user has faced. Because online user testing platforms are un-moderated, there is no opportunity to do this, leaving you to guess why users have struggled.
  • Professional testers – Participants who carry out testing on platforms such as this are able to complete multiple tests per day. This is made worse by the fact that many of these platforms ask companies to ‘rate’ the users who took part in their testing for how helpful they were. If participants receive good ratings, they get more opportunities to complete more tests. This creates a troubling environment for research, where participants are rewarded for saying things that they think a researcher wants to hear. This fundamentally undermines how good research should be conducted and seriously alters how comfortable and open a participant will feel.
  • Complicated, specialised software – Even more so than for typical online user testing, user testing platforms ask participants to download dedicated software for completing user testing sessions. This is often a complicated process, particularly for mobile testing where apps need to be installed via third-party developers. This drastically limits the types of participants who will be available for testing to those who are tech-savvy enough to understand how to download and install the software in the first place, meaning you may be ruling out the very people you’re wanting to test your product with.

So which one is right for me?

Running any form of user testing is better than running no user testing. However, when deciding on the best approach it’s important to recognise that you get out what you put in.

Whilst in-person user testing can be more time consuming and costly, this is balanced out by the authenticity and quality of the data that can be captured.

Online user testing is a great alternative when participants are difficult to reach. However, running this testing yourself rather than through a user testing platform can offer you some key advantages – you can talk to participants throughout the testing and also have greater control over who the users are.

If you’re curious about running user testing and are intrigued to try it for yourself, a user testing platform can be a useful tool for doing so, providing you are aware of the limitations, particularly for the types of users you are likely to end up testing.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy