3 Ways to More Impactful User Testing


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

Discovering UX problems and finding out how to fix them is a great thing, no matter when or how it happens…

Many organisations are blissfully unaware that they are losing business because their users are getting confused or frustrated and so user testing at any time puts you ahead of the game.

But, there are things you can consider that will make your user testing even more valuable…

1) Test early

User testing just before launch, having already invested considerable time and money in design and development, is a thousand times better than never user testing, but it can be a painful experience.  At this point issues are more expensive to fix, and you may not have lots of time or money left to make the improvements needed.  Early user testing, on initial designs or prototypes, means changes can be made quickly and at much lower cost and a user-centred solution can be built from the very start of the project.

2) Test your current solution before designing the new one

“There’s no point user testing the current site because we know it’s rubbish”; we hear this a lot and on the face of it, it makes sense.  However, we have found categorically that testing the current site is a brilliant way to better understand users, learn what problems they are having, and establish how the new solution could better meet their needs.  We use the current site to facilitate an in-depth discussion about the wants, attitudes and experiences of users and these always better inform the direction, structure and features of the new solution.

3) Don’t lose sight of the findings

Sometimes the outputs from user testing challenge the beliefs of members of a project team.  After the impact of the feedback presentation, and as the weeks pass, some of those findings are forgotten or watered down, or people put their own spin on them; this isn’t deliberate scuppering, it’s just human nature.  Where we stay engaged as part of a project team we are able to champion and defend the needs of the users and ensure they stay at the centre of design decisions.

Userfy’s extensive experience helping companies from multiple different sectors, whose turnover ranges from zero to £10 billion, to test and optimise their user experience has allowed us to develop and refine a highly effective, tried and tested user testing methodology.  We’d love to help you be even more successful.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy
07712669935

BRAVE PEOPLE COMMISSION USER TESTING


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

Over the last year, we have run in-depth, face-to-face User Testing for numerous organisations, each at different stages of their lifecycle and with different styles and cultures. Our clients have come from many sectors, such as Finance, Fitness, Retail, Hospitality, Education, Leisure and Engineering, and their turnovers have ranged from zero to £5 billion.

However, despite these differences, one common characteristic I have seen in people who commission User Testing for the first time, is that they are brave.

Why Brave?

I say brave because the first time someone invests in User Testing it is a step into the unknown. We are sometimes asked by teams that have not experienced user testing before “What will it tell us?”. And, of course, we can’t answer that. We can’t tell them precisely what they’ll discover about their product.

We can explain how User Testing will mean they deeply understand user behaviour and that User Testing will show them what is working well on their solution, what isn’t and why. We can even show them examples of how User Testing participants have suggested great new ideas. But initially commissioning the work requires a small leap of faith, coupled with a hope that the investment will deliver insights that lead to deeper user engagement, higher sales, greater loyalty and better conversion rates.

The Rewards

I had a great email from a client recently after we presented back our findings and recommendations. He had initially and understandably been unsure about commissioning User Testing, but having seen the findings, he was very happy he had…

“… I just wanted to say thanks for all your efforts on the UX review. It is very easy not to find time or budget to do such an exercise, but they are always hugely insightful, and this was no exception. It is invaluable. I am sure we will be back in touch with Userfy in the near future.”

Be Bold!

Despite the fact that I have been working in digital and software for 17 years and my Userfy co-founder, Dr Sam Howard is a very experienced User Researcher, we can never predict what will be learned from each project.

The amazing thing is, once we see users getting confused or frustrated by something in a design journey, it’s easy to understand why it’s happening and how to fix it; despite not being able to predict the problem beforehand. Only in-depth, face-to-face User Testing does this. These unexpected insights are the real beauty of User Testing, but for Heads of UX, Product Owners, Founders, Heads of Digital, Designers and Marketing Managers, it just needs that bit of boldness to go ahead and do it for the first time.

If you are considering User Testing, you won’t regret it. In fact, it could be one of the best things you’ve ever invested in.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy
07712669935

Pure “Usability” testing is no longer enough


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201
The way organisations engage with their customers online and via digital channels can make or break their business. If you’re not Google, Amazon, or another company with a brand identity and value proposition that are recognised and understood across the world, you can’t afford to just do usability testing…
Here’s why:

Traditionally, when ‘UX’ was in its infancy and the concept of designing digital interfaces to be user-focussed was still a novel premise – ‘Usability’ testing was conducted. This type of testing focused purely on whether a user was able to complete a task successfully without error and was mostly used to answer questions such as “Do users know where to click to perform this action?”, “Do users know what this icon means?” and “Do users know how to return to the homepage?”.

This was immensely important research to conduct and usability testing highlighted the tremendous confusion that can be created for a user when ambiguous or unfamiliar icons, layouts and terminology are used. Over time, companies learned of the impact that a poorly designed interface could have on their customers, and began to adopt the principles of user-centred design advocates such as Jakob Nielsen – who’s famous heuristics included the important rules of maintaining consistency, creating visibility, and matching systems to real world concepts.

Whilst usability and user-centred design principles are still the fundamental backbone of UX as an approach and a philosophy, their adoption has led us to a point where interfaces are now considerably more usable.

A side by side example of how the Nottinghamshire County Council has changed to deliver a clearer and simpler UX (2005 vs 2018).

The difference between the devices and interfaces that were used when Jakob Nielsen introduced his design heuristics, compared to the systems we use today is quite astonishing. Our computers and phones are immeasurably more complex and powerful, and the tasks we use them for are vastly more diverse.

Technological progress is made at an unprecedented rate, meaning that in what seems like an incredibly small amount of time, our industry has changed and adapted to one where you would now be hard-pushed to find a website with the classic UX issues which were so prevalent just a decade ago.

It would be foolish and naive to assume that this means we’re now all total, undeniable experts at designing for users. In reality, digital interfaces have become far more prevalent and are a considerably more essential part of our day to day lives. With the basics of UX principles now more widely understood and adopted, we need to be doing far more than just ‘usability’ testing.

One reason for this is the dramatic shift in the websites and apps we all use and have access to. In the early days of UX when usability testing first emerged, users were mostly testing the websites of large, existing companies that they already knew and were already customers of – such as supermarkets, high-street retailers, airlines, and travel agents. For these companies, a website was a new asset, but it wasn’t yet an absolutely integral and critical part of their business – much of their revenue and traffic was still from offline sources.

Now, we live in a world of industry disrupters and technology start-ups where their first and potentially only asset, is a website. Almost overnight, we have all become inhabitants of a new online battleground, where a plethora of companies you may never have heard of or used before are all competing and vying for your attention, your money and your custom.

The Internet has levelled the playing field for businesses in a way we have never seen before – meaning a new start-up with nothing more than a great marketing campaign and a flashy website can steal customers from a huge multi-million pound company who’s own website doesn’t quite stir up the same levels of excitement.

Usability testing is no longer enough because we now live in a world where a user’s first experience of a company is their website. We can no longer just ask questions about whether a user understands an icon, or knows where to click. Usability is just as important as ever, but we now also need to answer far more fundamental questions such as – “Does the user understand what this company is offering?”, “What does the user expect to be able to use this website for?” and “What factors would influence the users decision to sign up to this service?”.

This is the fundamental difference between traditional usability testing and how we conduct our research and testing at Userfy. We cover a lot more ground and generate insights far more paramount to the overall success of your business. We’ve worked hard to optimise our research process to reflect the fact that we now live in a world where a company’s online presence often is their business.

 

You can find out more about Userfy’s research process over on our Services page.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408

Should We User Test?


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

Since we launched Userfy 12 months ago, we have spent a great deal of time talking with clients about User Testing. Every discussion enhances our understanding, as each of our customers have their own slightly different set of needs, drivers and requirements. Over time, this has led us to spot the types of projects where User Testing can add maximum value and benefit.

Our approach to testing solutions with real users explores:

  • How effectively your value proposition is communicated and how to improve it.
  • Where users experience any confusion or frustration and how to develop a frictionless user experience.
  • How well the needs of your users are being met and what could be done better.

Here are four factors that affect the decision to invest in User Testing:

  1. How Much is at Stake?

One of the biggest factors in choosing whether or not to undertake User Testing is how much is riding on the project outcome.  By this, I mean for both the organisation and the individuals in the project team.

The organisation, may be entirely or heavily-dependant on the solution in question.  If the project falls short this can have serious consequences.

Additionally, if a project under-performs, it can reflect badly on the project team. It’s not a nice feeling being involved with a project that has missed the mark. Questions get asked, team members are left with nothing to shout about on their CV, and it can affect career progression.

Where the stakes are high and getting things right is critical, the case for User Testing is at its strongest.

  1. Solution Complexity

Even the simplest digital journeys benefit from User Testing as they can still cause a user to get confused or frustrated, leading to costly consequences.  But certainly, the more that’s happening, the more that can go wrong.  We’ve user tested the full range of solutions, from a brochure site with simple features, through to a financial application that was replicating an extended and varied off-line experience.

Whilst User Testing benefits the development of all solutions, more complex ones require thorough User Testing if they have any chance of working well and delivering a user experience that’s memorable for all the right reasons.

  1. Culture

Organisations, departments and teams sit on a wide spectrum when it comes to how customer focussed their culture is. When the loudest voice in an organisation or team (often a senior manager), thinks they already know all the answers, it becomes harder to champion the value of user research.

Fortunately, this type of culture is rare, as more and more companies recognise the need to develop a strong and effective user experience in order to stay ahead of the competition, retain existing customers and generate new ones.

At Userfy we have found that User Testing can be a catalyst for creating a more user-focused culture. Some of our clients have utilised short highlights of video footage from our User Testing sessions to help get buy-in from senior figures in their organisation – to show them undeniable evidence of the struggles users are having, and to help shift their perception away from thinking they already know everything about what their customers want, think and do.

  1. Previous Experience

People with previous experience of User Testing are often far more receptive to it. If they’ve already felt the reassurance and benefit of making evidence-based design decisions, there’s no going back to the old guessing games.

It’s a great feeling when we get inbound enquiries from a potential customer who already knows how they will benefit from User Testing. Once someone has witnessed the value that comes from testing a design, prototype or live solution with real users, they see it as an essential and integral part of the process and feel exposed without it.

Final Thoughts

In my own experience, these four factors all have a bearing on whether User Testing will be a good fit for a project. Crucially, they are interrelated, and can all affect each-other. A high-stakes project where a complex interface is being developed creates circumstances where User Testing is essential, yet if the culture isn’t right, the project can be put at much greater risk than necessary, by failing to engage with users as early as possible.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy
07712669935

Tools for Analysing Data from User Research


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

User research generates a lot of data. If you’re running user testing, interviews, ethnography, or anything where you’re capturing qualitative information, you’ll probably have hours of footage or recordings to revisit and analyse.

An often overlooked and undervalued skill for a user researcher is the ability to analyse and condense all of this data into something meaningful, useful, and representative of the views and behaviour of users.

The methods which can be used for analysing qualitative data can vary depending on the data and objectives of the research. Thematic Analysis is the most common approach – where a researcher sifts through the entire data set, searching for themes and patterns from which the findings of the research emerge.

So what tools can we utilise for conducting this type of analysis? Before delving into any of them, it’s important to point out that no tool will actually analyse the data for you – the responsibility to engage and understand qualitative data always lies at the door of the researcher – these tools can simply assist with sorting, coding and arranging.

Here, we look at software, apps and more traditional methods which can help you get to grips with a qualitative data set… 

 

1. The Heavy Duty Tool – NVivo

NVivo is widely regarded as the tool for analysing qualitative data. It’s purpose built for getting to grips with large qualitative data sets, and is packed full of features for importing, coding, connecting, visualising and representing data.

For many, this will be total overkill. It’s a fairly intimidating and overwhelming piece of software and you can usually achieve everything you need to using something much simpler.

2. The Traditional Method – Good old Post-It Notes

Personally, this will always be my preferred method. There’s something about the physical space that you can use with Post-It notes which makes the data so much easier to understand and visualise.

A clear downside to this method is the need to write out each post it note by hand. This can take hours. However, in my own personal experience this is often the best possible way to properly immerse yourself in the data. By writing out key points onto post it notes, your brain starts to join the dots and spot emerging patterns and trends as you go.

A Post-It wall can also be a fantastic way to collaborate and share your ideas with the rest of your team, allowing you to move notes and rearrange things based on your discussions.

3. The Digital Alternative – Post It Notes Plus

The Post It Notes Plus app has SO much potential, but has sadly barely been updated and improved since it’s original release in 2014.

The app allows you to create a digital post-it wall – you can either type or write out onto post-it notes and then arrange and group them on a board. This does work relatively well, particularly if you have an iPad.

However, a few things seriously let it down. Firstly, it’s buggy and prone to crashing, meaning you can end up losing your entire dataset with no way of recovering it. This happened to me on one occasion after over 5 hours of analysis and I lost everything (Yes, I’m still bitter).

Secondly, the app offers several ways of exporting your work, including to an Excel spreadsheet. However, it unfortunately exports every Post-It as an image! For a researcher this is far from ideal, as you’d really want it to be able to export everything in a textual format, so you have a usable database of your themes, quotes, comments etc.

Finally, the app doesn’t offer any form of collaboration. You can’t work with a fellow researcher on the same board, meaning its no where near as effective for collaborating on research findings as a real-world post-it wall is.

Verdict?

These are just 3 options, plenty of others are available but they all fall into similar categories to the tools mentioned above.

Ultimately, the tool that’s right for you may depend on how your mind visualises data and how deep you want to dive.

If your data set is so huge that post-its simply would not be feasible, a heavy duty tool such as NVivo may be the way to go. However, if time allows, my personal preference would always be to write out everything by hand and use paper post-it notes. Not only do I find this the best way to understand the data and visualise my findings, I also think it’s the best way of encouraging you to continually refine your thoughts. Whilst you can close down an application and put your research to the back of your mind, having a post-it wall physically present in your office can be a great way of keeping your data fresh in your head.

Which tool do you find works best for you?

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408

Podcast- Dr Sam Howard speaks to VUXWorld


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

Dr Sam Howard, co-founder and Director of Research for Userfy, recently sat down with Kane Simms, host of VUXWorld, to discuss how user testing can be adapted and successfully utilised when designing new systems and interfaces for voice-first devices such as the Amazon Echo and Google Home.

VUXWorld is new a podcast dedicated to voice user experience design and voice-first strategies. Sam was the first guest to appear on the podcast, featuring in episode 2 of this exciting new series.

Kane and Sam spoke about how Userfy started and the types of research Userfy specialise in, before delving into some deeper discussions about how user testing can be adapted and used in this new and exciting area of digital technology where visual screens are no longer needed.

You can listen to the podcast through the YouTube link below, or you can subscribe to the series and listen to other episodes via iTunes.

Phil Randall

Meet the author:

Phil Randall
Co-Founder and CEO at Userfy
07712669935

What Makes an Outstanding User Tester?


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

User Testing can be an art form. A performance that you repeat and replicate, over and over again. Any expert user tester has a finely tuned script in their head – a sequence of steps they rely on to welcome a participant, explain the research and put them at ease. This script develops and improves with practise and experience. Over time, an expert user tester also develops a rapid ability to assess the personality, confidence and comfort levels of participants – quickly identifying when extra time is required to put a participant at ease and ensure they feel happy and comfortable enough to be open, honest and behave naturally.

When considering what makes an outstanding user tester, it’s easy to only focus on their knowledge of research methods and overall experience working within a UX-oriented environment. However, there are some far more fundamental traits and abilities that can be the difference between an ‘ok’ user tester and one that truly excels in this type of research environment…

  1. Patient – Patience is about waiting for things to happen in their own time. In user testing, patience is vital for uncovering new insights into user behaviour. When a user gets confused or is struggling with a task, a natural response might be to help guide them back towards what you asked them to do. However, stepping back and taking a few moments to consider why a user is experiencing confusion and thinking about what isn’t clear enough in the digital solution you’re testing, can be incredibly valuable for learning where the user experience is failing.
  2. Approachable – When running user testing, it’s useful to think of your participants as customers. In the service industry, an unfriendly and hostile approach to customers can destroy a business. In user testing, it could seriously impact your findings. A user tester should convey their warm and friendly persona from the moment a participant steps through the door. This can do wonders for making a participant commit to the research and ensures they know that the fact that they have taken time out of their day to offer their feedback is both valued and appreciated. It can’t be underestimated just how much of a difference this can make to the findings you’re likely to obtain from a user testing session. If a participant senses that you’re not enthusiastic about the research, why should they give it their full attention?
  3. Observant – User testing sessions can be long, and it can be hard to stay focused when you know the website or app that’s being tested so well. However, an outstanding user tester stays observant and focused on exactly what a participant is doing. This is really important to do, so that when issues do arise, the user tester can immediately probe the participant for what is happening, what the issue is, and why. Being observant during the testing itself is also beneficial when it comes to analysing the videos and pulling out the key findings, as you will already have a good sense for when key moments have occurred.
  4. Flexible – It is an inevitability with any research project (particularly when technology is involved) that things will go wrong. Wi-Fi drop outs, unexpected system errors, and last minute changes to the interface you’re testing can all disrupt and change a testing session. Being a great improviser and being able to flexibly adapt to the situation at hand is therefore a really important skill to have when running user testing. When things go wrong, it’s important to quickly and intelligently respond in a way that means you can still obtain the maximum value possible from the session, despite the circumstances you’re now faced with.
  5. A Fantastic Listener – A user tester should always make the participant the central focus of the room. They are there to facilitate and guide conversation – particularly if a participant requires encouragement, but they should never lead conversation or speak over a participant. An outstanding user tester will not only be a great listener, but will also be able to withstand potentially awkward moments of silence long enough for a participant to be the one to continue talking and offering their opinion. If a user tester isn’t a good listener, it’s likely that the participant will sense this and as the session progresses will begin to talk less and less. If a user tester is a good listener, a participant will equally sense this as well, and as the session progresses will talk more and more openly.

On their own, these traits are not enough. An expert knowledge of research methodologies, UX principles and data analysis are just some of the other factors that are critical for conducting high-quality user testing. However, when it comes to dealing with real participants, face to face, an expert knowledge of research methods will be rendered entirely useless if you’re unable to connect and empathise with the person sat in front of you.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408

Should you be running user testing in-person or online?


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

Putting your design, solution or product in front of your target customers is the ideal opportunity to gather insightful feedback on how understandable, usable and engaging it is and allows you to test the critical assumptions you’ve made about what your users think, want and need. The findings from User Testing can help you improve your solution to align it with user needs and expectations, helping you to generate a higher return.

There are a few different approaches you can take to conducting user testing:

  • You can run the testing in-person, sitting down with users and observing them use your solution in a quiet space or dedicated lab facility.
  • You can run the testing remotely, using screen sharing software and a video calling suite such as Skype, FaceTime or Hangouts.
  • Or, a third option would be to utilise an online user testing platform such as UserTesting or WhatUsersDo.

 

How do you decide which option is best? Here, we consider the advantages and disadvantages of all three approaches to help you decide which one is right for your project…

 

In-Person User Testing

Sitting down with users and observing them use your solution in a quiet space or dedicated lab facility.

Advantages

  • Test incomplete designs – Often the material being tested is far from complete (after all, there are huge benefits to testing designs as early as possible). In these situations, a skilled researcher sat with a participant can set the scene, keep the test on track and explain things where needed. About two thirds of the User Testing we do is on designs or prototypes, rather than live, fully working solutions and in-person testing is ideal for this.
  • Better briefed participants – As with all communication, but particularly anything with an element of complexity, it’s much easier to brief someone and be sure they understand when you are face-to-face with them. When it comes to User Testing it is critical that participants are clear about the brief and have the chance to ask questions.
  • Better prepared participants – The value gained from user testing is impacted by how comfortable participants feel. Participants need to be in a frame of mind where they can safely and confidently provide feedback. In-person testing can provide a warm, welcoming and re-assuring environment that enables participants to be fully focused on the task in hand.
  • Recruitment – With online or remote testing you are already pre-selecting a tech-savvy participant by the very nature that they are confident using the very technology needed to undertake the research. In-person testing allows you to invite in and test with less tech-savvy participants; and these are often the users that you can learn most from.
  • The personal touch – Despite huge strides with technology, there are some things that skilled people can do best by deploying all their human senses, experience and intelligence. In-person testing allows a user researcher to deploy all of their skills, perform to their highest level and deliver their best work.
  • Generate deeper insights – In-person testing is an in-depth, two-way, flexible interaction. Of all approaches, in-person testing delivers the deepest, most reliable insights.

Disadvantages

  • Cost – Because in-person testing requires thorough planning, the right facilities, participant reimbursements, and is carried out by experienced experts, it can cost more than other approaches. However, as with many things, if you want the best and most valuable outputs, you need to be prepared to pay for them.
  • Observer effect – People change their behaviour when they know they’re being observed. This is a limitation of all human-centred research and you can reduce the effects of it through properly briefing participants and ensuring the test environment is welcoming and relaxed.

Online User Testing

Testing remotely, using screen sharing software and a video calling suite such as Skype, FaceTime or Hangouts.

Advantages

  • Still moderated – Online testing can be delivered by a skilled and experience user researcher.
  • Accessing participants anywhere – Sometimes it’s hard or impossible to reach your customers. They may live in a different country, or they may be dotted across many different locations. In this case, online user testing offers you an effective method for accessing users wherever they are, so long as they have a strong internet connection.
  • No lab required – Whereas with in-person testing you need a quiet room to run testing from, you could run online user testing from your desk.

Disadvantages

  • Tech inevitably goes wrong – You’re relying on your internet connection as well as the internet connection of your participant. Drop outs will happen. Furthermore, you’re relying on the speed and quality of your participants PC, smartphone, microphone and webcam – all of which might be slow or running out of date software.
  • Participants need to set up the tech on their side –To fully capture an online user testing session, a participant needs to set up screen sharing software and voice calling. It shouldn’t be underestimated just how many potential users would be unable to accomplish this on their own. Many users (typically of an older generation) who know how to use the internet may be unable or unhappy to install and run a new piece of software.
  • Sub-optimal testing environment – If a user is being tested remotely, it’s likely they’ll be doing this from home. Whilst you can advise they find a quiet time to do this, distractions can still happen that completely take a participant’s focus away from the testing.

Online User Testing Platforms

Utilise an online user testing platform such as UserTesting or WhatUsersDo.

Advantages

  • Convenience – An online user testing platform will typically recruit users, test users and sometimes even report the findings of the testing on your behalf. This can therefore significantly reduce the time or effort required to conduct user testing.
  • Cheap – This approach can, at least on the face of it, appear much cheaper than moderated testing.
  • Reduced ‘observer’ effect – As online user testing platforms are unmoderated with no facilitator present, users may act more naturally as they’re not being so directly observed. However, as will be discussed below, these platforms often use a system for rating how useful participants are, which actually increases the likelihood of biases.

Disadvantages

Online User Testing platforms have all of the same disadvantages as typical online user testing (as above), but also suffer from the following additional disadvantages…

  • Unmoderated – These platforms often display a set of instructions on the screen for a participant to follow but do not have a research or facilitator present. This drastically reduces how much support a facilitator or moderator can provide if a participant is stuck or unclear about the task they have been asked to carry out. Additionally, it makes ‘Think Aloud’ feel incredibly unnatural for a participant. Asking a participant to think aloud is a technique which is often used by a user researcher to help understand a participants actions, but having a participant do this alone, with no facilitator is a very forced and unnatural experience which they are unlikely to fully buy into.
  • No opportunities for follow-up – Moderated testing (in-person and online) gives you the opportunity to ask questions at the end of the session if a participant has done something unexpected. This can be incredibly useful for clarifying behaviour and understanding problems a user has faced. Because online user testing platforms are un-moderated, there is no opportunity to do this, leaving you to guess why users have struggled.
  • Professional testers – Participants who carry out testing on platforms such as this are able to complete multiple tests per day. This is made worse by the fact that many of these platforms ask companies to ‘rate’ the users who took part in their testing for how helpful they were. If participants receive good ratings, they get more opportunities to complete more tests. This creates a troubling environment for research, where participants are rewarded for saying things that they think a researcher wants to hear. This fundamentally undermines how good research should be conducted and seriously alters how comfortable and open a participant will feel.
  • Complicated, specialised software – Even more so than for typical online user testing, user testing platforms ask participants to download dedicated software for completing user testing sessions. This is often a complicated process, particularly for mobile testing where apps need to be installed via third-party developers. This drastically limits the types of participants who will be available for testing to those who are tech-savvy enough to understand how to download and install the software in the first place, meaning you may be ruling out the very people you’re wanting to test your product with.

So which one is right for me?

Running any form of user testing is better than running no user testing. However, when deciding on the best approach it’s important to recognise that you get out what you put in.

Whilst in-person user testing can be more time consuming and costly, this is balanced out by the authenticity and quality of the data that can be captured.

Online user testing is a great alternative when participants are difficult to reach. However, running this testing yourself rather than through a user testing platform can offer you some key advantages – you can talk to participants throughout the testing and also have greater control over who the users are.

If you’re curious about running user testing and are intrigued to try it for yourself, a user testing platform can be a useful tool for doing so, providing you are aware of the limitations, particularly for the types of users you are likely to end up testing.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408

Should you be user testing with existing customers or with new customers?


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

When it comes to user testing, you’re faced with two options:

1. Test with people who are already customers of your product or service
Or,
2. Test with people who have never used it before.

Either choice can have a profound effect on the insights you’re likely to obtain from user testing, so it’s important to make the right decision.

Testing New Users

Testing new users can be an incredibly insightful experience for revealing how people develop an understanding of your solution. Whilst your existing customers have already overcome any pitfalls in your current product, testing new users can identify issues related to customer attraction and retention that you never knew were there.

With on-boarding new customers being such a critical metric for many businesses, testing with users who are using your solution for the first time shows you where confusion, frustration or difficulties are being encountered. As a result, you discover where and how to increase acquisition rates by optimising the design of your onboarding experience.

Of course, for brand new solutions it is only possible to test with potential new users. This makes sense as initially all users will be new to the solution anyway and it’s essential to test their reaction to their first visit.

A key challenge associated with testing new users is working out how to recruit them for your research. You can’t simply dip into your current pool of customers – instead you have to consider the types of customers you’re looking to target and recruit people matching these demographics with no prior experience of your product.

Testing Existing Users

Sometimes the research requires participants to already have a good knowledge and understanding of your solution. For example, when it comes to developing new features, it’s often best to test your new design with current customers, as their prior knowledge of the solution will ensure the deepest insights are generated. If your solution is being completely redesigned, it’s also helpful to test the new designs with existing customers to ensure that you are not negatively affecting their experience and inadvertently removing or changing key features that they rely on.

Testing with existing users can be easier from a recruitment perspective as most organisations can get access to the contact details of their current customers. However, running the user testing sessions themselves with existing customers can sometimes be harder. There’s a risk that users will focus too heavily on superficial changes to the look and feel of the solution, rather than how easy it is to use. This is a natural response – particularly if they have been using the existing solution for a long time. It’s therefore important to properly facilitate the testing session to ensure users fully test the functionality of the solution, rather than just offer feedback on it’s appearance.

How we do it at Userfy

We have carried out user testing on solutions at all stages of the product lifecycle – from new concepts that are in the very early stages of design, through to well-established solutions.

We recently tested a well-established B2B website with target demographic users who had not experienced it before. This meant we could test how readily a customer can use the website on their first visit. On the other hand, we also recently tested a new feature for a financial website with existing customers, to test how well the new feature enhanced the current offer, how easy it was to understand and how much it was valued.

Importantly, with every project we do we think very carefully about who we should be testing. Whilst a lot of our time is spent planning our user testing sessions so that we can maximise the quality and value of feedback we are likely to obtain – this would be a fruitless exercise if we didn’t also carefully consider the exact types of users to test to ensure our findings are meaningful and representative.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408

 


iOS11 has an inconsistency problem


Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 201

Gestures are an integral part of touch-screen interfaces. By using ‘natural’ interactions such as tapping and swiping, a user can manipulate the interface directly to perform key tasks and operations.

Many common gestures are now second-nature to us. We know to swipe up or down to scroll a page, we know how to ‘long-press’ to reveal additional options such as select, copy and paste.

A critical part of gestures is their consistency. Gestures are useful when we consistently receive the same result when using them because this helps create an automatic reflex that we no longer even consciously think about.

However, when gestures are inconsistent, they become confusing and frustrating. Our automatic reflex may still be firing, but we’re halted in our tracks when the result we receive is suddenly unexpectedly different.

This frustration is occurring for users with multiple devices running iOS11. One of the key reasons behind this is an inconsistency in what the ‘swipe up’ gesture is used for.

What does swipe up do again…? 

If you have an iPhone 8 or an older iPhone swiping up reveals the ‘control centre’ for quickly changing key settings.

If you have an iPhone X, swiping up is now used to return to the homepage and to reveal the app switcher.

Finally, if you have an iPad Pro, swipe up is used to reveal the dock, or a slightly longer swipe up is used to reveal a combination page containing control centre and the app switcher.

Whilst swiping up on iPhone 8 and iPad Pro both allow you to view control centre through slightly different methods, with the iPhone X you now need to swipe down from the top right corner to access this.

People who use both an iPhone X and an iPad Pro currently have to learn and remember two different versions of the swipe up gesture dependent on the device they are using. This makes the gesture less of an automatic reflex, and more something they have to consciously remember and commit attention to, dependent on the device they are using at the time.

 

 

Why has this happened?

Several things have lead to this problem. Firstly, Apple drastically revamped iOS for the iPad Pro – introducing brand new ways of multitasking to allow users to use two or more applications at once.

Largely this was a fantastic step forward for the iPad, making it closer to a legitimate laptop replacement than ever. However, it did create a need for a whole new method of interaction, to allow users to access a new ‘dock’ of applications, much like on Mac OS. Swiping up on an iPad Pro now reveals this dock, allowing a user to touch and drag a second app onto the screen to be used alongside another.

With iPhone X, Apple ditched the home button. Face ID replaced Touch ID, allowing users to unlock their phone simply by looking at it. It’s a remarkable piece of innovation, but removing the home button created a need for a new way to return to the home screen. It made sense to keep this action towards the bottom of the screen, so Apple turned the swipe up into the new ‘go home’ gesture.

Why is this such a problem?

A golden rule of good user-centred design is consistency. A user should not have to guess if an action means the same thing in different situations, it should always remain consistent.

Whilst the swipe up gesture is consistent within the iPhone X, and is consistent within the iPad Pro, Apple is failing to account for the fact that many users own and use both devices. This inconsistency across devices removes the intuitive nature of the gesture and makes it far harder for a user to learn and use automatically.

Will Apple correct it? 

In time, yes. If early rumours are to be believed, the next iPad Pro will incorporate Face ID, allowing Apple to offer the same modes of interaction across both the iPhone X and the iPad Pro.

For now, we’re in a bit of a transition period. Apple took an unusually bold leap forward with the iPhone X and one of the repercussions of doing so is an inconsistent experience for users with multiple devices.

The gestures for the iPhone X are presumably the way Apple expects users to interact with iOS for the next 5 years or so and we’ll just have to wait for the rest of their product line to catch up.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy
07976260408