Should you be user testing with existing customers or with new customers?

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

When it comes to user testing, you’re faced with two options:

1. Test with people who are already customers of your product or service
2. Test with people who have never used it before.

Either choice can have a profound effect on the insights you’re likely to obtain from user testing, so it’s important to make the right decision.

Testing New Users

Testing new users can be an incredibly insightful experience for revealing how people develop an understanding of your solution. Whilst your existing customers have already overcome any pitfalls in your current product, testing new users can identify issues related to customer attraction and retention that you never knew were there.

With on-boarding new customers being such a critical metric for many businesses, testing with users who are using your solution for the first time shows you where confusion, frustration or difficulties are being encountered. As a result, you discover where and how to increase acquisition rates by optimising the design of your onboarding experience.

Of course, for brand new solutions it is only possible to test with potential new users. This makes sense as initially all users will be new to the solution anyway and it’s essential to test their reaction to their first visit.

A key challenge associated with testing new users is working out how to recruit them for your research. You can’t simply dip into your current pool of customers – instead you have to consider the types of customers you’re looking to target and recruit people matching these demographics with no prior experience of your product.

Testing Existing Users

Sometimes the research requires participants to already have a good knowledge and understanding of your solution. For example, when it comes to developing new features, it’s often best to test your new design with current customers, as their prior knowledge of the solution will ensure the deepest insights are generated. If your solution is being completely redesigned, it’s also helpful to test the new designs with existing customers to ensure that you are not negatively affecting their experience and inadvertently removing or changing key features that they rely on.

Testing with existing users can be easier from a recruitment perspective as most organisations can get access to the contact details of their current customers. However, running the user testing sessions themselves with existing customers can sometimes be harder. There’s a risk that users will focus too heavily on superficial changes to the look and feel of the solution, rather than how easy it is to use. This is a natural response – particularly if they have been using the existing solution for a long time. It’s therefore important to properly facilitate the testing session to ensure users fully test the functionality of the solution, rather than just offer feedback on it’s appearance.

How we do it at Userfy

We have carried out user testing on solutions at all stages of the product lifecycle – from new concepts that are in the very early stages of design, through to well-established solutions.

We recently tested a well-established B2B website with target demographic users who had not experienced it before. This meant we could test how readily a customer can use the website on their first visit. On the other hand, we also recently tested a new feature for a financial website with existing customers, to test how well the new feature enhanced the current offer, how easy it was to understand and how much it was valued.

Importantly, with every project we do we think very carefully about who we should be testing. Whilst a lot of our time is spent planning our user testing sessions so that we can maximise the quality and value of feedback we are likely to obtain – this would be a fruitless exercise if we didn’t also carefully consider the exact types of users to test to ensure our findings are meaningful and representative.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy


iOS11 has an inconsistency problem

Warning: Missing argument 1 for __(), called in /home/userfy52/public_html/wp-content/themes/FoundationPress/library/entry-meta.php on line 11 and defined in /home/userfy52/public_html/wp-includes/l10n.php on line 249

Gestures are an integral part of touch-screen interfaces. By using ‘natural’ interactions such as tapping and swiping, a user can manipulate the interface directly to perform key tasks and operations.

Many common gestures are now second-nature to us. We know to swipe up or down to scroll a page, we know how to ‘long-press’ to reveal additional options such as select, copy and paste.

A critical part of gestures is their consistency. Gestures are useful when we consistently receive the same result when using them because this helps create an automatic reflex that we no longer even consciously think about.

However, when gestures are inconsistent, they become confusing and frustrating. Our automatic reflex may still be firing, but we’re halted in our tracks when the result we receive is suddenly unexpectedly different.

This frustration is occurring for users with multiple devices running iOS11. One of the key reasons behind this is an inconsistency in what the ‘swipe up’ gesture is used for.

What does swipe up do again…? 

If you have an iPhone 8 or an older iPhone swiping up reveals the ‘control centre’ for quickly changing key settings.

If you have an iPhone X, swiping up is now used to return to the homepage and to reveal the app switcher.

Finally, if you have an iPad Pro, swipe up is used to reveal the dock, or a slightly longer swipe up is used to reveal a combination page containing control centre and the app switcher.

Whilst swiping up on iPhone 8 and iPad Pro both allow you to view control centre through slightly different methods, with the iPhone X you now need to swipe down from the top right corner to access this.

People who use both an iPhone X and an iPad Pro currently have to learn and remember two different versions of the swipe up gesture dependent on the device they are using. This makes the gesture less of an automatic reflex, and more something they have to consciously remember and commit attention to, dependent on the device they are using at the time.



Why has this happened?

Several things have lead to this problem. Firstly, Apple drastically revamped iOS for the iPad Pro – introducing brand new ways of multitasking to allow users to use two or more applications at once.

Largely this was a fantastic step forward for the iPad, making it closer to a legitimate laptop replacement than ever. However, it did create a need for a whole new method of interaction, to allow users to access a new ‘dock’ of applications, much like on Mac OS. Swiping up on an iPad Pro now reveals this dock, allowing a user to touch and drag a second app onto the screen to be used alongside another.

With iPhone X, Apple ditched the home button. Face ID replaced Touch ID, allowing users to unlock their phone simply by looking at it. It’s a remarkable piece of innovation, but removing the home button created a need for a new way to return to the home screen. It made sense to keep this action towards the bottom of the screen, so Apple turned the swipe up into the new ‘go home’ gesture.

Why is this such a problem?

A golden rule of good user-centred design is consistency. A user should not have to guess if an action means the same thing in different situations, it should always remain consistent.

Whilst the swipe up gesture is consistent within the iPhone X, and is consistent within the iPad Pro, Apple is failing to account for the fact that many users own and use both devices. This inconsistency across devices removes the intuitive nature of the gesture and makes it far harder for a user to learn and use automatically.

Will Apple correct it? 

In time, yes. If early rumours are to be believed, the next iPad Pro will incorporate Face ID, allowing Apple to offer the same modes of interaction across both the iPhone X and the iPad Pro.

For now, we’re in a bit of a transition period. Apple took an unusually bold leap forward with the iPhone X and one of the repercussions of doing so is an inconsistent experience for users with multiple devices.

The gestures for the iPhone X are presumably the way Apple expects users to interact with iOS for the next 5 years or so and we’ll just have to wait for the rest of their product line to catch up.

Dr Sam Howard
Meet the author:
Co-Founder and Director of Research at Userfy