The Two Fundamental Mistakes in Usability Testing

There are two mistakes that organizations make when doing usability testing. And once those mistakes are made, two things are certain: either end users won’t get an application they can actually use or, if they do, it will cost more and take longer.

usability testing

Organizations often make a fundamental mistake by thinking of usability testing as “the development team bringing in some users to see how well they can use the application.” Usability testing isn’t like that. It’s not anything like that. It’s like telling a joke (and if you don’t believe me, here’s what a UX/UI designer who moonlights as a stand-up comedian has to say).

When a joke is told, there are two roles in play: The person telling the joke and the person hearing it. Similarly, there are two players in usability testing: the developers and the end users. And, when you tell a joke two things are true: (a) If you have to explain it, it wasn’t very good, and (b) The person telling the joke doesn’t get to decide if the joke was funny. If the development team has to explain how the UI works, it’s not a usable design; if the end user says the design is unusable, the development team doesn’t get to say “Well, yes it is.”

Out of that fundamental mistake, come two more mistakes that will (inevitably) cost the organization time and money.

First Mistake – Who Decides if a Usability Test was a Success

In usability testing, the end user is not the subject of the test. In usability testing, the end user is the person performing the tests – the user is the tester, not the testee. The end user, not the development team, is the key role in usability testing (just like the person hearing a joke has the key role in deciding if the joke is funny).

If you make the mistake of thinking that the development team has the key role, you fall into the first mistake: Thinking that it’s the development team who decides if a usability test was a success. This is, of course, nonsense: The tester (the end user) is the person who decides if a test succeeds (in this, if the application is usable).

I’m not suggesting that the development team doesn’t have many critical roles to play in usability testing. Before the test begins (as we discuss in Learning Tree’s User Experience design course), the development team is responsible for scheduling and setting up the environment for any usability test; the team is responsible for designing the tasks that users will be asked to perform the test; the team is also responsible for observing and recording the test as it is performed. Finally, after the test is complete, the development is responsible for determining what needs to be done (if anything) and for driving the process of looking for solutions, if problems were discovered.

But the development team doesn’t get to decide whether the UX is good, bad, or indifferent. That responsibility lies with the tester – the end user.

Second Mistake – Deferring Testing

Given how critical users are to the success of usability testing, it’s surprising to me how often development teams make the second mistake: deferring bringing in users to do usability testing. Time and again I’ve heard development teams say that they’re not ready to do usability testing – that they want to defer testing “Until we have something to show the users.” This is, again, part of the fundamental mistake: that “the development team brings in users.”

Deferring testing always leads to problems.

For example, I worked with a team in Switzerland who had built a quite attractive UI that consisted of a series of bubbles. When users clicked on those bubbles, the application displayed the information the users wanted. However, when the team brought users in to do some usability testing they discovered that users had no idea what to do when presented with the UI: To the users, the bubbles didn’t look like “clickable” objects.

I don’t want to suggest that the team was foolish – I was surprised by the user’s reaction also. However, that doesn’t matter: The fact is that the bubbles didn’t work for the end users and, therefore, the test was a failure: the application wasn’t usable.

Fortunately, I think that the problem was easily fixable. All that the team had to do was add a semi-transparent dialog box that would appear the first time a user interacted with the application. That dialog would contain a half-dozen words explaining how to use the UI (“Click on a bubble to see the related information,” would probably do the trick) and a “Got it! Don’t show this again” checkbox. So, with a little more work, the development team could extend the application with a design pattern that, I suspect, their users would recognize immediately.

I did suggest that they test this addition to their UI before they built it, though.

You Can’t Start Too Early

The key words in that story were “a little more work.” The application was now going to cost more and be delivered later because of the work required to fix this UX problem. A usability test would have revealed this problem…and the right time to do that test was as soon as the bubble-based UI was sketched out on a piece of paper. The wrong time to do it was after the team had invested time and resources into building the UI.

Because, in the end, users decide if the development team got the design right, and testing is critical to delivering the “right” design, the development team can’t wait until there’s something “for users to see.” The reverse is true: End users should be breaking down the team’s doors to get involved as soon as possible.

Users should be validating outputs of every step in the process, including the early steps. End users should be validating:

  • The descriptions of the personas who will interact with the application’s UIs
  • The development team’s understanding of users’ goals, mental models, and tasks
  • Potential UIs as soon as there is a sketch to look at

The Punchline

About the only thing that end users can’t be involved with is developing the tasks that will be used during usability testing. When the development team turns that responsibility over to the end users then, effectively end users are being told what the right answer to the usability test looks like. It’s sort of the equivalent of providing a laugh track to your own jokes: Yes, you’re getting a laugh (the tests are passing) but, somehow, it feels like you’re cheating.

To quote the great comedian George Jessel, with UX design, the best solution is “to give ’em what they want”…or, in this case, to give your users what they can actually use. And the easiest way to find out “what users want” is to get them involved as early as possible.

Type to search

Do you mean "" ?

Sorry, no results were found for your query.

Please check your spelling and try your search again.