Return to site

Testing is Not a Strategy

How many iterations should I test before I launch?

I really hope this is not your plan.

It's not all bad. Much of it is worse.

That sign up there has been cropping up in a lot of places. It's a quote from Herb Kelleher, the founder of Southwest Airlines, and like most inspirational quotes, it tends to get misused. When I see it hanging up in my coworking community, I know from my experience what it means, when it's hanging there. When I see it hanging in the offices of a multi-national conglomerate or agency, I know it's there for decoration. You could replace it with this one, and absolutely no one would notice.

Iteration is the key to successfully looking busy

I am thrilled to see that rapid prototyping, frequent customer interaction, lightweight usability testing and other central tenets of User-Centered Design have made their way into the mainstream. Design Thinking, Agile development, Lean Startup/Enterprise have all adopted elements of a very well-established and time-tested strategy of creating products which solve real problems.

And all, to one degree or another, seem to embrace the idea that it's "better to do it than to talk about it." Which is usually awesome: get shit out there, see if it works, refine, release, repeat.

This is also a fantastic way to waste a lot of time while seeming cutting-edge.

How many times do you need to try this to see what will happen?

Movement is not the same as progress

Recently I was asked, "how can I recruit participants for my usability testing?" Which is a perfectly reasonable question, and I often point people to this answer.

But as we got into a deeper discussion, what emerged was a slightly different problem: testing exhaustion. This person was running so many usability tests, so frequently, in the name of "iterating and shipping", that she was exhausting her limited pool of test participants.

Now, the problem might simply be solved by recruiting more people. But that can be expensive, time-consuming and may produce diminishing returns: more participants means more people to coordinate, compensate and communicate with.

Alternatively, the problem may be that she was testing more often than was useful. This is an embarrassment of riches; many teams need to beg, borrow and steal just to get permission to test once or twice before launch.

Is it possible to test too much?

Yes, it is possible to test too much

Every time you "go to the well" to get feedback from customers in a focused, purposeful way, you should begin with this question: "what do I expect to learn?"

In other words, you should have an "hypothesis."

Here are some examples:

  • This design works better at connecting people to information they want than the one we already have
  • This new design solves a specific problem customers have
  • Our competitor is providing a better service than we are with this specific feature.

The key element of each of these hypotheses is disprovability: You can tell, from the data you collect in your testing, whether or not you are right or wrong in your assumptions.

This becomes even more helpful when you have a coherent product l based on a Customer Journey Map.

 Want More of These? Enter Your Email Address and hit "Subscribe".

 

All Posts
×

Almost done…

We just sent you an email. Please click the link in the email to confirm your subscription!

OK