Jakob has a few fantastic insights into this domain:
The most striking truth of the curve is that zero users give zero insights.
This should be blindingly obvious, but it’s actually a very valuable insight. You, as the developer of a piece of software, will never be able to figure out all of the usability problems with your software. You are simply too close to it to find everything. You know about the workarounds, the esoteric keyboard shortcuts, and exactly how everything works under the covers. My mom, on the other hand, understands none of this. She doesn’t understand the difference between a queue and a stack or quicksort and bubble sort, let alone when to double-click or single-click on UI elements in the software she uses. I love her to death, but she’s computer illiterate.
Of course, interestingly, my mom’s absolute lack of knowledge about how software works has always made her a particularly good research subject for informal usability tests that I have conducted. When I need to make sure that a feature in iRooster is obvious enough, I ask her to download the latest copy and give it a shot. If she can figure out how to use it, I am usually set. Of course, I do ask a few more people to give it a shot, just to make sure I’m doing the right thing. When I do this, though, I always cap myself off around five people, because:
After the fifth user, you are wasting your time by observing the same findings repeatedly but not learning much new.
It may seem shocking, but it’s true in the majority of cases. Save yourself time and energy by not asking everyone in the world to test your software, and by not conducting multi-week usability studies. Grab the guy in the office next door and ask him to test out your new feature.
Usability studies don’t have to be overwhelming, or the sole domain of people with Ph.D. appended to their names. You can conduct usability studies, and it really can be as simple as saying “Hey Susan, I want you to spend five minutes trying to do ‘X,’” where ‘X’ is your newest supported scenario.