50 Beta Testers. Not enough, or better than most?

Two software testers talked about the Joe Stump story this week.  One was Elizabeth Hendrickson and the other UTest.  The two attitudes to the same story are very different.

Let me compare two of the views:

First Elizabeth Hendrickson
“Problem #2: Thinking that “50 Beta Testers” and “200 Unit Tests” Constitutes Exhaustive Testing

Having beta testers and unit tests is a good and groovy thing. But it’s not sufficient, as this story shows. What appears to be missing is any kind of rigorous end-to-end testing.”

……….

“So even the most cursory Exploratory Testing by someone with testing skill would have been likely to reveal the problem.”

Now UTest

“Not to lay blame at Stemp’s feet.  We, of all people, know that bugs happen.  Plus he mentions utilizing 50 beta testers and 200 unit tests, so they’re doing more testing than many.  But IF these showstopper bugs had been caught in the initial version or even in the 2nd version of Chess Wars, then Chess Wars wouldn’t have its nose pressed up against the window of the App Store, waiting for the powers-that-be to bless the new version.”

Whilst both blogs encourage more testing, I think I prefer Elizabeth’s attitude to the whole thing. It’s hard to know if 50 beta testers is sufficient or not without the context (last phrase included to avoid references to Unicorn Question in any comments..you know who you are…). Regardless, the attitude that  if you ‘do more testing than many’ somehow vindicates the fact that showstopper bugs were found is I think a misguided one.

As Elizabeth rightly pointed out in her post, good exploratory testing would have picked up this bug and any good tester knows to test for these things.

I hark back to my post on other approaches to testing for Startups. There are better ways to test! Lets not fall into the pitfall of accepting a lower level of testing as sufficient. Just because Startups use Beta Testing as the preferred modus operandi , doesn’t necessarily make it a good approach.

Better than Beta? You Betcha!

Software testing has gone through a few renaissances in the last twenty years or so. From the dark ages of the waterfall process, there has emerged new theories, schools and test tools. We even have online clubs and networks.

One noticeable difference between then and now is  testing was traditionally part of development. Now, most larger companies recognise the need for some form of testing team.

So don’t you think it’s strange when it comes to software testing, startups still remain in the dark ages of developers testing their own code and then skipping straight to a beta test? The rationale is that it’s an easier, faster, cheaper approach than employing a software tester.

I ashamed to say that until recently I bought into this misconception. Its only when I started doing some research into beta testing, that I discovered the amount of work and effort it took to. So I’m doing a Mythbuster session.

Myth 1: Beta Testing is easy

Beta Testing is hard work. Like any major task it needs planning and effort. Successful Beta Testing takes lots of planning and lots of effort. Upfront analysis into the number of Beta testers required and how they will be sourced.  How much time is necessary for the Beta Testing,  how many releases will you have (yes, you need more than one!).  How will Beta testers sign up and will their feedback need to be secure? How will you keep track of the bugs found, and how much time will you need to support the effort?  There is no point having feedback from 200 Beta testers but not having the time to analyse the information, let alone fix the problems. What tools will you need to keep track of the Beta test?  These are just some questions that will need to be answered before you can begin your beta testing phase.

A couple developers talk about their experiences of Beta Testing..Joel Spolsky – founder of FogCreek Software and SyneRyder Journal

Myth 2: Beta Testing is Quick

The length of time beta testing takes depends on the complexity of the software being tested and the number of releases you plan to have. Typical estimates range from four to ten weeks depending it seems on your experience of Beta Testing.  One constant that remains, is that beta testing always takes twice as long as you estimate. That’s because one of the difficulties with Beta Testing is your dependence on people who don’t owe you anything and in that sense are under no obligation to meet any of your deadlines.

Myth 3: Beta Testing is Free

The greatest myth of all.   The man hours spent in planning, setting up, monitoring makes beta testing a time consuming task. Time far better used by a developer in turning the software into a great product. The end result is your developer is distracted and performing tasks that many software testers can perform quicker and cheaper. On top of that, testers know how to test software really well, something you can never guarantee from a Beta tester.

Software Testers do it better

Software testing has really matured in recent years, and there are many ways to provide quality testing that meet the unique demands that startups face. Software testing does not necessarily equate with streams of documents and restrictive process. You only have to look at techniques such as Rapid Software Testing by James Bach and open source tools to know that there are many alternatives to traditional software testing. There are so many good software testers out there, available for hire on a freelance basis, it makes these age old myths about software testing and startups, a bit passed their use by date.

It’s time for startups to do themselves a favour. Hire a software tester, even for a couple of days. It’s worth every cent…