Pre Software Testing Checklist for Startups

Most Startups will not use large amounts of documentation unless it adds value. In an environment where the one thing you can rely on is change, sometimes it doesn’t pay to put too much detail down in formal software test plans. That’s why I am in such a fan of checklists. A software test checklist is like a cheatsheet for testers. It’s a list of reminders to make sure you’ve covered all the basics. Nothing fancy, I prefer a one sheet, but I’ve seen checklists two or three pages long.

Lots of people in software testing loves checklists. It’s our way of ensuring we don’t forget stuff, well at least that’s why I use them. They work really well with Startups who either don’t have the time, need or inclination to load themselves down with lots of methodologies and process.

The first checklist that I am in favour of handing to my Startup clients is the Pre Assignment checklist. My Pre Assignment Software Testing Checklist provides Startups with a list of essential to do items. Most of my Startup work is short assignments between 3 ~ 5 days as time and budget are limited. So, if they can’t supply these things before I start, I politely suggest that they’re wasting my time and their money.

Proviso:
The assumption in this checklist is that I’m software testing on-site and the assignment is a short term one. Otherwise some of this stuff I would be responsible for. For long term assignments part of these may be written into the software testing assignment.

So here it is in its full glory with added explanations for clarity.

Pre Assignment Software Testing Checklist

Software Test Goals

  • What do you want to get out of the software test effort?

Confidence, Knowledge? Software that works better? The sheer nature of software testing means that a software tester can provide you lots of  interesting information apart from bugs found.

For example, if it’s knowledge,  A software tester can collect lots of data on performance and reliability which may be helpful in marketing your product.

SoftwareTest Reporting

  • Who is going to see this report?

A good software tester can write the same information differently depending on who is going to read the report. For example, you may want a report written in word format for business purposes. If its a developer, then perhaps they have a favourite defect tracking tool they want to use.

  • What sort of reporting do you want?

How often do you want a software tester to update you? Daily, Weekly or just at the end?

Test Strategy

  • Do you have a long term software testing strategy in mind?
    Perhaps you wish to automate your testing in the future, or you want to introduce a process, if you let the software tester know up front, they can perhaps include some structure for you to keep once the software testing is complete.

Software Test environment

  • Determine the Software Test Environment
    Has a decision been made on the system requirements? Software Testing ought to use this as the default test environment. Make some decisions on the following:

    • Operating System
    • Browser
    • Printer
    • Machine
    • Any other supporting software

Don’t forget to decide on a version number for the software

  • Assign Machine
    The developer’s machine won’t do, sorry. You might have a machine in mind, but have:

    • Started it up recently?
    • Made sure no-one else is using it?
    • Does it require updating?

My point is don’t assume that the box you have earmarked is available.

  • Setup Software Test Environment
    I know it works fine on your machine, but its worth ensuring it installs on the test machine.

    • Install all the support software the system needs
    • Install the correct version of the support software
    • DO NOT install the product to be tested if installation and configuration is to be tested
  • Test the Setup
    Start up the software that has been installed to make sure it works properly

Finally and maybe most importantly!

Software Test Support

Ensure:

  • support by development is available to support the tester
    Don’t give the developer a two week holiday for working so hard to complete the software on time. He still needs to support the tester when bugs are found
  • IT support is available if necessary
    If your luck enough to have an IT team, make them aware of the software testing work that is going on and ensure someone is there to help if necessary

This type of work is often overlooked in the eagerness to get the testing started. If you want you’re software testing to start on time and keeping within budget than completing this software testing checklist is essential.

Here’s a link to a scaled down version. Pre-Assignment Software Testing checklist. Feel free to download it, but please leave the reference to Testing Times in either the header or footer.

50 Beta Testers. Not enough, or better than most?

Two software testers talked about the Joe Stump story this week.  One was Elizabeth Hendrickson and the other UTest.  The two attitudes to the same story are very different.

Let me compare two of the views:

First Elizabeth Hendrickson
“Problem #2: Thinking that “50 Beta Testers” and “200 Unit Tests” Constitutes Exhaustive Testing

Having beta testers and unit tests is a good and groovy thing. But it’s not sufficient, as this story shows. What appears to be missing is any kind of rigorous end-to-end testing.”

……….

“So even the most cursory Exploratory Testing by someone with testing skill would have been likely to reveal the problem.”

Now UTest

“Not to lay blame at Stemp’s feet.  We, of all people, know that bugs happen.  Plus he mentions utilizing 50 beta testers and 200 unit tests, so they’re doing more testing than many.  But IF these showstopper bugs had been caught in the initial version or even in the 2nd version of Chess Wars, then Chess Wars wouldn’t have its nose pressed up against the window of the App Store, waiting for the powers-that-be to bless the new version.”

Whilst both blogs encourage more testing, I think I prefer Elizabeth’s attitude to the whole thing. It’s hard to know if 50 beta testers is sufficient or not without the context (last phrase included to avoid references to Unicorn Question in any comments..you know who you are…). Regardless, the attitude that  if you ‘do more testing than many’ somehow vindicates the fact that showstopper bugs were found is I think a misguided one.

As Elizabeth rightly pointed out in her post, good exploratory testing would have picked up this bug and any good tester knows to test for these things.

I hark back to my post on other approaches to testing for Startups. There are better ways to test! Lets not fall into the pitfall of accepting a lower level of testing as sufficient. Just because Startups use Beta Testing as the preferred modus operandi , doesn’t necessarily make it a good approach.

Black and White and Red all over….

There’s an add on telly about this guy who writes his book on a remington typewriter and then the post office sends it and suddenly its in the window of a bookshop. It think the ad is for the Irish Post Office.

Regardless, there is a moment when the author looks into the bookshop window and glows with satisfaction and pride on the achievement.

That’s a bit how a feel like today.

I just have had my first article published in T.E.S.T magazine and I’m on the front cover no less. You can read it here

It’s about one of my favourite topics, software testing and startups.

It’s these types of achievements along with getting work, that keep me working as a independent test consultant because, working on your own can be really tough! It’s a constant act of self promotion, bookeeping, customer liason and oh, occassionaly I get to do some software testing.

There was a post on the software testing club recently on ‘what are the things to consider when outsourcing?” I think the things to be considered are those outside of your area of expertise. Software Testing is only a small percentage of what you will do in order to be successful at freelancing or working as an independent.

Anyhow, back to the article.

As I was reading it, I felt a mixture of pride, satisfaction and surprise. I didn’t realise I could write so well….maybe it’s to do with the fact its a published article, or maybe its because the self satisfaction is putting a rosy tint on the whole thing. I don’t know….I don’t really care.

Now leave me alone whilst I bask in my halo of self content…..

Software Testing & Startups Talk in Dublin and Belfast

If anyone is interested there will be a softtest event this month (May) in Dublin and Belfast. I am giving a talk on software testing and startups.

Here’s the abstract

Software Testing in the world of Start Ups

This presentation offers a glimpse into software testing in the world of startups. It looks at the constraints and benefits facing software testers in this unique environment. It examines what it takes to be a software tester in a startup.

It will also be a bit of a myth buster session, looking at some of the myths around testing in startups. It closes by asking are we as a software testing industry doing enough to help startups improve their software.

Dates

Belfast : 26/05/09 4.30 pm, Venue is the Radisson SAS. To register your attendance email Nicola.McManus at momentumni.org

Dublin: 27/05/09 4.30 pm, Venue is the Holiday Inn, Pearse Street. To register your attendance email Anna.Donegan at ibec.ie

Agenda

16:30 – 17:00 Registration

17:00 – 18:00 Professionalism in Testing, John McArdle

18:00 – 19:00 Software Testing in the World of Start Ups, Anne- Marie Charrett, Testing Times

19:00 Close

I think you might need to be a member of softtest which is free, go to the softtest website for more information

I lookforward to seeing you there!

Better than Beta? You Betcha!

Software testing has gone through a few renaissances in the last twenty years or so. From the dark ages of the waterfall process, there has emerged new theories, schools and test tools. We even have online clubs and networks.

One noticeable difference between then and now is  testing was traditionally part of development. Now, most larger companies recognise the need for some form of testing team.

So don’t you think it’s strange when it comes to software testing, startups still remain in the dark ages of developers testing their own code and then skipping straight to a beta test? The rationale is that it’s an easier, faster, cheaper approach than employing a software tester.

I ashamed to say that until recently I bought into this misconception. Its only when I started doing some research into beta testing, that I discovered the amount of work and effort it took to. So I’m doing a Mythbuster session.

Myth 1: Beta Testing is easy

Beta Testing is hard work. Like any major task it needs planning and effort. Successful Beta Testing takes lots of planning and lots of effort. Upfront analysis into the number of Beta testers required and how they will be sourced.  How much time is necessary for the Beta Testing,  how many releases will you have (yes, you need more than one!).  How will Beta testers sign up and will their feedback need to be secure? How will you keep track of the bugs found, and how much time will you need to support the effort?  There is no point having feedback from 200 Beta testers but not having the time to analyse the information, let alone fix the problems. What tools will you need to keep track of the Beta test?  These are just some questions that will need to be answered before you can begin your beta testing phase.

A couple developers talk about their experiences of Beta Testing..Joel Spolsky – founder of FogCreek Software and SyneRyder Journal

Myth 2: Beta Testing is Quick

The length of time beta testing takes depends on the complexity of the software being tested and the number of releases you plan to have. Typical estimates range from four to ten weeks depending it seems on your experience of Beta Testing.  One constant that remains, is that beta testing always takes twice as long as you estimate. That’s because one of the difficulties with Beta Testing is your dependence on people who don’t owe you anything and in that sense are under no obligation to meet any of your deadlines.

Myth 3: Beta Testing is Free

The greatest myth of all.   The man hours spent in planning, setting up, monitoring makes beta testing a time consuming task. Time far better used by a developer in turning the software into a great product. The end result is your developer is distracted and performing tasks that many software testers can perform quicker and cheaper. On top of that, testers know how to test software really well, something you can never guarantee from a Beta tester.

Software Testers do it better

Software testing has really matured in recent years, and there are many ways to provide quality testing that meet the unique demands that startups face. Software testing does not necessarily equate with streams of documents and restrictive process. You only have to look at techniques such as Rapid Software Testing by James Bach and open source tools to know that there are many alternatives to traditional software testing. There are so many good software testers out there, available for hire on a freelance basis, it makes these age old myths about software testing and startups, a bit passed their use by date.

It’s time for startups to do themselves a favour. Hire a software tester, even for a couple of days. It’s worth every cent…