That coverage problem

Thanks mostly to the BBST classes the AST run, I’ve come to understand and appreciate the impossibility of complete testing and the challenges surrounding ‘complete test coverage’.

I find 100% testing coverage a pretty meaningless phrase. Imagine trying the calculate the amount of water in a swimming pool by multiplying the length and width of the pool. Ha! we’d snort and sniff at the stupidity of the idea. But when test managers, project managers and testers ask for “complete test coverage” that is in essence what they are asking testers to do.

In fact, getting to grips on complete testing is even harder than the pool problem, because at least with a pool, once you know the width, length and depth, the volume can be calculated.

In testing though,the number of models we use to design our tests is limited only by our imagination. To try and measure test coverage is a bit like trying to cover a black hole with a tea towel and say you’ve measured it.

Trying to test like this in a short space of time is really stressful and it seems the agile solution is to automate regression tests. But really, is this the best solution we can come up with?

It seems to me this desire to cover as much functionality as possible ends up in this Wac-A-mole game with testers frantically hitting features as fast as possible in order to be able to say its done.

Well, I’m trying a thought experiment at the company I’m consulting at. I’m challenging everyone to drop the idea of coverage and instead focus on some meaningful and valuable testing.

I’m doing this because it seems to me, we testers are far too hung up on this coverage problem and we bias our testing to try and solve it. Yes, this may mean that we fail to test all the functionality – quelle horreur! But get this, when do we *ever* test all functionality?

Dropping the desire to “test everything” is very liberating. We no longer have to fret about trying to test all the functionality in a week (we do weekly releases with multiple hot fix releases). Instead, it’s freed our minds up to reflect and ponder on what is the most beneficial testing to perform given we only have a few days to test. It’s also freed up our toolsmith, allowing them to spend time solving some testing problems through testabilty instead of frantically creating regression tests.

I’m fully expecting some bugs will get released to production, but you know? that’s nothing new! We’re finding bugs in production all the time, they’re called customer support issues.

Time will tell if this approach proves to be beneficial or not. It’s a little scary, dropping the regression test safety net, but when safety nets start obstructing your work its time to question its validity.

4 Comments

  1. Anne-Marie,
    Agreed. Test Coverage has become a Unicorn in a sense. The idea of 100% coverage is just not realistic. What we have to focus on is the ‘risk’ areas and impacts associated to them. Automation will never give 100% coverage either. That is another Unicorn in its own rights.

    I believe Risk Based Testing is propbably the best shot we have, but even then it has holes in it (thus production defects/issues). Nothing is perfect. We can strive to do our best and be as thorough as possible, but we can’t know it all.

    Great post. Thanks.

    Jim

    Reply

  2. God… I wish I worked for you!

    One of the bigger challenges I’ve faced is convincing people that testing 100% of the /requirements/ doesn’t mean that we’ve tested 100% of the product. It’s surprisingly difficult considering how logical it is.

    I’d love to hear how this goes AM. Good stuff. :0)

    Reply

  3. Wow, this is… great, confusing and a little scary. All wrapped up into one.

    Coverage is one of “those things” you tend to use because… well, it makes sense. But you’re also talking a lot of sense as well.

    Thanks for the great blog post, you’ve given us a lot to think about…

    Reply

    1. Coverage is only one model, its an important one but its not the only one and its good to introduce other models. If we keep testing with coverage we are missing out on lots of other dimensions to our testing.

      I like to think of it as scary in the way bungee jumping is scary, once you let go, its incredibly liberating. I read up on your blog, its a good start, I’m glad you liked the Foundation course given by the BBST.

      Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>