That coverage problem

Thanks mostly to the BBST classes the AST run, I’ve come to understand and appreciate the impossibility of complete testing and the challenges surrounding ‘complete test coverage’.

I find 100% testing coverage a pretty meaningless phrase. Imagine trying the calculate the amount of water in a swimming pool by multiplying the length and width of the pool. Ha! we’d snort and sniff at the stupidity of the idea. But when test managers, project managers and testers ask for “complete test coverage” that is in essence what they are asking testers to do.

In fact, getting to grips on complete testing is even harder than the pool problem, because at least with a pool, once you know the width, length and depth, the volume can be calculated.

In testing though,the number of models we use to design our tests is limited only by our imagination. To try and measure test coverage is a bit like trying to cover a black hole with a tea towel and say you’ve measured it.

Trying to test like this in a short space of time is really stressful and it seems the agile solution is to automate regression tests. But really, is this the best solution we can come up with?

It seems to me this desire to cover as much functionality as possible ends up in this Wac-A-mole game with testers frantically hitting features as fast as possible in order to be able to say its done.

Well, I’m trying a thought experiment at the company I’m consulting at. I’m challenging everyone to drop the idea of coverage and instead focus on some meaningful and valuable testing.

I’m doing this because it seems to me, we testers are far too hung up on this coverage problem and we bias our testing to try and solve it. Yes, this may mean that we fail to test all the functionality – quelle horreur! But get this, when do we *ever* test all functionality?

Dropping the desire to “test everything” is very liberating. We no longer have to fret about trying to test all the functionality in a week (we do weekly releases with multiple hot fix releases). Instead, it’s freed our minds up to reflect and ponder on what is the most beneficial testing to perform given we only have a few days to test. It’s also freed up our toolsmith, allowing them to spend time solving some testing problems through testabilty instead of frantically creating regression tests.

I’m fully expecting some bugs will get released to production, but you know? that’s nothing new! We’re finding bugs in production all the time, they’re called customer support issues.

Time will tell if this approach proves to be beneficial or not. It’s a little scary, dropping the regression test safety net, but when safety nets start obstructing your work its time to question its validity.

Recession Testing is the new RegressionTesting

Its time to retire the idea of Regression Testing folks. Regression testing, at least the way its being performed today is typically a value free, wasteful exercise and falls into the category of  “bad testing”.

“Regression testing is any type of software testing that seeks to uncover software errors by partially retesting a modified program.” 

In fairness to Regression Testing. I’m not opposed to the above ideology(except for the partially retesting bit, thats stupid).  I think it has some merit. The concept that modifications in code add risk which testing needs to address is a sound idea and worth taking into account while testing.

But we testers know, that this intent turns out to a different beast.

What gets called ‘Regression testing’ in many companies in my view is not very valuable and has a different intention.  Regression testing ends up a packaged set of tests ( I wince to call them that, as they typically have the same idea repeated over and over) that get repeated at the end of testing to validate that nothing’s broken.

It’s more about maintaing the status quo than any in-depth testing. This to me seems wasteful.

It seems wasteful to me to repeat the same tests giving an illusion of repeatability when in reality we know that each test can never be exactly the same and that getting the same result in a test does not mean for certain that the test has passed.

It seems wasteful to me to exercise a feature using the same testing idea again and again when a different test idea might offer new information about the system

It seems wasteful to me to ask testers to perform tasks that result in the tester disengaging from the activity because its boring.

It seems wasteful to suggest that somehow new features need to be tested differently to older features. Why? Bugs are not ageist.

It seems wasteful to suggest that somehow regression testing demands less cognitive and skeptical thinking.

I think we’re looking at this problem the wrong way. I’d like to suggest a different paradigm.

I’d like to offer up the idea of Recession Testing. Where Regression Testing does its best to test as little as possible, Recession testing insists of focusing on value and removing waste, just like we need to do in a Recession in order to be competitive.

This means, instead of splitting a product into new and existing features, lets test all parts of a product with equal aggression, equal skepticism. When it comes to regression vs new feature bugs, lets make all bugs equal, not some more equal than others!

If we do need to prioritise our work, lets do so on the basis of risk, What are the impact of change on the feature?  What is the importance of the feature being tested?

If you’re going to test a feature, do everyone a favour and test it like you mean it! Don’t give it some half baked, wishy washy run over, to check that its ok and then give the false assumption that you’ve properly tested it. Put the feature through its paces. The feature deserves your respect and  more importantly you deserve to test in a cognitively challenging way.

You know it makes sense!

I wrote this post for Kim, one of my amazing testers who wanted to find out more about regression testing.

I’m sure there are many posts on this topic (if you know of some good ones, do us a favour and add a link in the comments).



Software Test Templates – scoping out your tests

I wrote a blog recently “the value of a software testing process” in which I questioned the traditional benefits of software test scripts based on the re-use and repeatability theory.

My approach to any rapid change software development is to make a some basic assumptions

1) The code is going to change quickly
2) The person who coded won’t necessarily be there for the next release
3) The person who tested the last time, won’t necessarily be there for the next release
4) Test documentation helps to provide structure and is excellent as a guide to ensure adequate coverage
5) Testers are able to think outside the square and aslo be able to fill in the gaps in a test

I use one spreadsheet (if possible) to track the following information;

a) test script number
b) test link (if available which is a reference to requirements etc)
c) test purpose – a clear concise description of what I am planning to test
d) test results – Pass or Fail
e) defect number – I assign a number of use the defect tracking system
f) test estimate – the time I anticipate it will take to test this functionality

I use the document to first scope out what I want to test, a quick and dirty overview of what I am planning. I also use it to estimate how long testing is going to take. This is helpful if I am ever asked to substantiate my estimates.

Once all stakeholders are in agreement onto the scope, I create a concise and descriptive purpose of each test. This is in essence my test script. I use the same spreadsheet to enter results, defect numbers and to calculate simple metrics such as the pass rate.

I have uploaded an example below;

Software Testing Template

I hope you you find this post useful. Please feel free to use the ideas I have here. Do us a favour though, leave a comment, or digg the post. Thanks !!

The value of a software testing process

Does a software testing process provide a best approach to testing? Coming from a background in engineering I have always been a firm believer in the benefits of a sold and dependable testing process which includes test planning, design, building, execution and reporting. So, how come I find it hard to convince others of its merits? After much soul searching, I’ve come down to the conclusion and it’s this. A rigid and formal testing process doesn’t work in many of today’s software projects. For example, some of the benefits of a software testing process are to provide
re-usability and repeatability of results Re-use is cited as one of the benefits of a software testing process as it reduces long term cost both in resources and time. For example, test scripts once written can be used again in future releases reducing the upfront resources.
But, in my experience, I’ve rarely used the same test script twice. This is because each time a new release is planned, its markedly different to the previous one. Add into the mix, new developers, new testers and soon the amount of flux necessitates re-starting the test planning, design and building from first principles. I question the value of re-use when it ends up costing similar or more in time and resources.
Another benefit of a solid testing process is the repeatability of the results. Again though, if your code change is that significant, you will have to question the validity of the expected result. If resources are required to confirm or update the expected results, I’d have to argue time may be better spent in beginning again. This is especially true if the software tester is new to the project and requires analysis time to fully understand a product and its environment.
In summary,
I believe that where rapid change exists, you need an adaptable approach and is lighter on documentation
In industries with little change there is much benefit to a repeatable, re-usable approach, but where rapid change exists, I think spending lots of energy on paper documents which in the end will only get used once is a waste of time and cost.
Testing is essential, just that the process to perform testing must always be up for review and change too!
my two (Aussie) cents for the day….