Recession Testing is the new RegressionTesting

Posted by on October 4, 2011 16 comments

Its time to retire the idea of Regression Testing folks. Regression testing, at least the way its being performed today is typically a value free, wasteful exercise and falls into the category of  ”bad testing”.

“Regression testing is any type of software testing that seeks to uncover software errors by partially retesting a modified program.” 

In fairness to Regression Testing. I’m not opposed to the above ideology(except for the partially retesting bit, thats stupid).  I think it has some merit. The concept that modifications in code add risk which testing needs to address is a sound idea and worth taking into account while testing.

But we testers know, that this intent turns out to a different beast.

What gets called ‘Regression testing’ in many companies in my view is not very valuable and has a different intention.  Regression testing ends up a packaged set of tests ( I wince to call them that, as they typically have the same idea repeated over and over) that get repeated at the end of testing to validate that nothing’s broken.

It’s more about maintaing the status quo than any in-depth testing. This to me seems wasteful.

It seems wasteful to me to repeat the same tests giving an illusion of repeatability when in reality we know that each test can never be exactly the same and that getting the same result in a test does not mean for certain that the test has passed.

It seems wasteful to me to exercise a feature using the same testing idea again and again when a different test idea might offer new information about the system

It seems wasteful to me to ask testers to perform tasks that result in the tester disengaging from the activity because its boring.

It seems wasteful to suggest that somehow new features need to be tested differently to older features. Why? Bugs are not ageist.

It seems wasteful to suggest that somehow regression testing demands less cognitive and skeptical thinking.

I think we’re looking at this problem the wrong way. I’d like to suggest a different paradigm.

I’d like to offer up the idea of Recession Testing. Where Regression Testing does its best to test as little as possible, Recession testing insists of focusing on value and removing waste, just like we need to do in a Recession in order to be competitive.

This means, instead of splitting a product into new and existing features, lets test all parts of a product with equal aggression, equal skepticism. When it comes to regression vs new feature bugs, lets make all bugs equal, not some more equal than others!

If we do need to prioritise our work, lets do so on the basis of risk, What are the impact of change on the feature?  What is the importance of the feature being tested?

If you’re going to test a feature, do everyone a favour and test it like you mean it! Don’t give it some half baked, wishy washy run over, to check that its ok and then give the false assumption that you’ve properly tested it. Put the feature through its paces. The feature deserves your respect and  more importantly you deserve to test in a cognitively challenging way.

You know it makes sense!

I wrote this post for Kim, one of my amazing testers who wanted to find out more about regression testing.

I’m sure there are many posts on this topic (if you know of some good ones, do us a favour and add a link in the comments).

 

 

Crossing the bridge over no-man’s land

Posted by on September 22, 2011 6 comments

Indiana Jones in the Temple of Doom  has a “moment of indecision” when half way across a bridge he realises he’s been cornered by Mola Ram(the baddy) and his henchmen.

He looks back and there’s a hoard  of lusty savages baying for his blood, no luck there. He looks ahead only to find an equal challenge ahead. What is poor Indy to do?

Anyone who has introduced change into a corporate environment will empathize with his situation. You know the decision to break away from the past is the right one, you run eagerly and embrace change, only to find half way through the journey, your path to success becomes blocked.

I’ve been working with my test team for a while now moving from a process driven approach to a more of an Exploratory one.

Its not been without its challenges. Some concepts I’ve introduced have been welcomed warmly but the reception to others has been a little icy. In particular I’ve tried to move the team away from a test case management system. This was met with real concern and there was quite a resistance to the idea.

This troubled me as while I understood their concerns, I knew the system was limiting the generation of new testing ideas.

But how could I overcome this resistance? And really was it worth it? Perhaps the changes I had already made would be enough? The company was already more than impressed with the changes I had made so far.

I felt like Indy at the foot of rope bridge, how the hell was I going to solve this one?

So I stood at my crossroads and dithered. Oh God, did I dither. I ummhed and ahhed and pondered what to do. . But worse, I  knew my indecision was making the situation worse, and that the more I dithered, the harder it would be to rid ourselves of the dust bag of tired and well worn ideas.

Indy at this point, decides his only move is to cut the bridge leaving everyone to hang on for their lives.

Fortunately, unlike Indy I had a reliable and trustworthy sidekick.  Together, we setup a task force within the team to attack the problem. After some discussion we decided our approach needed four cornerstones. They were:

1) Creativity.

However we tested, our approach needed to enable us to foster and encourage creativity. With creativity comes new ideas, new models, new tests and so discover new bugs.

We’re covering this one with a number of approaches. One is to improve tester skill through practice and coaching. I’ve also created a folder of ideas for people to draw upon to help trigger new ideas.

2) Visibility

We wanted to be able to provide reporting on any testing we do. The reporting has to be simple yet with sufficient detail to ensure that our stakeholders understand what we have tested and why.

We have our trusty whiteboard which mostly hits the spot. We need to be able to pull up our actual testing including results in an easy to manage way. We’re looking into BBExplorer to handle that.

We will also track any essential test results on wiki in the form of a test report at the end of each iteration.

3) Coverage

We wanted to have some way of ensuring that key functionality/key features are always tested.

We most likely will rely on our test case management system for this, but we’re cleaning out all the dead wood and making the tests lighter and less authoritative.

4) Knowledge

We wanted to create a knowledge base. Our system is complex and it requires in-depth knowledge to test some areas. We want to store that information and knowledge. We also have a serious amount of test data we want everyone to be able to access, modify and improve.

We’ll use our internal wiki for this.

What I really like about what’s happened here is that the team came up with a solution to solve the problem. It’s a team decision which has got to mean easier implementation.

I think a couple of really powerful things have come out of this. I’m listing them here:

1) Change can be scary. Not changing is worse. Get on with it.

2) Use people around you to help bring about change.

3) Never lose site of your goal. This reminds me of Scott Barber’s email signature: “”If you can see it in your mind…you will find it in your life.”

I feel good. I hope my team does too. We faced a challenge. We examined it, questioned it and overcame it and we’ve all come out sharper, enlightened and positive about the changes ahead.

Now that’s what Exploratory Testing is all about.

 

On conferences and insomnia

Posted by on September 3, 2011 1 comment

For me, the sign of a good conference is insomnia the night after the conference.

Its as if my brain is unable to let go the new ideas and discussions I’ve had with other testers. Ideas that haven’t had an opportunity to be digested and reflected upon and usually around 2am after the close of a good conference my eyes snap open, my brain alive and alert ready for action.

Ideas and discussions from the day merge and meld into a boiling cauldron of fizzling synapse and bubbling endorphins and, as much as I try to breath deeply relax and let it all go, I know deep down its all pointless.

I’m going to have to get up and write down my thoughts and ideas.

STANZ Melbourne is one such conference and its given be a double dose of insomnia resulting in frenetic writing at 12, 2 and 4 am until finally my brain exhausted became compliant and allowed my poor weary body to sleep.

STANZ is sponsored and hosted by SOFTED. These folks at SOFTED really understand and ‘get’ software testing and it shows.

As well as  hosting this conference and getting some pretty impressive speakers in(I urge anyone who has the opportunity to hear Goranka Bjedov speak to do so) they also sponsor the Sydney Testers Meetup by supplying thirsty and hungry testers with drinks and nibbles at networking events.

Whats more they host peer workshops. The last one they hosted in New Zealand which was a huge success so much so that next year they hope to host one in Sydney.

Watch this space.

For me, STANZ gave me two core learnings.

The first was Goranka’s talk about the future of quality. I think this was really insightful and gave me much food for thought. The concept that quality is dead and that as testers we need to reflect how this will impact us. I’m not sure yet what this means for me(I need some more 4 am thinking on that one!) but  somewhere deep down, this struck a real chord.

The second re-enforced to me the power of sharing problems and getting ideas from your network of testers. In 30 minutes, Trish Khoo had a plethora of new ideas and suggestions for me to take away. Many thanks Trish.

Now off to order a double shot espresso….