Recession Testing is the new RegressionTesting

Its time to retire the idea of Regression Testing folks. Regression testing, at least the way its being performed today is typically a value free, wasteful exercise and falls into the category of  “bad testing”.
“Regression testing is any type of software testing that seeks to uncover software errors by partially retesting a modified program.” 

In fairness to Regression Testing. I’m not opposed to the above ideology(except for the partially retesting bit, thats stupid).  I think it has some merit. The concept that modifications in code add risk which testing needs to address is a sound idea and worth taking into account while testing.

But we testers know, that this intent turns out to a different beast.

What gets called ‘Regression testing’ in many companies in my view is not very valuable and has a different intention.  Regression testing ends up a packaged set of tests ( I wince to call them that, as they typically have the same idea repeated over and over) that get repeated at the end of testing to validate that nothing’s broken.

It’s more about maintaing the status quo than any in-depth testing. This to me seems wasteful.

It seems wasteful to me to repeat the same tests giving an illusion of repeatability when in reality we know that each test can never be exactly the same and that getting the same result in a test does not mean for certain that the test has passed.

It seems wasteful to me to exercise a feature using the same testing idea again and again when a different test idea might offer new information about the system

It seems wasteful to me to ask testers to perform tasks that result in the tester disengaging from the activity because its boring.

It seems wasteful to suggest that somehow new features need to be tested differently to older features. Why? Bugs are not ageist.

It seems wasteful to suggest that somehow regression testing demands less cognitive and skeptical thinking.

I think we’re looking at this problem the wrong way. I’d like to suggest a different paradigm.

I’d like to offer up the idea of Recession Testing. Where Regression Testing does its best to test as little as possible, Recession testing insists of focusing on value and removing waste, just like we need to do in a Recession in order to be competitive.

This means, instead of splitting a product into new and existing features, lets test all parts of a product with equal aggression, equal skepticism. When it comes to regression vs new feature bugs, lets make all bugs equal, not some more equal than others!

If we do need to prioritise our work, lets do so on the basis of risk, What are the impact of change on the feature?  What is the importance of the feature being tested?

If you’re going to test a feature, do everyone a favour and test it like you mean it! Don’t give it some half baked, wishy washy run over, to check that its ok and then give the false assumption that you’ve properly tested it. Put the feature through its paces. The feature deserves your respect and  more importantly you deserve to test in a cognitively challenging way.

You know it makes sense!

I wrote this post for Kim, one of my amazing testers who wanted to find out more about regression testing.

I’m sure there are many posts on this topic (if you know of some good ones, do us a favour and add a link in the comments).



18 thoughts on “Recession Testing is the new RegressionTesting

  1. In situations like this I usually contrast ‘X’ testing to good ‘X’ testing., where the conscious thinking and consideration aspect is the differentiator.
    This (for me) is an effective way to wake up folk to any activity where the following are symptomatic
    – complacency
    – template-thinking
    Taylorist testing
    non-thinking testing

    Maybe new terminology works as a differentiator – maybe not, I’m not sure – but highlighting the ways it’s not done well is definitely good!

  2. Attempts to have everything at the same time: economic growth, social benefits, military expenses and other nice things, without any sacrifices result in large debts, financial crisis and recession.Ideas of equal aggression, equal skepticism and equal treatment for all bugs are inspiring. Unfortunately, they are close to the very political populism that leads to Recession.

    Anne-Marie: I’m not sure if I see the similarity. What exactly will be the downfall? All I can see is a well tested product? What is the harm in that?

  3. Gold Maverick!
    Drop the label anyway, its all ‘testing’! The important part is how you come up with the tests in the first place. What ideas drove you to want to test if the system has regressed or not?

    A message to the haters (if there are any), call them what you want… As long as they answer, or seek to answer, a valuable question about the product, then execute them.

    If you do need a label, then yeah… Recession is cool. ;0)

    Anne-Marie: Yeah I agree, no label would be best. Its just testing….

    1. I’ve always dealt with regression testing where most tests are automated.
      Anne-Marie’s Reply: In my view, this is checking not testing.

      Those which can’t be automated are run manually. But in the end, I put a scripted test (manual or automated) into the regression bucket whenever the area is high risk: critical app path, failed test cases and a random sample.

      But once a regression bucket from previous releases has been tested and passed, are failed test cases from previous releases still high risk? It’s kind of like a bolt of lightning falling twice on the same spot, isn’t it? I’d rather take a small sample of those.

      Anne-Marie’s Reply: Why not create a new set of tests? Why are old tests better than new tests?

      I don’t see the need to drop the label, though. If anything, name the thing smoke testing. And complement smoke testing with manual tests inspired by results from the automated “regression” suite.

      Anne-Marie’s Reply: I agree, you don’t need a label

  4. Hi Jose,
    thanks for posting a comment. I’m glad you brought up the topic of automation. I’ve embedded my reply:

    I know automation is a sensitive issue but aren’t Regression Tests the firsts to try to automate because they are repetitive and boring as you said?

    Anne-Marie: Automation is all fine and good if you want to check a product, but it fails to test software. See Michael Bolton’s essay on Checking vs Testing.

    I also agree that thorough testing is always the preferable way to test critical features of the product but the reality is that we often don’t have the time to do this on all the features and so we end up to only test thoroughly the ‘new’ features and leave the ‘old’ ones to regression testing.

    Anne-Marie: Having enough time to test is always a challenge. That’s why we you risk to help us focus our testing on essential testing. All I’m suggesting is you use risk as opposed to what has been tested previously to help us determine what to test.

    I think this is why Iosif Itkin makes the political analogy, because we may risk to accumulate testing debt on the SUT.

    Anne-Marie: We ought to not test aggressively because we may find too many bugs? There’s a simple solution to that: stop testing! Oh, no wait, that would make people uncomfortable and they might lose confidence in the product. Much better to perform some pseudo testing that gives people enough of a security blanket to be able to ship the product. I think not.

    1. With my last paragraph I’m just advocating that it’s better to run automated regression tests than to leave them out of the test plan even if one considers it checking instead of testing. And only for the case of features that don’t have a high enough priority to recession testing them.
      Anne-Marie: Thanks Jose. I would advocate running checks at the end of testing for the low priority component too. I think there is nothing wrong with that, as long as everyone is clear that is being done. Bear in mind though, automation has a cost too (maintenance etc) which sometimes is overlooked. Is the time spent maintaining those scripts (as opposed to testing the lower priority features really a saving?).

  5. Hi Kobi, thanks for taking the time to comment. You raise some interesting points.
    Nice post, but I can’t say I fully agree,
    Regression is there, since we assume the risk in stabilized feature is lower than the one of a new feature mainly since it’s unstable yet.
    (Past feature risks were mostly mitigated already, apart for newly affects which might result from newly added changes which are unrelated or indirectly / foreseeably unrelated)
    I haven’t seen any studies which claim that past features are as risky as new features, nor did I experience anything of that sort so far.

    Anne-Marie: What do you call risk? I use Gerry Weinberg’s definition: Its a combination of Impact of Failure & Likelihood of Failure. If the impact of failure is sufficiently large, then we ought to test it, not check it.

    With that said, I fully agree that one should be conscious of what and how he tests, and actually hate seeing people using tests they developed for first time testing of a feature, wasting efforts while testing the same feature with atomic test cases after it has been mostly stabilized.
    The context of these features has changed, and we do need to change our strategy regarding them.

    While developers mainly focus on new functionality, we as testers have an ever growing backlog of past functionality, slowing us down more and more from one release to the other.
    I do believe we can assume these are less risky than newly added features, though not all of them are at the same level of risk, and prioritization of these items should lead us to define the exact amount of focus each should receive.


  6. Hi Iosif, thanks for replying.
    Sorry for political analogy, it was caused by the reference to recession times.

    that’s ok, it was kinda funny!

    Nobody objects that risk based testing is the right thing to do. However, risk is a product of probability and impact. Issues are more likely to happen in the new functionality.

    If we are following risk based approach, we should allocate more resources to test the new functionality. Regression is a good way of checking believes regarding high impact areas.

    Anne-Marie: As a rule, I’m not in favour of checking high impact areas, I would be testing them.

    Also, sometimes the new and old are so closely combined it makes little sense to retest in regression where more effort testing the features the first time would mean you wouldn’t have to do any regression at all!

    To me thats efficiency!

  7. “Also, sometimes the new and old are so closely combined it makes little sense to retest in regression where more effort testing the features the first time would mean you wouldn’t have to do any regression at all!”
    Yep – Lets take product A, which is made up of function 1, 2, 3, & 4.

    1 – Major change.
    2 – Minor change.
    3 – No change.
    4 – No change.

    Risk levels are implied for this example.

    Traditionally you would focus initial efforts on 1, then 2, then ‘regression test’ 3 and 4, yes?

    Why not take product A, look at the holistic change, develop test ideas using your various techniques, oracles, and go for it! Do we need to separate them into change and regression? Could we not just test the product capturing all areas that carry whatever risk you’re trying to mitigate?

    No checking required… Just testing. It could be automated, it could be manual… Whatever suits the context of your project/environment.

    I think I’m getting back to my ‘label’ point… But anyway. ;0)

  8. Agree to the idea in the post. Regression testing should not be done mechanically just to get it over with.I have always maintained that regression testing is more than test execution. Regression testing should also involve analysis, planning and preparation. Out of these, test preparation includes examination of the regression test suite/ test data – retiring useless/ duplicate test cases, updating test cases to sync with the current application functionality and writing new test cases with greater likelihood of discovering bugs.

    Anne-Marie: I’m not going to argue with that!

  9. Some to agree with here, and plenty to disagree with too…
    1. Regression testing isn’t about finding bugs the same way as one might test new software to detect bugs (testing actually does not detect bugs, it exposes failure. Whatever.) It is about detecting unwanted changes in functionality caused by a change to software or its environment. Good regression tests are not necessarily ‘good functional tests’. They are tests that will flag up changes in behaviour – some changes will be acceptable, some won’t. A set of tests that purely achieve 80% branch coverage will probably be adequate to demonstrate functional equivalence of two versions of software with a high level of confidence – economically. They might be lousy functional tests “to detect bugs”. But that’s OK – ‘bug detection’ is a different objective.

    2. Regression Testing is one of four anti-regression approaches. Impact analysis from a technical and business point of view are the two preventative approaches. Static code analysis is a rarely used regression detection approach. Fourthly…and finally … regression testing is what most organisations attempt to do. It seems to be the ‘easiest option’ and ‘least disruptive to the developers’. (Except that it isn’t easy and regression bugs are an embarrassing pain for developers). The point is one can’t consider regression testing in isolation. It is one of four weapons in our armoury (although the technical approaches require tools). It is also over relied-on and done badly (see 1 above and 3 below).

    3. If Regression testing is about demonstrating functional equivalence (or not), then who should do it? The answer is clear. Developers introduce the changes. They understand or should understand the potential impact of planned changes on the code base before they proceed. Demonstrating functional equivalence is a purely technical activity. Call it checking if you must. Tools can do it very effectively and efficiently if the tests are well directed (80% branch coverage is a rule of thumb). Demonstrating functional equivalence is a purely technical activity that should be done by technicians.

    Of course, what happens mostly is that developers are unable to perform accurate technical impact analyses and they don’t unit test well so they have no tests and certainly nothing automated. They may not be interested in and/or paid to do testing. So the poor old system or acceptance testers working purely from the user interface are obliged to give it their best shot. Of course, they try and re-use their documented tests or their exploratory nous to create good one. And fail badly. Not only are tests driven from the UI point of view unlikely to cover the software that might be affected, the testers are generally uninformed of the potential impact of software changes so have no steer to choose good tests in the first place. By and large, they aren’t technical and aren’t privy to the musings of the developers – before they perform the code changes so they are pretty much in the dark.

    So UI driven manual or automated regression testing is usually of low value (but high expense) *when intended to demonstrate functional equivalence*. That is not to say that UI driven testing has no value. Far from it. It is central to assessing the business impact of changes. Unwanted side effects may not be bugs in code. Unwanted side-effects are a natural outcome of the software changes requested by users. A common unwanted effect here is for example, a change in configuration in an ERP system. The users may not get what they wanted from the ‘simple change’. Ill-judged configuration changes in ERP systems designed to perform straight-through processing can have catastrophic effects. I know of one example that caused 75 man-years of manual data clean-up effort. The software worked perfectly – there was no bug. The business using the software did not understand the impact of configuration changes.

    Last year I wrote four short papers on Anti-Regression Approaches (including regression testing) and I expand on the points above. You can see them here:

  10. Hi
    I disagree – Allow me to explain!

    Regression – A known to be working functionality gets impacted due to a code change or environment change. Code change can happen due to 2 reasons – bug fixing or new feature enhancement.

    Anne-Marie: The developer refactors their code, updated 3rd party libraries, OS changes there are lots of reasons why code may change

    Regression Testing – Test analysis, selection/design and execution to find these possible defects.

    Anne-Marie: A lot of people would disagree with that statement, saying that its to check that features are not broken.

    There are “various” straties on performing regression testing.
    The strategy you mention is known as Impact Analysis.

    Anne-Marie: You misunderstand. I’m not suggesting we perform impact analysis to determine what regression testing, I’m asking you not to make a distinction between regression testing and other testing
    The impact of the change is analysed to understand whcih possible areas of the system may have got impacted.

    However, this is just one of the strategies. This strategy is in my opinion and experience the most popular one. However this “really” cannot be applied for safety critical systems.

    Anne-Marie: Why not?
    On the other hand, executing each test case again is also possible. Should we do it? If the risk is very HIGH.. why not?
    If possible, automation should be strongly considered.

    Another one, is failed test case. In this we execute all those test cases which failed in the history of the project. Or, verifying that each defect has actually been fixed, by executing them again.
    When should we follow this strategy? In case Development team does not configuration management system is place? of even if they do have configuration management system in place, they do not have a process. The process is not mature. Or just because development team makes a mistake since they are human beings.
    In big shops, development teams are “expected” to follow configuration management principles, however we really cannot be sure? can we? *smile*

    Anne-Marie: What I’m suggesting is an alternative approach – number 4: design new tests

    We can also club these strategies with mandatory/critical/legal requirements testing.
    For instance, if my impact analysis says that THERE IS NO WAY requirement number 10 would be impacted, I still go and test it. Since Impact Analysis is done by a human being, we have the possibility of making mistakes. Also, What if?

    Environment change can also cause regression. That is the reason we say – Tets environment should be as close to production environment as possible. Also, during deployment, testing performed by deployment/system admin team is also performed.

    Gaurav Pandey

  11. Great article, Anne Marie.
    I particularly love this one: “It seems wasteful to me to ask testers to perform tasks that result in the tester disengaging from the activity because its boring.” You can see which testers are running the regression test suite from a distance, based on their “barely-awake” look.

    I agree with what you want to happen here but it’s unfortunate that most companies I’ve been with have such a rigid idea of regression testing that it’s hard to change things.

    Anne-Marie: Unfortunate yes, but it wont remain that way always. People thought smoking was fine until enough scientists said “no, smoking is bad for your health, stop it” and so most of us eventually did….

Leave a Reply

Your email address will not be published. Required fields are marked *