BOB series: Shallow Testing gets a bad rap

In my Exploratory Testing class, I talk about shallow and deep exploration. When you say the word exploratory testing, many think, ‘just playing around’ on the product. And to some extent that’s true, a good true. So when I talked about shallow exploration it seemed to fit. Different from deep exploration where a systematic approach is used to observe, model, use and evaluate the product.

More often, I’m seeing the phrases ‘shallow testing’ and ‘deep testing’. These to me are problematic. Shallow testing used in this way to suggest that simple functional testing (potentially automated) is trivial. Deep Testing, on the other hand, requires going beyond the superficial and delving into the product often at a technical level. An example of each would be, testing at the GUI level,(Shallow Testing), removing the covers and performing testing under the surface(Deep Testing)

Right now I’m with a client and I’m performing functional tests that have been written in a test case management tool. I specifically avoid any kind of investigative testing. I perform the test and I pass or fail the test. Is this testing confirmatory in nature? Yes! and what’s more, this is providing huge value to my client.

There’s nothing trivial about it and by using the word shallow it dismisses the importance of this type of testing to a stakeholder. There is nothing like a functional test to give confidence to a business user that the product is ready to go.

There’s something else. These tests are often essential beginnings to discovery. When you have a largely unknown problem space that needs exploration, we first need to make sense of what we are working with. In David Khlar’s book “Exploring Science: The Cognition and Development of Discovery Processes” a common heuristic was to confirm an initial hypothesis which allowed a springboard to further investigation.

So what’s the alternative? Anything similar to shallow will have the same connotation. Perhaps we don’t need a distinction at all? Is it really useful to describe our testing in these terms? Most people want a shippable product and don’t give a flying koala what the software testing is called. What’s more, do you hear developers talk about deep coding and shallow coding?

If you really want to classify testing what about the old but familiar black box testing and white box testing?

(This series of blog posts is called ‘blogging on the bus’ – BOB )

3 thoughts on “BOB series: Shallow Testing gets a bad rap

  1. Excellent post and well formulated.
    We do testing to gain information and knowledge, usually to then provide this to a stakeholder.
    this information is usually requested by said stakeholder, and will have some value to them.
    might be a good thing to question the information request, but other than that what we call what we do to gain the needed knowledge is irrelevant, Black Box, Test cases, Automation, shallow, deep exploration etc.
    in the long run this is just words.
    We gain knowledge by doing certain activities – full stop

  2. In my opinion, the term black box testing is well established and understood by everybody in software development. If we need a different one for the “outside” world, “shallow” sounds too negative. In the end, for me this is “user” testing, as we try to mimic in our tests the knowledge, environments and expectations of the people that actually interact with the software.
    Please keep these posts coming. They are great food for thought!

  3. Interesting post.

    I really like using Cynefin for classification of different types of test problems. Simple, complicated and complex. Where simple and complicated test problems can be automated or solved efficiently with rudimentary knowledge, or if complicated, deep system knowledge, but without any higher level of test competence. Complex problems where you explore, investigate and experiment then requires a higher level of test competence. I really like how Jeff Nyman frames questions around test competence in his blog.


Leave a Reply

Your email address will not be published. Required fields are marked *