Courage in Exploratory Testing

Exploratory Testing takes software testing skill. It also requires the tester be courageous. Let me explain.

Exploratory testing is an approach, not a technique. Exploratory Testing is simultaneous learning, design and execution. What information we learn about a product, helps dictate our tests providing us with information that we can share with people around us.

Exploratory Testing is tester centric, meaning the tester is central to the testing taking place. The tester has the autonomy and the responsibility to make decisions about what to test, how to test, how much to test and when to stop. This may seem blatantly obvious to some, but its surprising the number of test teams where this is not the case.

Its all powerful stuff, generating an environment where the tester must constantly reflect upon the changing project and product environments, enabling the tester to be engaged and mindful as they test.

I honestly can’t think of a better approach to software testing.

But there is a catch. Exploratory Testing also demands great courage.

When you start to take responsibility for your testing it’s not done in isolation. Testing requires interaction with many different people such as developers, project managers, scrum masters, product owners, customer support and business people. We share information that’s not always good news. Its in the form of bugs found and the possible impact of this information on business. The resulting consequences of the information we share often leads to delays in releasing, changes in work load, context switching and revising strategies.

In scripted testing, testers have artifacts which they measure and count giving an illusion of of certainty but really this is smoke and mirror reporting and generally offers little genuine information. “We have reached 78% test coverage, with a DDR of 85%”

Exploratory Testing doesn’t have ‘dutch courage’ to rely on. It requires us to have conversations about our information in potentially hostile environments. Sometimes we can feel like the lone fish swimming against the tide of the silent majority. It can be tough and as testers we need to learn to develop how to speak about our testing, how to tell a story (James Bach and Michael Bolton have both written on this). courage in Exploratory Testing

Here’s a list of ways that have helped a quaking knock kneed tester like myself discover her backbone:

Speak to someone you trust about your concern. Vocalisng a fear helps to make it tangible and sometimes gives strength when you discover its a shared concern.

Be coached or mentored on how to speak about testing with confidence

Take small steps. Speak to people sympathetic to your cause, sound out ideas. See if other people can help.

Try not to lose faith, be persistent. Keep your eyes on the goal, even if sometimes you fail to speak out.

Emotions are your toolbox. Anger and frustration can be very useful emotions! Use your emotion to give you courage to speak out. (I learned that at PSL this year..thanks to Jerry, Johanna & Esther)

Sometimes you need help. Be humble enough to know that sometimes change is out of your capabilities. See if you can find help through the testing community or see if you can bring someone in to help affect the change.

But mostly, its about practice. Courage breeds courage. Standing up to little things helps give you courage to stand up to greater things in the future. Be brave. Be strong.

What drives me most of all is that I want to be able to walk away from a situation with my head held high in the knowledge that I may not have changed the world, but I’ve had my say.

Now that’s a good days work.





Speed Kills

Some of my testers have become embedded on a newly formed agile team. Its been a roller coaster ride for sure. Lots of fun, thrills and a few scary points in time where I thought for certain we were not going to make it, or we were heading in the wrong direction.

From a testing perspective, we were quietly confident. Our testing is exploratory by nature, and so it lends itself easily to being flexible and adaptable.

We’re testing before the acceptance criteria are developed by attending the workshop and asking questions about the upcoming feature, learning why its needed and what problem its trying to solve.

And when it comes to testing the feature, we all jump head first  into the deep end, swimming with long powerful strokes through the water of doubt and complexity and only surfacing for air to congratulate and high five each other on another completed testing charter.

Its been a wet wild ride, but not without some uncomfortable moments.

While we’re revelling in the warm waters of upfront information and involvement, we’ve also noticed the cooler waters of reduced timeframes. Shorter iterations has placed a lot of pressure on the testers. There’s a feeling that we’re unable to pause and reflect during our testing, the pressure to be done within the iteration lends to the temptation of minimalist testing.

We had to change something to make sure we were testing well enough.

So, we slowed down.

We made sure we spent time to think critically about our testing, coming up with test ideas, understanding what done meant, and most importantly sharing these ideas with each other (including developers).

Now we test like this:

We still jump into the deep end, splash out, learn some stuff about the product but before we continue with some serious swimming we stop and confer. We reflect on our learning so far, we discuss our ideas and how we know we are done. Only then do we continue with our swimming, our strokes confident and strong.

Its been a revelation for some testers that are new to exploratory testing. They thought the objective was to test as fast as you can without stopping for a breath.

We still have the same time pressure on us, but that stop, that moment of pause and reflection has been sufficient to gain confidence in our approach and confident in delivering the best testing that we can.



Crossing the bridge over no-man’s land

Indiana Jones in the Temple of Doom  has a “moment of indecision” when half way across a bridge he realises he’s been cornered by Mola Ram(the baddy) and his henchmen.

He looks back and there’s a hoard  of lusty savages baying for his blood, no luck there. He looks ahead only to find an equal challenge ahead. What is poor Indy to do?

Anyone who has introduced change into a corporate environment will empathize with his situation. You know the decision to break away from the past is the right one, you run eagerly and embrace change, only to find half way through the journey, your path to success becomes blocked.

I’ve been working with my test team for a while now moving from a process driven approach to a more of an Exploratory one.

Its not been without its challenges. Some concepts I’ve introduced have been welcomed warmly but the reception to others has been a little icy. In particular I’ve tried to move the team away from a test case management system. This was met with real concern and there was quite a resistance to the idea.

This troubled me as while I understood their concerns, I knew the system was limiting the generation of new testing ideas.

But how could I overcome this resistance? And really was it worth it? Perhaps the changes I had already made would be enough? The company was already more than impressed with the changes I had made so far.

I felt like Indy at the foot of rope bridge, how the hell was I going to solve this one?

So I stood at my crossroads and dithered. Oh God, did I dither. I ummhed and ahhed and pondered what to do. . But worse, I  knew my indecision was making the situation worse, and that the more I dithered, the harder it would be to rid ourselves of the dust bag of tired and well worn ideas.

Indy at this point, decides his only move is to cut the bridge leaving everyone to hang on for their lives.

Fortunately, unlike Indy I had a reliable and trustworthy sidekick.  Together, we setup a task force within the team to attack the problem. After some discussion we decided our approach needed four cornerstones. They were:

1) Creativity.

However we tested, our approach needed to enable us to foster and encourage creativity. With creativity comes new ideas, new models, new tests and so discover new bugs.

We’re covering this one with a number of approaches. One is to improve tester skill through practice and coaching. I’ve also created a folder of ideas for people to draw upon to help trigger new ideas.

2) Visibility

We wanted to be able to provide reporting on any testing we do. The reporting has to be simple yet with sufficient detail to ensure that our stakeholders understand what we have tested and why.

We have our trusty whiteboard which mostly hits the spot. We need to be able to pull up our actual testing including results in an easy to manage way. We’re looking into BBExplorer to handle that.

We will also track any essential test results on wiki in the form of a test report at the end of each iteration.

3) Coverage

We wanted to have some way of ensuring that key functionality/key features are always tested.

We most likely will rely on our test case management system for this, but we’re cleaning out all the dead wood and making the tests lighter and less authoritative.

4) Knowledge

We wanted to create a knowledge base. Our system is complex and it requires in-depth knowledge to test some areas. We want to store that information and knowledge. We also have a serious amount of test data we want everyone to be able to access, modify and improve.

We’ll use our internal wiki for this.

What I really like about what’s happened here is that the team came up with a solution to solve the problem. It’s a team decision which has got to mean easier implementation.

I think a couple of really powerful things have come out of this. I’m listing them here:

1) Change can be scary. Not changing is worse. Get on with it.

2) Use people around you to help bring about change.

3) Never lose site of your goal. This reminds me of Scott Barber’s email signature: “”If you can see it in your mind…you will find it in your life.”

I feel good. I hope my team does too. We faced a challenge. We examined it, questioned it and overcame it and we’ve all come out sharper, enlightened and positive about the changes ahead.

Now that’s what Exploratory Testing is all about.


Rapids Software Testing

As some of you know, I’m in the process of creating an Exploratory Testing workshop. It’s been a bit of a wild adventure, but hey, I’m clinging tightly to my oars as I hurtle down the rapids of ET adventure.

Have you ever been white water rafting? I have, and here’s a tip, don’t bother going if there is a drought.

Trust me, I learned the hard way on the Tully River in North Queensland. Tully is one of the wettest populated towns in Australia with an average annual rainfall exceeding 4000 mm (13.1 ft).

But not the year I went. I went when there was a drought and the water levels on the river had dropped.While the day’s outing was great fun, it never reached the hair raising exhilaration that I had anticipated.

It can be a bit like that in testing I guess.  If you want to have fun and be challenged, it helps to go where the water is deep.

Well I’m in deep testing water and I’m loving it! A day doesn’t go past where I’m not motivated to learn more and to challenge myself. To hell with the life jackets, watch me go!

Why? Because I’m learning something that is fundamental to any tester.

I’m learning how to teach  testing.

Precisely, I’m learning how to teach testing through Socratic Examination. This means, that I’m learning to ask the questions, pose puzzles and push students to struggle through testing principles so they come to a better understanding.

If this style sounds familiar, its because James Bach is teaching me this stuff. Its all part of this new coaching program which I’m aligning myself with. I will also be collaborating with him on a book he’s writing on the topic.

My experience on learning to teach suggests to me that this book is much needed. Practice is key to being a good teacher, but having a few strategies and heuristics to guide you along the way is essential too. This book will go some way to demonstrate that.

So what have I learned so far?

Lesson 1: A Mental Model

When working with a testing exercise you need a mental model of what you are aiming to teach.

Its not an easy task. There is no one strategy or model that fits all students. All students differ in their learning needs and in temperament. what works for one person, may not be suitable for the next, yet your mental model needs to cater for each individual.

You need to know your outcome, and where you are taking the exercise and still allow the student capacity to explore and come to some learning outcome.

I’ve noticed that James starts his coaching sessions with a mental test. He uses that to observe a tester’s thinking. He then frames his coaching session around a key thought or lesson, allowing  the tester to explore, yet always bringing them back to the intended final outcome.

All without one powerpoint slide.

I’m learning how to do  that too.

Lesson 2: Observation.

The coaching sessions may seem unstructured and ‘ad-hoc’, but as I mentioned there is always an underlying model or framework in use.  I’ve been observing some of these coaching sessions, and I’m starting to see patterns of behavior. I asked James about this and his comment was this:

Anne-Marie Charrett: When you are having these conversations do you consciously have an idea of the types of patterns you are going to use?

James Bach: Yes

James Bach: I’m trying to become more conscious of them and to make them easier to teach

James Bach: that’s what I’m using you for.

James Bach: we’ll learn them together

Observing patterns is essential to honing your teaching skills. Only through observation can you identify how you teach, what your natural strengths are or where you are biased. But also identifying patterns, helps you know what pattern (or heuristic) to use next.

Naturally, being taught directly by James Bach is helping a lot too. I think confidence in yourself is critical, both as a tester and a trainer. After all, how can you confidently explain your testing story if you have little confidence in yourself or what think you believe?

So that comes to lesson 3:

Lesson 3: A Testing Story

Teaching testing gives you confidence in your testing story. Yes, I read and study Exploratory Testing, I use Exploratory Testing. But standing up and talking about Exploratory Testing to me is the ultimate test in what you believe. If you can stand up and talk about testing, its a great boost to your testing story. Well , it is for me anyhow.

This confidence comes by first willing to put yourself in a vulnerable position, where you are willing to learn. It was only when I blogged about my difficulties about creating an ET workshop did help arrive.

It also comes through practice. I’m doing that too now, by blogging and testing out by challenges on fellow testers. I’ve already asked a few of you to testing challenges on skype or IM. I need to practise my exercises and puzzles against a variety of people.

If you want to be part of the fun, skype or IM me. I’m happy for anybody to take up my challenges. I need practice to improve my skills.

There’s lots more I’m learning, most of which my mind has yet to digest and formulate into identifiable ideas. But are you starting to see something here?

Teaching testing is very similar to testing itself, maybe a bit more intense…. like ET on steroids perhaps.  I strongly urge any tester looking to improve their skills to consider this option. Even if you never end up teaching formal workshops, the insights you get about yourself, the confidence it builds in yourself and your ideas in testing will stand you in good stead.


I value your thoughts, in particular if you disagree, or question what I’ve said. Every discussion on this helps me refine and consolidate my understanding on the subject.

reinventing the wheel

I liken my experience of creating an Exploratory Testing workshop as similar to recreating the wheel. It would be easy to copy someone else’s version of a wheel, after all there are a lot of great wheels out there. Alternatively, I could create my own wheel.

But how do I do that? How do you improve on the wheel?

My feelings about holding such a workshop range from massive excitement to extreme anxiety as I grapple will the question, “What the hell am I going to talk about”?

I’m not talking about the content, there as been plenty written on Exploratory Testing and I could spend weeks just reading up on other peoples articles, notes etc. If I wanted to, I could re-regurgitate lots of excellent material on the subject. But I have a problem with that approach. In fact I have two problems (maybe even three if you take into account the massive attack of self doubt I had this morning!)  with that approach.

The first one is this.

How do you teach Exploratory Testing? As James Bach writes in his article Exploratory Testing Explained and something I WHOLEHEARTEDLY concur with (through bitter experience!) is that:

Among the hardest things to explain is something that everyone already knows. We all know how to listen, how to read, how to think, and how to tell anecdotes about the events in our lives. As adults, we do these things everyday. Yet the level of any of these skills, possessed by the average person, may not be adequate for certain special situations. Psychotherapists must be expert listeners and lawyers expert readers; research scientists must scour their thinking for errors and journalists report stories that transcend parlor anecdote.

So it is with exploratory testing (ET)

The second problem I have is that

If the workshop is going to be any good, I know it has to come from the heart, my heart.

So even if I was tempted to say take James Bach’s course and deliver it (anyone who takes his course has this permission, as long as credit is given) it would be pointless.  Because I know it has to be my story, what my understanding of ET is, not anyone else’s. Its one of the reasons why James’s course is so damn good. His conviction comes through because its HIS story.

As an exercise, creating an ET workshop is a great challenge. It demands the ultimate in story telling. There is nothing better to confirm/challenge your beliefs and/or understanding than to articulate it in front of a class.

There is also nothing more humbling. It makes you realise how much I have still to learn. As part of my effort in creating this course I’m reviewing  familiar documentation again. In particular James’s RST slides.Looking at these slides in a critical way has been very beneficial. Its made me question my depth of understanding of some of its content.

Its been a good morning though. I’ve made some baby steps and have got some ideas that I feel I can call my own.

I know that the workshop is to be practical, and I have a few ideas in mind on that. The challenge is to use the exercises to teach an ET point.

What the workshop(or the wheel) will end up looking like, I’m not yet too sure. One thing I do know is that as time evolves it will become more and more my own story and yes, my wheel.

Get out your tin whistles

At last after a year of wrangling, shuffling and even some pleading Michael Bolton in association with Testing Times is coming Dublin to give his wonderful Rapid Software Testing Course.

Not that Michael needed persuading to come. He jumped at the opportunity. Mostly because he loves giving this course and helping testers well, develop sense. But I will let you into a little not so well known fact about Michael. He loves Irish Music and his a keen Mandolin player.

So we knew we were onto a winner straight away!

For those not familiar with Michael Bolton and his course.

Rapid Software Testing is “a course, a mind-set, and a skill set about how to do excellent software testing in a way that is very fast, inexpensive, credible, and accountable.” Its written by James Bach and Michael Bolton

This course is excellent, its practical and thought provoking!  I can personally say that because I’ve taken it. If you have ever asked yourself the question:

“Is there a better way to test this stuff ?”

Then I suspect this course is for you.

Some of the issues it addresses are:

  • Are you finding it difficult to assess how much time and effort you’re going to need to test effectively?
  • Are you overwhelmed by or uncertain about approaches to test planning, design and execution?
  • Are you working in an environment where some people aren’t following “the rules”?
  • Are you having trouble finding the right balance between planning, documentation, and testing?
  • Are you interested in learning skills and techniques that will help you to become a better tester?
  • Are you finding that “industry best practices” are infeasible and a poor fit for your organization?
  • Do you want to get very good at software testing?

Read more information about Michael Bolton and the course go to his website: Michael Bolton Rapid Software Testing

Even better Skillnet has agreed to partially fund the course, so you are getting this 3 day course at a knock down price of 770 euros.

If you have any money in your training budget, this course is the one to go for!

Rapid Software Testing Details

  • Date: Monday 13th to Wednesday 15th September 2010
  • Venue: Xilinx, Citywest Business Park
  • In association with Testing Times & Xilinx
  • Duration: 3 day course (9.00am to 5.30pm)
  • Cost to non-members: €1,700 per person
  • Cost to Software Skillnet Members* after Grant aid: €770 per person

*Membership to Skillnet is Free

For more details and booking go to the skillnet website:  Skillnet Rapid Software Testing

Are you a Buccaneer Parrot?

I’ve had a couple of ‘epiphanies’ this morning and consequently have that weird floaty, happy/anxious feeling that I get at moments like this.

Courtesy National Geographic

I didn’t go to StarEast but I, along with countless other software testers have been watching the virtual show through twitter.

Some of it I let go, but a couple of links I’ve clicked on to see what all the fuss is about. Boy, if the two links I clicked on are anything to go by, it would have been a great event to be at.

The first seismic changing event for me was listening to James Bach’s clip on what it means to be a buccaneer tester.  Now James is a tester that I have mixed feelings about. I went on his RST course a few years ago and really enjoyed it. I loved his FOCUS/DEFOCUS heuristic and its one of my key techniques that I apply when I test. BUT, sometimes I find his style can be overly aggressive, especially when it comes to:  oh crap, here I go, CERTIFICATION.

But not today. Today he was sublime and this key point hit me:

“We have got to be testing from our own place, not copying like parrots”

This resonated heaps with me as its a trap that I constantly fall into.  Now, I pride myself on my software testing skills. I do have a fondness for exploratory testing and have a knack in asking the ‘right’ question when determining context. Here’s the crunch though. I call my approach pragmatic. In other words, I DONT ROCK THE BOAT.

So when asked, I will happily manage a team of scripted testers. Why? Because that is the process. That is how things are ‘done’ in that company.

This creates a dilemma for me, because it leads me to ask what in  software testing do I really believe in? Where is my “own place”?   To what lengths would I go to ensure Exploratory Testing was used in a team I led? (I know I would never personally create a test script, or follow one for that matter). This is pertinent to me because I’ve just gone for a test lead role that requires ‘strict adherence to a process’.

In order to be able to answer these questions, (and writing this post is helping me decide mine) and as James puts it ‘test from your own place’, y ou have to know where and what “that place”  is. The only way your going to find it, is by taking a stand, having an opinion, suggesting a new approach and being prepared for people to disagree with you.  To quietly agree(or disagree) is not going to help you know what you believe in.

It’s funny you know, I’ve always thought that being outspoken is such an ‘American’ thing. We ‘Europeans’ are way too polite to express our views so, well, blunt. Perhaps there is an element of difference in culture, but I don’t think I can hide behind that excuse any more.

Here’s the great thing though,  when you find ‘your place’ you will be a lot more secure and that is going to help you become a better tester.  Why? Because being secure about your place means you don’t have to worry about what other people think. You are less fearful one thing I know for sure:


It kind of goes with Elizabeth Hendrikson’s article that says “Not knowing answers isn’t sign of weakness; not asking question is”  Being fearful of looking stupid, prevents you from asking the dumb question. By the way, before you pat yourself on the back about being able to ask ‘dumb questions” I believe its easy to ask the dumb question when your experienced in an area, but try doing it in a field where your not so experienced. It can be a real challenge.

And it can be a real challenge for experienced testers to ask ‘dumb questions’ when they want to appear knowledgeable. I think for wannabee experts this is a real trap. If you try and spend all your time looking like you have the answers,  you put yourself in a situation where you stop asking ‘dumb questions’

Why do I believe all these things, because its what I feel and do sometimes. No often. And its something that it going to change. Now.

So thats why James Bach’s Video resonates so strongly with me.

Thank you James, you have once again been instrumental in my growth as a software tester.

Arghh, what’s that pieces of eight, pieces of eight?

Holding the cat by the tail

I thought Jack Margo’s interview by UTest was very interesting. What caught my eye was the following statement:

The days of specialists are mostly killed from the recession…you have to be flexible and know multiple disciplines to exist in today’s dev environment.  In web development alone, you need to be proficient with XML, DHTML, JS, a DB flavor, an OS flavor, a programming language and some semblance of UI Design to even handle front-end.  I have friends who knew only HTML or only PERL.  They are struggling to say the least

It made me think  the same applies to us as software testers.

Have specialists in software testing being killed by the recession? Is it necessary for software testers to be ‘flexible’ and know ‘multiple disciplines’?

Personally, I think so. Its not good enough these days to be a ‘manual tester’ or an ‘automated tester’. Instead you need to be able to do both. I don’t think that means you have to be ‘expert’ on both, but it does mean you have to have knowledge of both and a good knowledge in one area.

That’s why I’m excited about Nathan Bain and the free automated testing sessions he’s starting up.  As he puts it:

Come to meet fellow testers, share stories and experiences about tools and techniques which may, or may not, have solved testing problems on other Agile projects.

This is also a place of learning, where live demonstrations of tools will be given for FREE – no more expensive training courses for simple (and free) open-source testing tools.

What a fantastic opportunity to learn about automated testing!

To complement this, Rob Lambert has setup some free Exploratory Testing Sessions.

Both organisers have mentioned that these sessions could also be performed online.

I am not going to miss out on either opportunities. I would encourage those interested to sign up to both, either to contribute so others can learn, or learn from someone else.

BTW: two quotes were in contest to head this post. The first one was by Mahatma Gandhi:

“Live as if you were to die tomorrow. Learn as if you were to live forever.”

The other was:

“If you hold a cat by the tail you learn things you cannot learn any other way.”

Mark Twain

I love both for different reasons, but I thought the second one appealed to me as a tester, hence the title :)

Do your bugs only glow when its dark?

Have you ever been in the situation where no-one else can find and repeat the bugs you find? Perhaps you have a canny knack of finding unusual bugs, or maybe it’s time to improve and update how you write bug reports!

I do a lot of offsite exploratory testing in very aggressive timelines and consequently I have to admit, I sometimes start making basic mistakes.

My common mistakes are:

1) I don’t write the bug report up straight away

2) My bug reports are not detailed enough

3) I underestimate the time it takes to write up defects

4) I don’t use a defect tool

You can imagine, when you’re working offsite without even meeting the developer, this can lead to all types of complications. So I need to put myself in that poor developers shoes and make my bug reports as clear, concise and as detailed as possible.

I think it’s tempting when there is little formalised structure in testing to overlook writing a disciplined bug report, but it’s worthwhile to you and the customer. I mean what’s the use of paying someone to find bugs when no-one else can find and fix them?

Anyhow to prevent myself repeating such unforgivable mistakes, I have developed a set of guidelines to follow.

Offsite Exploratory Testing Guidelines to Bug Reporting

1) In the estimate to the customer, make sufficient time to write your bug reports as you go. It’s impossible to know how many bugs you are going to find but you can do a bit of math. If you’re planning three    days of testing, and you find 100 bugs @ 10 mins average, then that’s two days of your testing filled up with just writing up reports.

2) Agree on a detailed bug report with your customer. Try if you can to get agreement with the developer. Try and include as much background information as you can. There has been a load written on what constitutes a good bug report, so I won’t repeat it here.

3) Don’t delay; write up the bug report straight away. This is hard when you’re in the middle of some exciting analysis and you really just want to keep testing in the timeframe you agreed on. But trust me; it takes longer to write them up at the end, when you have to review heaps of cryptic phrases in Session Tester or in your notebook. A capture tool may be helpful in recording data, but if you leave reviewing to the end, it will add additional time and effort to the testing

4) Encourage customers to use a defect tracking tool. It saves everyone a lot of headache and heartache. There’s lots of open source defect tracking tools out there. TRAC and Bugzilla are the two most common.