Leave the testing to the experts!

Why is it that in a company almost everyone has an opinion and knows the best way to test? What makes so many of these people feel they know how testing ought to be performed?

Business people tell me, the tester:

“You should be writing test cases and reporting against them”

Developers & technical people tell me to:

” automate the lot using cucumber/PACT/Selenium or some other niche technical tool” 

Test Managers (many who have never tested) tell me:

“You should be following a formalised signed-off testing process”

I find this quite extraordinary. I don’t tell you, oh developer how to code your program. I don’t tell you oh, sales person, how to sell your product.

So why do you think its reasonable and perfectly acceptable to tell me how to test software? 

Perhaps its because we all test to some degree. But just because I can build a car or sell one doesn’t mean I can drive it. Just because I know how to drive a car, doesn’t mean I go and tell a rally driver how to race around a track.

The truth is, most developers, project managers and business people don’t understand testing in any deep way. Instead they often resort to shallow imitations, an emperors test suite, if you will.  That’s partly the fault of the testing profession. We’ve failed to claim our turf and instead tugged our forelock, deferring ideology to those with authority but little understanding.

It’s time tester’s to claim some turf back. I suspect (actually I know) for many places it won’t be given up without a struggle. You’re going to need courage backed by confidence in your ability if you want to claim some of the territory. But it’s worth it. With space comes freedom and the autonomy to drive your testing strategy and process in a way that you know will offer value to your organisation and your clients.

Now, please get out of my way and leave the testing to the experts!

Scientia potentia est

“Knowledge is power”

Jerry Weinberg cites courage as the most important trait in a tester. Quoting Kipling, Jerry says testers need to “keep your head when all about you are losing theirs and blaming it on you.”

But I see a another type of courage at play in software testing. Testers are foremost learners. Through enquiry,they learn about a system. The information they gather facilitates many kinds of decision making from releasing to designing new features.

Observe great testers and you will discover an insatiable desire to learn more, not only about the product, but the world around us, often incorporating what they learn into their testing.

For many of us, discovering that our learning is within our control and within our means, can itself be a road of discovery. It takes courage to start that journey, but it also take courage to continue along its path.

Young Luke Skywalker found the ‘Force’ early on in life. Yoda helped him connect with that power, but even then, he needed guidance and a reminder to ‘Use the Force’ when up against the evil Empire.

Some of us need that reminder now and then. We know we have the ability to learn, and we know of its power, but we forget to use it. Especially when things get tough.

“Knowledge is power”

When things don’t go the way you want, when the pressures of daily life cloud your ambitions and goals, it can be easy to lose site of learning. Here’s what I’ve discovered though, through focusing on learning in these times you gain great strength.

Will the actual knowledge you learn help you succeed? Perhaps. What really counts is that through learning comes power in the form of ownership and self belief. You may not be able to control the situation at hand, but through being open to and owning your learning, you regain a sense of control and a sense of focus.

So when the dog bites, when the bee stings and when you’re feeling sad, remember there is solace in learning. It’s not only as an escape, or way of learning how to deal with the situation, but helps you take ownership and responsibility over the next step. Who knows, learning may be the just the ticket you need to recharge those batteries, giving you the juice to continue on your journey or perhaps, dump it for a different destination.

This post was first published on medium. Mauri Edo tweeted about it recently and I’ve decided to post it here too because I like it so much. 

Question with sprinkle of humility on the side

Michael Bolton tweeted yesterday:

Why do testers insist on trying to be influential? I suspect part of the reason is that part of a tester’s job is to recognise problems. Too often , testers see that the *real* problem is not the software itself, but the process behind the software and go into bug prevention mode. That sort of change requires influence.

Often though, we simply don’t have the authority or the influence to do make change. We may moan and tear our hair out in frustration, but at the end of the day, without the mandate to make change and the influence to implement it, there’s little we can do to change an organisations culture or process.

Trying do do so regardless, can lead to a sense of helplessness and even slowly, over time, a sense of powerlessness. I suspect most of us at one time or another have felt like this. It’s not a great position to be in.

I know I’ve been there. I tried to change the culture of a company (company no less!) that had some  negative ideas about testing and teamwork in general. (I’m talking about this at Eurostar this year). As a consultant, I probably should have known better, but I argued (to myself) that the culture was affecting the tester’s ability to perform their job. It needed to change!

We  testers are gifted with keen observation skills and the nature of our role sometimes means we get to see and recognise problems that perhaps others don’t. But lets not get away with ourselves. Without a mandate for change, we become close to the schoolyard tale tattler, dobbing in on everyone and despised by all (including often the teacher). I failed to recognise that I hadn’t the authority or the influence to make these sorts of changes. I fell into a classic consultants trap. I really should have known better.

It’s not that we *should* or *should not* ignore these problems. Often these challenges are too complex to be solved with simple answers and probably need to be dealt with on a case by case basis. But I think adopting the tone of helper (or servant) goes a long way to contributing to an answer. In some cases a question sprinkled with a little humility can be more helpful than smothering the problem with large doses of  tester sauce.

I hope I’ll remember that next time!

Yu Han on what Software Testing means to me

Last year I had the honour of teaching post graduate students the subject of software testing at the University of Technology Sydney. I asked students if they would like to write a post on testing on my blog. Yu Han did, and here it is. 

The subject of Enterprise Software Testing demonstrated the existing and interesting aspects of testing. Those lectures and the project in Suncorp are unforgettable experiences.

During the classes, I felt that, most of time, I was acting like a child, playing different kinds of games and drawing pictures with colorful pens. But at the end, there would be some serious discussions, analyses and reporting always broke my sanity.

Those reflections make me realize that my actions and thoughts were more complicated than I could realise, which encouraged me to keep reviewing what I did and exploring how I thought. By doing that, I better understood the purpose behind those activities, and I am being able to sense the essence of what a good testing could be.

I am aware of that my mind prefers to use memory or imagination to fill the gap between the reality and the information once I have received. By linking their similarities, I can better understand the information. But as soon as I came to this stage, my thinking could stop going deeper. I may feel satisfied with the simple explanation, or could be distracted by other information which will draw my attention to something else. Then I may lose the chance to find out the depth meaning of the information or even misunderstand it.

Now I am interested in paying more attention on my flow of thinking. By questioning appeared ideas could be a way to slow it down, which may clarify the understanding or identify potential obstacles hiding behind. It could also be possible to bring back those ideas or considerations which were once brought up then been omitted or developed during the incredible thinking speed. Those ideas and considerations could be questionable as soon as I try to challenge them.

This could turn out that there are missing facts behind the imagination which I feel reasonable but not actually practical. Once I try to gain the evidence to support those ideas, I could realize they are incorrect or unproved. I may need to confirm them before I go further, especially when the goal is based on such ideas. Like those assumptions made in the Big-Track exercise, our group was needed to test our ideas of what other buttons can do, before we finally test how the targeted button works. We questioned our ideas but hard to move forward until we know which ones were correct. We came up with many hypothesis, but failed to prove them at once in practice. After we had some sort of understanding, we found out that we should confirm those assumptions before we continued, which led us to come up a debugging strategy to process our tests. It appears that questioning the information not only could encourage us to seek the truth but also could inspire us to develop our ideas. In this sense, critical thinking could help one to wisely seek information and to reorganize the information into knowledge for idea development.

This now also reminds me the concept of Exploratory Testing. In the previous example, we learned the mechanism, built the tests and proved the ideas all by interacting and exploring with the actual system. It seems that Exploratory Testing could be a natural way to build and run tests while we also need to learn about the system. By understanding how it works, we could come up with how to do the test and find bugs. However, it’s difficult to tell how much time I need to finish the task.

Thanks from the experience in Suncorp, I see the efficacy of Scripted Testing. Our team only performed the testing within several hours. We didn’t need to worry about anything else, knowing other parts of the system and even the depth of the targeted section. It did take a couple of weeks for us to learn the background information and to write the strategy and test scripts, but we could save most of the time if now we are going to test another section. It makes me feel that with a good development and a clear testing goal, the job can be easily done by writing and following test scripts. But it is true that I also feel difficult to understand the system by reading documents, having meetings and even watching demonstrations. It seems easier for me to concentrate and memorize such information when I can apply it, which means learning the system by actually using it.

I am inexperienced to conclude what a good testing is or which testing method is more superior, but running the system to get reliable information, questioning the information to dig insightful connections, finding actual evidence to support the thinking seem to be the correct manners in testing.

Thanks for this subject which let me feel the enjoyment of testing, and provided a real business environment to gain practical experience. I will be happy to get involved in such field and learn more in the future.

I’ll be teaching this subject at UTS again in February next year. The course is open to students and practitioners alike .

Knowing the unknown

As a teenager, I remember being struck by the poignancy of the tomb of the unknown soldier. I’m not sure which tomb it was, in which city. I don’t recall there being one in Ireland, so perhaps it was in London. I remember feeling sad and perhaps a little understanding of the horror of war crept into me. For such a simple monument, the message was powerful.

For me, the tomb itself signified a lot more than missing soldiers. Somehow it symbolises those untold stories. Who was that person, why did they die? Was it painful, did they have a wife, children? it makes me think about history too, and how really its a story told by the victor. We rarely hear the story of the defeated. So many stories untold.

Roll on many, many years and I realise that we in software development, particularly in agile, we have many untold stories that only make the light of day when we find bugs in software. We fail to hear the stories that stakeholders wanted to say, but fell aside because of time pressure. We fail to hear stories that stakeholders have not realised exist. We fail to hear stories because some stakeholders weren’t seen as needed. We failed to tell the story because we simply didn’t think it was important enough.It’s a wonder software works at all!

Testing to helps us uncover and tell these untold stories. How? Each bug we find, has a story behind it. It may be story of why the bug came into being in the first place. Or the story may turn out to be unimportant. Often not only do we uncover stories,we also end up adding meat the stories that exist.

I find this particularly true of agile. The simplistic approach to stories that start with “As a <insert oversimplified description of user here>, I want <some oversimplified goal> so that I can <insert oversimplified ambiguous reason>” . I do understand that the point of these stories is to initiate conversation, but it’s my experience that often conversations are skipped and this skeleton like sentence ends up becoming the story. And as Allister Scott pointed out these stories would totally fail the bedtime story test performed by any 5 year old. Regardless these skeleton like stories then become converted into automated acceptance tests which significantly influences the decision to release or not. Well, in my experience anyhow.

As I understand it, one of the reasons for these simplistic stories is to achieve the goal of “dealing with the problem at hand”. By focusing on only what needs to be done now, we avoid over engineering a product. Now believe me, after working on some large scale telecommunications switches in the 80’s and 90’s, I totally appreciate this sentiment. However, the drive to simplicity has I think led to deficiencies in how we model our systems.

For example, we fail to recognise that firstly, some systems cannot (and perhaps should not) be modelled from a user perspective. (I’m happy to be proven wrong in this, my experience in testing r&d products and layer3 protocols leads me to believe this is the case though). Also by focusing on only the problem at hand we fail to appreciate the subtleties and impacts of the unknown unknowns.

I’d like to see software development attempting to re-address this balance. When I went to Agile Australia lots of people were talking about systems thinking and my first response was “Brilliant!” but it seemed that the systems thinking was focused primarily on business systems, as opposed to products. Perhaps its me with too narrow a focus, but I’d really like to see more discussion on how we can become better at modelling software in agile.For one thing, lets move away from telling stories ONLY from a user perspective. There are many ways to model a system and greater diversity of models may bring about deeper appreciation and understanding of the problem were trying to solve.

And so to the unknown solider. I’m glad no-one has tried to simplify his story into a sentence beginning with “As a soldier…”.  The monument conjured up more questions than answers, questions about the unknown. Questions that can never be answered. He was the unknown solider and it was fitting. i guess one question to ask is this:  is the unknown story a fit for agile?

The title for this post is inspired by Colin Cherry’s talk at KWST3 about Johari windows. [thanks for the spelling check Srinivas]

Big Trak Robots at KWST3

I had a fantastic time at KWST3 this year. There were a lot of firsts for me. The first time I was in New Zealand, the first time I had been to KWST and the first time I met Brian Osman, plus a whole heap of other testers.

I don’t think I’ve been to such a learning event for a while. It was truly a place of inspiration, introspection and challenge. Brian and Colin have already written about their thoughts and I will add mine too in a different post, but first I want to talk about Robots. Oliver asked me to bring over by Big Trak Robots after seeing them in action at CITCON in Sydney.

I split the group into two teams and set them a testing challenge, which involved running experiments in order to determine one of the buttons. What ensued was an hour of fantastic learning, mostly for me, as I watched a group of highly skilled testers apply their minds to the exercise. The testers did some really interesting work. One team started performing fairly complicated tests, which turned out to be a real blessing for them. They then made a model of the functionality and made a hypothesis on what the solution might be.


Team Two took a different tactic. They started with simple tests (a common focusing technique and one I often use) but the tests didn’t offer enough information to the testers. In the debrief that followed at the end, we had a discussion about how though simple tests are quick and easy, sometimes they fail to offer sufficient and meaningful information. Team two, recovered by defocusing and both teams offered their solution.


There’s so many lessons to learn from this exercise, and depending on the group, different lessons are learned. This was one of the most enjoyable times I’ve run it, I think because the testers were skilled and could apply themselves with confidence. They also were very self-aware and so debriefing was more about letting the testers volunteer information than asking probing questions. Or maybe I’m getting better at handing over the learning to those really in charge.

Andrew Robbins and Richard Robinson are running the test lab at Tasting Lets Test and the Robots are going to have their moment in the spotlight there too! See you all there!

Where have all the good jobs gone?

If you follow and believe the twitter conversations, it seems that the main reason for getting ISTQB certification is to pass the screening when applying for work. No ISTQB? No interview!

My personal experience has been that yes, on one or two occasions I’ve failed to be interviewed based on lack of certification. Rather then see this as a negative, I see this as a blessing. After all, if your idea of a tester is that narrow, then I’m probably not suited to your company. On occasion I’ve cited ‘NO CERT’! to justify why I can’t apply for a role. “I’m so sorry, I’m not ISTQB certified…” as I edge my way to the door.

I think it’s naive to rely on a certification as a means to getting work. Especially if you are a thoughtful and intelligent tester who cares about the quality of your work, and who wants to be taken seriously in the industry.

But forget about certification for one minute. Are you seriously willingly going to put your career into a strangers hand who then decides your fate by a keyword? There are cleverer ways to play this game.

Have you not noticed that the way companies recruit is rapidly changing? To get a job that interests you, its not good enough to send in your resume or hold a silly piece of paper with a over ornate stamp on it. Those days are long gone. Now you need passion, you need to keep up to date with whats happening in your field, you need to be committed to keeping yourself relevant.

Our family has experienced this first hand when my husband was looking for work last year. He suddenly discovered at the ‘old age’ of 42 that he was unemployable. He is intelligent and personable(yes I am biased) with a first class degree in Electrical Engineering and he had made the assumption that good people always find work. But that’s not enough. Today, if you want a job that’s worthy of you, you need make sure you earn the companies respect.

This is not only based on my personal experience, I’ve spoken to many recruiters in the last few months, and all seem to have similar stories. Companies are more reluctant to use recruitment agencies to find their staff. They want recruiters who know and understand the specific skills they are seeking. I’m seeing recruiters leave the industry, or re-invent themselves as specialists in one field. Other ways the industry has changed is that many recruiting companies work for one company and are in effect the procurement arm of the company.

Of course, the testing industry has changed significantly too. Now that the major consultancies have successfully sold testing as a commodity, testing (or an excuse for testing) is being performed wherever people are cheapest. The adoption of Agile as a development process and its dependency on automation has also reduced the need for testers (though this is not necessarily a bad thing). The fact is, there are less testing jobs out there.

That doesn’t mean there are less quality testing jobs though. While its true that the crumbs from the table need to be shared among more, there is still plenty of meat and gravy at the table. The question is, have you earned a spot there? Here’s a fact. You are not going to earn a spot on this table with a certificate. The path to this table is through credibility and reputation.

My first bit of advice is if you want a worthy job, then you need to be worthy. Examine the work that you have done to date. Does it reflect your skill and perhaps more importantly, your ability? What about your attitude? Does your work reflect that of someone who is passionate and who loves testing? You don’t need rockstar status, but you do need to be an eager apprentice. If you have aspirations to get a great testing job, but you’re not prepared to put in the hard work, then why should you deserve a great role?

Here’s something I do that has proven to be very useful. When I start a new role, I ask myself two questions. The first is “If I leave, how do I want people to remember me”? and second is “what legacy do I want to leave behind me”? It may sound ruthless to think about an exit strategy when starting a role, but the reality is, NO job is permanent so why treat it as one?

So, you are now a worthy tester, the next step is to be able to demonstrate this worthiness and please, put away that tired old resume! I’m talking about blogging, and speaking and contributing to the testing community. Contributing to the community is a great way of meeting local and international people and you learn so much. Regarding speaking, this doesn’t have to be large conferences, there are plenty of small local meetups that offer you a space to speak. No tester meetups near you? Why not create one, or speak at a developer meetup.

Thirdly you need to network. I can’t emphasis this enough. This is where the jobs are. You need to consider two types of networking, local and online. Local is essential if you want to find work in your area. This means meeting people face to face at the local meetup. Yes, I know masterchef is on a Tuesday night and this clashes with the meetup, but hey, do you want a great job or not? Online networking is important too because it allows you to connect with like minded people, plus its a great source of learning. Many jobs come from both online and local networks.

You also need to research. Find out the good companies, speak to people through your network (not agencies) about the ‘good places’ to work, and make a plan on how you are going to work there. Having an online presence helps a lot here, but so does face to face networking. And be patient, great jobs don’t just drop off trees and fall into your lap.

Yes, ultimately, these jobs don’t come easy. They require hard work, and a willingness to put yourself out there. It comes at a cost to your personal lifestyle. but hey! It’s all about choice. Great jobs are around, but its about seizing the day and making the opportunity instead of relying on an agent to do it for you. This is a good thing. Trust me. As I said at the start, why should you put your fate into someone’s hands?

The good news is more than ever before, companies who recognise and value their staff, who recognise and value quality testing, are recruiting in a grass roots way. If you want these types of jobs, it’s easier to get them.

Now maybe this all seems like too much work and you know? I can live with that! Seriously, its your call. But don’t tell me that certificate is mandatory to work in testing, because its not true. Many companies who ‘get’ testing will hire you without a cert. The question that is probably more pertinent is: “Do they want you?”

So get out there, work your butt off and then market your fabulous testing skills. If you stop putting your pearls before swine, one day that dream job will be yours!

That coverage problem

Thanks mostly to the BBST classes the AST run, I’ve come to understand and appreciate the impossibility of complete testing and the challenges surrounding ‘complete test coverage’.

I find 100% testing coverage a pretty meaningless phrase. Imagine trying the calculate the amount of water in a swimming pool by multiplying the length and width of the pool. Ha! we’d snort and sniff at the stupidity of the idea. But when test managers, project managers and testers ask for “complete test coverage” that is in essence what they are asking testers to do.

In fact, getting to grips on complete testing is even harder than the pool problem, because at least with a pool, once you know the width, length and depth, the volume can be calculated.

In testing though,the number of models we use to design our tests is limited only by our imagination. To try and measure test coverage is a bit like trying to cover a black hole with a tea towel and say you’ve measured it.

Trying to test like this in a short space of time is really stressful and it seems the agile solution is to automate regression tests. But really, is this the best solution we can come up with?

It seems to me this desire to cover as much functionality as possible ends up in this Wac-A-mole game with testers frantically hitting features as fast as possible in order to be able to say its done.

Well, I’m trying a thought experiment at the company I’m consulting at. I’m challenging everyone to drop the idea of coverage and instead focus on some meaningful and valuable testing.

I’m doing this because it seems to me, we testers are far too hung up on this coverage problem and we bias our testing to try and solve it. Yes, this may mean that we fail to test all the functionality – quelle horreur! But get this, when do we *ever* test all functionality?

Dropping the desire to “test everything” is very liberating. We no longer have to fret about trying to test all the functionality in a week (we do weekly releases with multiple hot fix releases). Instead, it’s freed our minds up to reflect and ponder on what is the most beneficial testing to perform given we only have a few days to test. It’s also freed up our toolsmith, allowing them to spend time solving some testing problems through testabilty instead of frantically creating regression tests.

I’m fully expecting some bugs will get released to production, but you know? that’s nothing new! We’re finding bugs in production all the time, they’re called customer support issues.

Time will tell if this approach proves to be beneficial or not. It’s a little scary, dropping the regression test safety net, but when safety nets start obstructing your work its time to question its validity.

Training Testers

I’m having a complete blast at the moment adding the finishing touches to my upcoming workshops in Dublin,Belfast and London.

The Dublin and Belfast Exploratory Workshops sold out in a couple of days but there are still some seats on the coaching testers workshop in London. This is shaping up to be a great workshop – its jam packed with exercises and I’m very excited to be able to share with other testers the approaches that James Bach & I have honed over the past years.

Then its off to the inaugural Lets Test Conference in Stockholm where I’ll be giving a talk on coaching testers.

The Lets-Test conference is shaping up to be a huge event and personally I’m thrilled to be making part of history by speaking there.

Hope to catch up with you at one of these events!

The Buzz On Testing

We’re witnessing a revolution my dear comrades. The tide is turning on drone testing  The word is out. Skill Matters!

Classes such as Rapid Software Testing (developed by James Bach and Michael Bolton), Just in Time (Rob Sabourin) and Elizabeth Hendrikson Exploratory Testing classes are becoming more popular.  They say “Test is Dead” but I say “Bad Test is Dead”.  Hurrah!

Slowly the realisation that tester’s need skills as such as bug recognition, critical thinking, the art of questioning, influencing and developing strategies to help them effectively test software.

I have a theory that the majority of tester training has been focused on process and documentation because its easy to teach that way. Instead of having to working on skill you simply point to a structure and say “follow that”.

Its much more challenging to develop a tester’s skill.

Skilled people earn respect and rightly so. We admire a skilled musician – even if the music doesn’t appeal to us. Developing skill is hard work. It’s the accumulation of understanding, practice and application. This takes time and effort.

As someone who has worked in the industry for twenty years, I know how hard it can be to allocate time for training. We’re deadline orientated and rightly, a Test Manager’s goal is to complete ‘good enough’ testing with the time and resources available.

Coaching is the antidote to this. It allows the tester breathing space to reflect and work on their skill.

The coaching I perform focuses on developing skill through understanding & practice. It takes into account the testers skill base, context, aspirations and the challenges a tester is facing in their current work.

As opposed to arbitrary training, coaching complements a working environment, and often the problems worked on are those that exist at work.

In conjunction with James Bach, I’ve developed systematic approach to coaching a tester’s skill. This approach is a result of coaching hundreds of students, evaluating transcripts and refining the coaching model.

It’s this model that I will be teaching in my workshops on coaching testers. I’ll be holding a workshop in London on the 4th May, and one in Sydney on the 29th May (with James Bach).

Come and join me!