Pair testing

Pair testing can be challenging for a several reasons, not the least of which is just finding the time for it. But I still try to do it when I can. It's more difficult now that I'm a manager. I haven't really tested in months, much less pair tested. I need to work on that... but I have other things I need to work on too. Always to much to do.

When I'm pair testing with someone for the first time, I'm looking for some specific things:

• How does the person I'm pairing with like to work? Do they take notes realtime? Do they use multiple screens and laptops, multitasking their work? Do they use tools? If so which ones?


• How does the person I'm pairing with like to communicate? Do they talk out loud? Are the good at stream of conscious speaking? Do they IM a lot? Do they write stuff down, and if so, what?


• How does the person I'm pairing with like to test? Do they use scripts or charters? Do they have data handy? In what format, a spreadsheet, text files, database, other? Can I see them using specific test patterns, heuristics, or techniques? As their testing unfolds, is it always the same or does it change? How does it change?


• What does the person I'm pairing with value? What's a bug to them? How do they assess priority? Who do they view as their stakeholder(s)? What information is credible? What oracles are credible? What parts of the system do they focus on?


• How does our dynamic come into play while we are pairing? If they are a programmer (testing their own code or not), what words do they use or concerns do they exhibit that would tell me that by their testing? If they are a tester, what words do they use or concerns do they exhibit that would tell me that by their testing? If I manage them, does that affect their behavior in a noticeable way? (And how would I notice?)


• How do I behave while we're testing? Do I take control? Not speak up? How do I feel as we test (about the product, our testing, my interaction with the other person)?


• What do we do after we test? Do we combine and clean up our notes? Does one of us write a script (manual or automated) for anything we tested? What defects get written up, which ones don't, and what information goes in them? What follow up charters do we think of? Do we both do a debrief?

I suppose that somewhere in there I also find time to think about the testing that I'm suppose to be doing. But I can't be bothered by those details. :)

Comments

[JessD]

Excellent post. This really got me thinking not so much about what the other person might be doing (since we don't have enough testers to pair off), but it was a nice list of things to consider about my own work.

(BTW, your captcha is insanely difficult to read.)

[Mike Kelly]

I know, I know.... I get a lot of captcha comments. I need to switch it. I don't like it either, but it was super easy to setup and run.

Thanks for the comment. One of the things I do when I'm working on a team where there aren't other testers to pair off with is pair with a programmer, PM, or business user. The perspective can often help me and the other person.

[Simon]

Hi Mike,

Interesting post. I asked a couple of guys in my team to try testing in pairs, using my knowledge or it and some of the information avaialble on the internet as a basis on which to try it. Unfortunately, I didn't get to do what I wanted to achieve as I had planned to sit with them and observe how it was working but I do have a session with them, and their test lead, booked to review how it went. It'll be interesting to see if their feedback relates to some of the above. Infact, I may use some of your post to guide discussion.

Cheers,

Simon

[Mike Kelly]

Simon, that's great! I hadn't thought about using the list to help with a debrief, but now that you bring it up, I can see it helping for teams new to pairing.

Thanks for pointing that out.
-Mike

[Nancy Knettell]

As a Former Developer, Senior Level Development Manager, Program Manager for both large and small companies, and now a Quality Assurance Professional I am not and never will be a proponent of Agile in its present form. Having taken companies who was barely able to release 1.5 a year to 18 the year later with 80% reduction in post release bugs with no increase in staff other than myself, the only aspect of Agile I see that has any value, is small increments of feature sets, never small increments of no documentation or process.

The key to rapid fast paced development is not more undocumented collaboration, but more documented collaboration. Without documentation and respectful collaboration to produce that accurate documentation you lose all potential for communication not only between engineering and qa, but between engineering and the rest of the company.

I often have to remind Development Management that software is not the only product produced by a company. Accurate testing, useful help manuals, training, sales, sales training, marketing literature can all be derived from correct requirements documentation.

So, how do you do this without the process becoming fossilized. You do it in parallel. Say your development team is coding project A. You should be now writing the design specs for Project B. But you don't want your engineers writing them. They hate doing it. They do not know how to write them such that they are useful for QA and the rest of them company since they rarely have been trained or expected to write them correctly.

You hire PM's, qualified tech writers, or good QA people who are trained in the correct methodology to create requirements documents. They write them for project B while the engineers are coding A.

Yes, this requires multitasking on the part of the engineers to start thinking about B while they are doing A. But, what is more important is that the collaboration must be there to. The engineers must be able to communicate their ideas to the documentation individuals with the same respect that they must maintain in a "throw the code over the wall" conversation that I see in your "pairing" recommendation.

What impact is there with this type of methodology. I was able to document complex requirements with a bi-weekly design meeting and spot conversation with the coders. Yes, the developers did the design of the new product, the part they like to do, so I did not rob them of the fun part. But, I did what I know how to do and was trained to do, take high level technical concepts and put them in words that all could understand and use for their products. Is this any more of an intrusion that the so called "pairing" you advocate during the development process?

You will forgive me but I respectfully really do see Agile as cheap, lazy, and disrespectful. You can get so much more with the addition of one more individual who can do the forward looking work and create the documentation so that the engineers can continue to code. I have seen it happen. I was able to produce complete product design documentation, have my qa reports produce testplans and complete automation suites, have sales have their sales training in place as well as produce alert literature for the new product features, have product training have their trianing modules in place even before the development even started on B.

All of the problems that development normally faces as most of the technical challenges were already taken care of before they started coding because the product had been throughly thought out before hand in the process of writing the documents. In fact we were able to do QA on A and Development was able to move to develop B immediately with very few problems. Yes, there were a few changes that needed to be made in B as the engineers went but because the feature set we were working on was small -- the only value of Agile that I see -- the maintenance of the documents was minimal, but yes, still required. The engineers communicated the changes to me. I updated the documents and pushed them out to QA so that they could make changes in their scripts. Rarely was there changes in the rest of the company documentation. As B entered QA training was already happening on B. Sales was already selling B accurate, with no promises that would add to feature creep. After all C was coming in another three to six weeks. No need to wait for a year for a customer to see their pet requirement. I see this as pride in workmanship and the responsibility of a good PM. Now that is what I call an agile (small "a") response to a customer.

Having been forced to read code to determine quality and stability (did the developers code it right) or Use Cases which are now so full of code speak and are uninteligiable to anyone outside of engineering (how does this contribute to good communication if you do not take into account the learning style of your company internal customers) and rarely talk about how the actual software is supposed to work in an error condition, I am always trying to ensure quality at the end of the development cycle and shaking my head. I choose not to work for companies who continue to follow this misguided path.

I will continue to challenge the concept of Agile as it is being championed today. I have been in companies -- Motorola for example -- where Agile has destroy the company's reputation because the products produced are of such poor quality and reliability. Yes, the company can pump out "the goods" faster, and yes the developers are less bored or bothered by task they feel are uneeded, but I have yet to see the value that Agile brings to the table. In fact, the only people that I see support it are developers and people who know nothing about, have not been trained in, or never experienced the efficiency that correct methodology can bring to creating rapid but high quality software releases. I would challenge you to give this method of correct "pairing" a try. I am sure if done correctly you too, will see the light.

Respectfully,

Nancy Knettell

[Mike Kelly]

Hi Nancy,

Thank you for the wonderfully detailed comment. I appreciate the time that must have taken you. However, I'm not sure where to start...

Part of that is my confusion. My post, on pair testing, had nothing to do with Agile. I've pair tested in Waterfall, RUP, Scrum, XP, and ad hoc project methodologies. I've pair tested on FDA regulated projects, where documentation and demonstrative evidence where critically important to the success of the project. Pair testing is one approach I have to testing. It has nothing to do with the project methodology I'm working under.

I pair test for two reasons:

1) I think it makes the product better. I believe the test ideas that come from two people, working collaboratively, are better then the ideas that come from one person. I think pair testing leads to /better/ documentation, since it frees one tester to focus on capturing what really happens and recording what takes place in a more detailed fashion. I think pair testing leads to better coverage, as two different perspectives of risk and application coverage will most likely lead to /more/ testing, or at least, better testing coverage selection based on risk.

2) I think it makes the testers better. If you notice what I look for, I'm looking for opportunities for learning. I'm looking for differences in how I work, so I can figure out if I like the other way better or if there is something I could improve in my work. I'm also looking for opportunities to teach the person I'm pairing with. To share my ideas and techniques. To help them learn from me.

I've worked on successful and failed waterfall projects. I've worked on successful and failed agile projects. I've never viewed the methodology selected for project delivery as the most important factor in project success. (It's on my list, but not very high.)

For me, pair testing is about managing my learning and improving the quality of the testing I do. Regardless of the project methodology. Some aspects of the testing change (format and extensiveness of documentation, freedom of realtime test design vs. following a plan, etc...), but for the most part I'm focused on me, my partner, and the product.

I do think there is a time and place for agile methodologies. But that was not the goal of this particular post. I hope that helps.

Thanks again. I believe your comment and my follow up comment have made this a more valuable posting.

-Mike

[David Christiansen]

Hi Nancy,

When germs were first discovered, the doctor who discovered them was ridiculed by his peers for advocating hand washing. He died in an insane asylum. His "successful" peers didn't see the value in what he was doing either.

I think you've made a fundamentally unfair comparison by comparing your successes to Agile's failures. If we applied your logic, which I believe can be summed up as "because some agile projects fail miserably it is a worthless approach", to the approach you are successful at and devoutly attached to, we would have to come to the same logical conclusion: that your approach is equally worthless.

I don't think I have to trot out the list of massive project failures behind the approach you are advocating - it is very long and the cost of these failures is in the billions of dollars.

I personally have been in involved in four major project failures, three of which exceed $100M dollars, but which followed the approach you described. In each case, the project never delivered a single line of code to production. Almost a third of a billion dollars wasted in my brief decade as a software professional!

Does that mean your approach is worthless? Certainly not. In fact, if you had been running the show, those projects may very well have succeeded. Your approach seems no more worthless than Agile, at least to me.

In spite of your aspersions on the value of agile, there are many, many software development efforts that are using the approach to deliver high quality software cheaply and, best of all, happily. The fact that you've never been involved in one of those doesn't surprise me, but it doesn't sway me either.

I commend you for being so good at what you do. Congratulations. But it's a fatal flaw in your logic to compare the best of your approach to the worst of mine. It's not credible and you should re-think your position.

But don't call us lazy, cheap, and disrespectful just because your approach doesn't work for us. Project management is not a moral debate, no matter how emotionally intense your feelings for your approach may be. You're just being rude and it's unbecoming to someone as obviously well-qualified as yourself.

[Simon]

For those who may be interested (following on from my above comment)...

Background: I asked our testers to use Paired Testing on a project (integrating a WSUS solution with our core product) at a point when we were required to do some testing of the product but didn’t have much in the way of prior product knowledge or any documentation to aid that.

Armed with some information provided from on-line sources and a couple of chats on the basic principles the testers used paired testing to test this integration and below are some of the lessons learnt / observations which they came up with.

Summary: It was absolutely a positive and enjoyable experience with some of the key benefits being a better quality of testing, more idea generation & better bug success. By better bug success, the testers found that by two testers needing to verify a bug any assumptions/ambiguity is removed prior to it being raised with a developer.

On the negative side, the testers did struggle to avoid distractions when performing paired testing – either from people or other priorities and this sometimes hampered their ability to get any rhythm going. They also found that sudden stoppages in testing, e.g. a showstopper defect, were difficult to deal with if you didn’t record exactly where you were, and where you were heading at the point where you stopped testing.

Overall, testing in pairs felt a more natural way to test because the “driver” could get on with his testing without needing numerous pauses for note taking and bug raising. They felt that more testing in a pair with the same person would improve the quality of testing performed because they’d be able to build a relationship and understand some of the questions raised in the initial blog entry. There’s more information which came out of the review and I think I’ll blog about that separately, but for anyone interested in paired testing, I think this gives a good overview of the lessons we learnt when trying it for the first time.

[Mike Kelly]

Simon,

That's great stuff man. Thanks for posting your feedback here. We struggle with uninterrupted time as well. Sometimes you can find a solution that works for the short-term (police tape, red lights, etc...), but over time, people ignore them and interrupt anyway. I don't know how to fix that.

Great stuff!
Mike