I was recently involved with testing greats Matthew Heusser, Jonathan Kohl, Jonathan Bach, Paul Carvalho, Danny Faught, and Michael Bolton in an email discussion on credibility in bug reporting. In passing, I mentioned that I like to save up my "trivial" defects and submit them in batches (for various reasons).

This sparked the question, "That's an interesting take. How do you decide if a bug is trivial?"

The following is my answer...

Well, I'm not sure how I do that. I've never thought about it.

I guess I might do something like this:

1. Is it like a past bug? (I'm thinking they might already know how to fix it.)


2. Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)


3. Is this something I would care about as a user of the product? (I'm thinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)


4. Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)


5. Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)


6. Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)

I'm sure there are more, but that's what I can think of now...

I was then treated to a clinic on heuristics by Michael Bolton. My statements are in normal text. Michael's comments are in bold.

--------

Well, I'm not sure how I do that. I've never thought about it.

You apply heuristics; it's just that you're not particularly conscious of them.

0) On anything other than the strongest possible conviction that it's trivial, I assume that it isn't.

I guess I might do something like this:

1) Is it like a past bug? (I'm thinking they might already know how to fix it.)

That's the "consistency with history" heuristic, which is often very powerful.

2) Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)

That's the I Am But A Simple Tester heuristic. That heuristic gets weaker and weaker as you develop your programming skills and your imagination.

3) Is this something I would care about as a user of the product? (I'mthinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)

You can strengthen this by imagining at least three customers, each with different personas and agendas, who might be annoyed by this bug, each for different reasons. If you can do it, it's probably not trivial.

4) Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)

Not only that, but while being incredibly smug about the bug I was seeing, I'd be oblivious to a different one pointed out by James; I'd forget to report a different one one that Jon provided a beautiful description for; I'd waste time writing too much about it, when I could have included the annotated screenshot that Paul provided; I'd be unable to write a quick script that would exercise the other 9999 entries in the table, as Danny could; and even if it were something really cool and instructive, I'd neglect to write about it in my blog with a compelling story about how both the developer and I learned something, as Jonathan Kohl would.

But thanks for the nod, even if you were joking. :)

5) Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)

In this case, I include an appropriate disclaimer; "I recognize that this might seem nit-picky; here's why I think it's something more than that." (And then I try to apply Weinberg's Rule of At Least Three; if I can come up with at least three reasons why it's not nit-picking, it probably isn't.)

6) Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)

I'm sure there are more, but that's what I can think of now...

Note that thinking about more of them in collaboration with other people is a great and revealing exercise.

I recently visited a friend who presented me with the following problem from Mensa Logic Puzzles:

There is logic behind the distribution of numbers in the grid. Work out what it is and then fill in the missing numbers.

When he first presented me with the problem, I spent about half and hour staring at the grid. I attempted to perform all sorts of arithmetic operations on the data to try to find the pattern - but with no luck. After half and hour I gave up and told him I would sleep on it and solve it the next day.

The next day, when he gave me the problem again, I thought of something James Bach told me the last time we got together, "If you have data then graph it." With that in mind, I flipped open the laptop, fired up Excel, and came up with the following:

Solution One

The first thing I attempted to do was sum up the rows and columns. I found a repeating pattern for the rows: 35, 27, x, 35, x, 35. I quickly found that I could make that 35, 27, 35, 35, 27, 35. But that solution had two problems. First, the columns didn't add up to and sort of pattern. Second, the numbers in row five could be several variations to get six (2 and 4, 3 and 3, 4 and 2, 1 and 5, etc...).

Solution Two

Next I tried looking at evens and odds. I was looking for some sort of pattern to the way they were distributed. I quickly gave up on that...

Solution Three

Finally, I color coordinated the data. I gave each number its own color. It was then that I noticed that each color seemed to appear the number of times as the number I chose it to represent. When I looked at the difference between the number and the number of times the color appeared, I had a total difference of four. So I figured I had my numbers, now how do I put them in?

I then noticed that no color was touching itself. When I started plugging in the numbers I thought solved the problem, there was only one way they would fit that would keep them from touching their own color.

I had solved the problem. I gave him my answer, and he was shocked to learn that I had solved it correctly.

When trying to notice patterns in your testing, if you have data - graph it.