Last week I was working on an article for InformIT. As part of the article, I was testing Google Earth using heuristic test oracles (which happens to be what the article is about).

In the course of my testing, I came across an interesting bug where Google Earth lost resolution (the picture became fuzzy when I zoomed in). When I had started my testing, zoom was crystal clear. Indianapolis was a bright shinny city!

Towards the end of my writing session, after switching back and forth between my article and Google Earth repeatedly while doing my testing, I noticed that resolution was not what it once was. Indianapolis was now dull and blurry. Not at all like I remembered it. My inconsistent with purpose heuristic kicked in. I now had another bug for my article!

By this point, I had a good list of bugs. Four or five I think - a good number for an article. But this bug was a beauty! It affected the largest piece of functionality in Google Earth - rendering a map.

Enter the two rules for when you find a bug...



Rule number one: Don't get cocky

I had spent about a total of ten minutes actually testing and I had a nice list of bugs. I was feeling good:

"What do those guys at Google know that I don't know? Hah! I rock!"

Well, as it turns out, they probably know quite a bit more then I do, because I then made an amateur mistake due to my over confidence. A silly mistake that cost me my prize catch. I figured since I had found the others so easy, and clearly I had found this one, I could reproduce the bug. I even thought I knew what the problem was.

I closed Google Earth and restarted it. I tested for about 20 minutes. Nothing. No dice. I tried everything I tried before. No good.

#%!*$

Rule number two: Take a screenshot

Because I didn't follow rule number two, I am now writing about an experience where I made a mistake instead of writing about my cool Google Earth bug I found. All I needed was a screenshot. Simple proof. Maybe not permissible in court, but certainly permissible in blogs - earning me valuable testing "street credit."

Oh well...

Keep in mind that rule number two can be a bit more involved then taking a screenshot. Anything that sets the scene of the crime is good to keep: log files, screenshots, data used, system state, video, DNA samples - whatever. Stay humble and collect the evidence. You'll end up using it in your defect report either way...

I was recently involved with testing greats Matthew Heusser, Jonathan Kohl, Jonathan Bach, Paul Carvalho, Danny Faught, and Michael Bolton in an email discussion on credibility in bug reporting. In passing, I mentioned that I like to save up my "trivial" defects and submit them in batches (for various reasons).

This sparked the question, "That's an interesting take. How do you decide if a bug is trivial?"

The following is my answer...

Well, I'm not sure how I do that. I've never thought about it.

I guess I might do something like this:

1. Is it like a past bug? (I'm thinking they might already know how to fix it.)


2. Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)


3. Is this something I would care about as a user of the product? (I'm thinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)


4. Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)


5. Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)


6. Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)

I'm sure there are more, but that's what I can think of now...

I was then treated to a clinic on heuristics by Michael Bolton. My statements are in normal text. Michael's comments are in bold.

--------

Well, I'm not sure how I do that. I've never thought about it.

You apply heuristics; it's just that you're not particularly conscious of them.

0) On anything other than the strongest possible conviction that it's trivial, I assume that it isn't.

I guess I might do something like this:

1) Is it like a past bug? (I'm thinking they might already know how to fix it.)

That's the "consistency with history" heuristic, which is often very powerful.

2) Can I envision a fix for this that would only take me an hour? (I'm thinking if I can fix it, anyone can.)

That's the I Am But A Simple Tester heuristic. That heuristic gets weaker and weaker as you develop your programming skills and your imagination.

3) Is this something I would care about as a user of the product? (I'mthinking if I don't care - and I typically care about everything - then this is really low risk and probably trivial in terms of priority.)

You can strengthen this by imagining at least three customers, each with different personas and agendas, who might be annoyed by this bug, each for different reasons. If you can do it, it's probably not trivial.

4) Is this something that I would be embarrassed if I didn't report it? (I'm thinking, if Michael Bolton was looking over my shoulder and saw the bug, would he ask my why I didn't log that obviously critical defect yet.)

Not only that, but while being incredibly smug about the bug I was seeing, I'd be oblivious to a different one pointed out by James; I'd forget to report a different one one that Jon provided a beautiful description for; I'd waste time writing too much about it, when I could have included the annotated screenshot that Paul provided; I'd be unable to write a quick script that would exercise the other 9999 entries in the table, as Danny could; and even if it were something really cool and instructive, I'd neglect to write about it in my blog with a compelling story about how both the developer and I learned something, as Jonathan Kohl would.

But thanks for the nod, even if you were joking. :)

5) Is this something I would by embarrassed if I did report it? (I'm thinking will the developer laugh at me for submitting such an obviously nit-picky bug.)

In this case, I include an appropriate disclaimer; "I recognize that this might seem nit-picky; here's why I think it's something more than that." (And then I try to apply Weinberg's Rule of At Least Three; if I can come up with at least three reasons why it's not nit-picking, it probably isn't.)

6) Is this something that's probably already fixed and I just need a new build to find it? (I'm thinking the developer already caught this, I just need to see the new code to confirm it. If not, I'll log it then. Note: perceived priority plays a roll here.)

I'm sure there are more, but that's what I can think of now...

Note that thinking about more of them in collaboration with other people is a great and revealing exercise.