Skilled Bug Investigation

At STAREAST 2004, I had the pleasure of attending Robert Sabourin's keynote A Whodunit? Testing Lessons from the Great Detectives.

In this keynote, Rob listed several detectives, discussed their problem solving style and techniques, and then related their skills to the skills that testers use everyday. It was entertaining and enlightening, and it was easily the best keynote I have ever seen.

In the keynote, Rob recommended reading detective novels and watching old detective TV shows as a fun way of developing your skills and looking at testing case studies. Each detective he presented was presented to show off a particular style of testing. For exploratory testing and bug isolation, Rob used Sherlock Holmes for illustration of the deductive reasoning that is crucial for those skills. I was so motivated, I went out and purchased no less then the complete anthology of Arthur Conan Doyle's works on Sherlock Holmes.

Fast forward a couple of months. I have read several Sherlock Holmes stories and have found them entertaining and enlightening. Last week I stumbled across a presentation titled Skilled Testers and Their Enemies that James Bach gave at Trends in Testing: Theory, Techniques and Tools 2004 at the University of Washington and Microsoft Research Summer Institute.

In this presentation James has a slide titled "Skilled Bug Investigation." Feel free to check out the entire presentation, but here are the main contents of that slide:

Skilled Bug Investigation

---

Identify

• Notice a problem.


• Recall what you were doing just prior to the problem.


• Examine symptoms of the problem w/o disturbing system state.


• Consider possibility of tester error.

Investigate

• How can the problem be reproduced?


• What are the symptoms of the problem?


• How severe could the problem be?


• What might be causing the problem?

Reality Check

• Do we know enough about the problem to report it?


• Is it important to investigate this problem right now?


• Is this problem, or any variant of it, already known?


• How do we know this is really a problem?


• Is there someone else who can help us?

I was not at the presentation, so I can't speak to the intent of the slide, but it seems clear that James is implying that a skilled investigator might follow these steps.

At this point, everything clicked: Rob's presentation, James's slide, and Sherlock's skills at investigation. The following is a detailed example for each skill identified by James, using A Study in Scarlet by Arthur Conan Doyle. If you are already familiar with the text, read on. If not I would suggest finding a copy or reading the online version. It is a great story and is worth the time.


Notice a problem.

The problem is identified at No. 3, Lauriston Gardens:

"At present my attention was centred upon the single, grim, motionless figure which lay stretched upon the boards, with vacant, sightless eyes staring up at the discoloured ceiling. It was that of a man about forty-three or forty-four years of age, middle-sized, broad-shouldered, with crisp curling black hair, and a short, stubbly beard. He was dressed in a heavy broadcloth frock coat and waistcoat, with light-coloured trousers, and immaculate collar and cuffs. A top hat, well brushed and trim, was placed upon the floor beside him. His hands were clenched and his arms thrown abroad, while his lower limbs were interlocked, as though his death struggle had been a grievous one. On his rigid face there stood an expression of horror, and, as it seemed to me, of hatred, such as I have never seen upon human features. This malignant and terrible contortion, combined with the low forehead, blunt nose, and prognathous jaw, gave the dead man a singularly simious and ape-like appearance, which was increased by his writhing, unnatural posture. I have seen death in many forms, but never has it appeared to me in a more fearsome aspect than in that dark, grimy apartment, which looked out upon one of the main arteries of suburban London."

Recall what you were doing just prior to the problem.

Just before the problem statement above, Holmes was examining the path leading up to the house:

"I had imagined that Sherlock Holmes would at once have hurried into the house and plunged into a study of the mystery. Nothing appeared to be further from his intention. With an air of nonchalance which, under the circumstances, seemed to me to border upon affectation, he lounged up and down the pavement, and gazed vacantly at the ground, the sky, the opposite houses and the line of railings. Having finished his scrutiny, he proceeded slowly down the path, or rather down the fringe of grass which flanked the path, keeping his eyes riveted upon the ground. Twice he stopped, and once I saw him smile, and heard him utter an exclamation of satisfaction. There were many marks of footsteps upon the wet clayey soil; but since the police had been coming and going over it, I was unable to see how my companion could hope to learn anything from it. Still I had had such extraordinary evidence of the quickness of his perceptive faculties, that I had no doubt that he could see a great deal which was hidden from me."

This examination yielded:

"The very first thing which I observed on arriving there was that a cab had made two ruts with its wheels close to the curb. Now, up to last night, we have had no rain for a week, so that those wheels which left such a deep impression must have been there during the night. There were the marks of the horse's hoofs, too, the outline of one of which was far more clearly cut than that of the other three, showing that that was a new shoe. Since the cab was there after the rain began, and was not there at any time during the morning -- I have Gregson's word for that -- it follows that it must have been there during the night, and therefore, that it brought those two individuals to the house."

and:

"I'll tell you one other thing, patent-leathers and Square-toes came in the same cab, and they walked down the pathway together as friendly as possible -- arm-in-arm, in all probability. When they got inside, they walked up and down the room -- or rather, Patent-leathers stood still while Square-toes walked up and down. I could read all that in the dust; and I could read that as he walked he grew more and more excited. That is shown by the increased length of his strides. He was talking all the while, and working himself up, no doubt, into a fury. Then the tragedy occurred."

Examine symptoms of the problem w/o disturbing system state.

Holmes examined the room at No. 3, Lauriston Gardens:

"As he spoke, he whipped a tape measure and a large round magnifying glass from his pocket. With these two implements he trotted noiselessly about the room, sometimes stopping, occasionally kneeling, and once lying flat upon his face. So engrossed was he with his occupation that he appeared to have forgotten our presence, for he chattered away to himself under his breath the whole time, keeping up a running fire of exclamations, groans, whistles, and little cries suggestive of encouragement and of hope. As I watched him I was irresistibly reminded of a pure-blooded, well-trained foxhound, as it dashes backward and forward through the covert, whining in its eagerness, until it comes across the lost scent. For twenty minutes or more he continued his researches, measuring with the most exact care the distance between marks which were entirely invisible to me, and occasionally applying his tape to the walls in an equally incomprehensible manner. In one place he gathered up very carefully a little pile of gray dust from the floor, and packed it away in an envelope. Finally he examined with his glass the word upon the wall, going over every letter of it with the most minute exactness. This done, he appeared to be satisfied, for he replaced his tape and his glass in his pocket."

and the body:

"As he spoke, his nimble fingers were flying here, there, and everywhere, feeling, pressing, unbuttoning, examining, while his eyes wore the same far-away expression which I have already remarked upon. So swiftly was the examination made, that one would hardly have guessed the minuteness with which it was conducted. Finally, he sniffed the dead man's lips, and then glanced at the soles of his patent leather boots."

Consider possibility of tester error.

Holmes is certainly not the poster child for modesty. In fact, in this case he is completely confident of his abilities. It should be noted that in others he is not always so confident. In this tale, the closest I could come to admitting error is the following:

"'Commonplace,' said Holmes, though I thought from his expression that he was pleased at my evident surprise and admiration. 'I said just now that there were no criminals. It appears that I am wrong -- look at this!' He threw me over the note which the commissionaire had brought."

How can the problem be reproduced?

The cause of death (which could be considered the bug by some) was poison. Here Holmes proves it:

"Give them here," said Holmes. "Now, Doctor," turning to me, "are those ordinary pills?"

They certainly were not. They were of a pearly gray colour, small, round, and almost transparent against the light. "From their lightness and transparency, I should imagine that they are soluble in water," I remarked.

"Precisely so," answered Holmes. 'Now would you mind going down and fetching that poor little devil of a terrier which has been bad so long, and which the landlady wanted you to put out of its pain yesterday?"

I went downstairs and carried the dog upstairs in my arms. Its laboured breathing and glazing eye showed that it was not far from its end. Indeed, its snow-white muzzle proclaimed that it had already exceeded the usual term of canine existence. I placed it upon a cushion on the rug.

(...)

With a perfect shriek of delight he rushed to the box, cut the other pill in two, dissolved it, added milk, and presented it to the terrier. The unfortunate creature's tongue seemed hardly to have been moistened in it before it gave a convulsive shiver in every limb, and lay as rigid and lifeless as if it had been struck by lightning.

What are the symptoms of the problem?

"There has been murder done, and the murderer was a man. He was more than six feet high, was in the prime of life, had small feet for his height, wore coarse, square-toed boots and smoked a Trichinopoly cigar. He came here with his victim in a four-wheeled cab, which was drawn by a horse with three old shoes and one new one on his off fore-leg. In all probability the murderer had a florid face, and the finger-nails of his right hand were remarkably long. These are only a few indications, but they may assist you."

How severe could the problem be?

"Any delay in arresting the assassin," I observed, "might give him time to perpetrate some fresh atrocity."

What might be causing the problem?

Lestrade and Gregson glanced at each other with an incredulous smile.

"If this man was murdered, how was it done?" asked the former.

"Poison," said Sherlock Holmes curtly, and strode off.

Do we know enough about the problem to report it?

"I have now in my hands," my companion said, confidently, "all the threads which have formed such a tangle. There are, of course, details to be filled in, but I am as certain of all the main facts, from the time that Drebber parted from Stangerson at the station, up to the discovery of the body of the latter, as if I had seen them with my own eyes. I will give you a proof of my knowledge."

Is it important to investigate this problem right now?

While not the answer we might look for, this topic does come up. Holmes does make a conscious decision of determining if the problem is important to him:

"I was amazed at the calm way in which he rippled on. 'Surely there is not a moment to be lost,' I cried, 'shall I go and order you a cab?'

'I'm not sure about whether I shall go. I am the most incurably lazy devil that ever stood in shoe leather -- that is, when the fit is on me, for I can be spry enough at times.'

'Why, it is just such a chance as you have been longing for.'

'My dear fellow, what does it matter to me? Supposing I unravel the whole matter, you may be sure that Gregson, Lestrade, and Co. will pocket all the credit. That comes of being an unofficial personage.'"

Is this problem, or any variant of it, already known?

"'Then, of course, this blood belongs to a second individual -- presumably the murderer, if murder has been committed. It reminds me of the circumstances attendant on the death of Van Jansen, in Utrecht, in the year '34. Do you remember the case, Gregson?'

'No, sir.'

'Read it up -- you really should. There is nothing new under the sun. It has all been done before.'"

or perhaps:

"The forcible administration of poison is by no means a new thing in criminal annals. The cases of Dolsky in Odessa, and of Leturier in Montpellier, will occur at once to any toxicologist."

How do we know this is really a problem?

Holmes's user community was the police force. It was his problem because it was thiers:

"Still I recognized that justice must be done, and that the depravity of the victim was no condonement in the eyes of the law."

Is there someone else who can help us?

"I telegraphed to the head of the police at Cleveland, limiting my inquiry to the circumstances connected with the marriage of Enoch Drebber. The answer was conclusive."

or perhaps the help of "half a dozen of the dirtiest and most ragged street Arabs":

"There's more work to be got out of one of those little beggars than out of a dozen of the force," Holmes remarked. "The mere sight of an official-looking person seals men's lips. These youngsters, however, go everywhere and hear everything. They are as sharp as needles, too; all they want is organization."

While certainly this was probably not a practical use of my time, it was fun to identify each portion of the thought process identified by James. It would seem good thinking is timeless!