Over the weekend I was telling Jon Bach about a problem my wife and I have. We have Cicada Killer Wasps. They are big, scary, and when you see them, you run. Worse, they are hard to kill and they are really mean. This made both Jon and I think of James Bach's dead bee heuristic and we developed a new one for bigger and scarier bugs like Cicada Killer Wasps.

The dead bee heuristic works like this, when you see a bee in your living room, you run for something to kill it. When you get back and you don't see the bee any longer, you don't just say, "Oh, I guess he left." and stand down. No you say, "That darn bee's here somewhere..." and you stay on high alert! You wait for the bee to come back. The dead bee heuristic says I need to see the dead body or the bee fly out the window before I stop looking.

This applies to testing in the following way. If you find a bug, and you apply the dead bee heuristic, you will keep trying to reproduce the bug until you find it (dead body) or until you convince yourself it was truly user error (fly out the window).

Now what about Cicada Killer Wasps? I came up with a new heuristic. If the bug's big enough, outsource the problem. I thought of this because I thought about how I don't want to kill the Cicada Killer Wasps because all the scenarios involve me getting stung - badly. That made me think of testing problems I don't want to solve because they may involve me getting stung - badly.

For example, I may be working on a project that needs configuration testing. I'm not equipped to do configuration testing! I don't know how to kill configuration testing bugs. I don't have the tools or the highly skills configuration testing bug hunters. So, like with Cicada Killer Wasps, it may make sense for me to call in the configuration testing experts and outsource my configuration testing to a test lab (like Jon's).

That way I can be reasonably sure I don't get stung.

On a recent project where we were implementing a service oriented architecture (for the first time on a large scale) we wanted to leverage the design of the system to test early. The plan was to have testers run local servers, get the latest complete code (after developers were done and everything was unit tested), and run component level tests based on risks identified by the test team. We would use automation and manual testing to flush out edge conditions, mapping errors, and un-captured exceptions.

On schedule, several of the developers indicated their code was complete at a status meeting. I configured my local server to run the code and began my testing. Surprisingly (or perhaps not) this caused some problems. Developers were quick to come over and say things like:

• "You can't test in a development environment!"


• "We're not done with that functionality (even though we said we were)."


• "I'm still unit testing, the code won't work until I'm done unit testing."


• "We are working over 30 tickets, things can break at any time."

This was taken by some people on the project as a reason to stop our test-often test-early strategy. I would think each of these indicate a reason why you would want someone testing in the development environment. Let's address these one at a time:

"You can't test in a development environment!"

Actually I can. It's really easy. I just update the code, compile, and start a server. We have the technology. Is there some reason I shouldn't test in development? Isn't the whole idea to find issues earlier when they are supposedly easier and cheaper to fix?

Ok, that's my smart ass inner voice. That's not what I really said (but I really wanted to). I think I really said something like:

"I understand this is something we are doing for the first time on this release. Changes like this are painful as the team learns to deal with new ways of working. We need to test in development if we want to have any chance of hitting our release date. If we wait until all the development is done, we will need to do two months worth of testing in three weeks. We just can't do that.

Can you help me make this easier on you? What can I do to lower the visibility this adds to your work and the pressure that added visibility results in? How can I provide feedback to you in a way that doesn't cause problems for you?

Once the developers (or most of them) accepted we were going to do this testing, they started to warm up to it. After the first couple of weeks (very emotional weeks) we actually hit a groove. By the end of the release, we had very close relationships with many of the developers.

"We're not done with that functionality (even though we said we were)."

Our testing exposed a misunderstanding about project progress. What management thought the status was and what the actual status was did not match. We were told it was done, but it was not. This enabled management to correct their misunderstanding of project status so they could plan and react accordingly. This - aside from all the testing related issues identified - is valuable information. People hold meetings and purchase expensive tools to find out what we found out in 5 minutes of testing and asking questions.

Code does not lie. It works or it doesn't.

"I'm still unit testing, the code won't work until I'm done unit testing."

This is closely related to the above point. Up to this point, developers were in the habit of telling management they were done when they were done coding, not done unit testing. This caused a perception problem. I also have a suspicion that this was a sneaky way of asking for more time in a politically correct way. Who's going to say you can't do more unit testing?

Some of them were reporting they were still unit testing when they were still writing code. They looked farther along then they were. On the other hand, the developers who really were unit testing when we started our functional testing welcomed our scrutiny. "Find a bug if you can!", they said. Sometimes we did.

"We are working over 30 tickets, things can break at any time."

This is the, we break the build on a regular basis argument. That's a fine argument, except that like most teams, we're not suppose to break the build. Ever.

This statement means that developers regularly checkout code, make changes, then check it back in, and then test. This was a "deeper" problem. The developers should have been testing the code before they checked it back in. Some were not. There was an automatic expectation of failure on any code that was checked in, not an expectation of it working. Note, this expectation was the expectation of the development team.

This problem solved itself as we started reporting those defects (normally the same day). The added visibility provided by our defects for these types of issues tipped us into a more positive workflow of testing before integration.

All said and done, we worked side by side with the development team for a little over two months. When our testing stopped, the traditional functional and system level testing began. We had automated hundreds of tests at the component level. We had validated many of the mapping documents. We had become active participants in design meetings (not all of them, but enough that we felt like we had a small victory). And by the end of our testing, we had developers (again not all of them) coming over to us and asking us to review their work. After the initial growing pains of the added visibility to the status of the code, most of the emotions died down, and we were able to release some great software.