Testing is questioning

I think it was Jon Bach's work on open book testing that really drove the analogy of every test being a question home for me. Each test you execute is really just a question you're asking the software. If you can think of a question to ask, you've though of a test to execute. It's elegant in it's simplicity.

So what does this mean for your testing? Because for each test you're thinking of running, you can use this analogy as a tool to develop a better test:

• Before you run each test, in your mind, clearly articulate the question that you're asking. I've found that this not only can help clarify for me what I'm really trying to accomplish with any given test, but it also helps me think of things that I need to verify after my test has been executed. Once I think of a test as a question, I recognize that there might be multiple ways to answer it. That gives me multiple things I need to verify (like log files, database entries, etc...).


• Before you run each test (or more practically, any given set of tests), try to express the tests in several different ways to help clarify them. This is an attempt to clarify the scope of the test. For example, what do I really mean when I say "Ensure that when you stack an utterance you can't bypass required script logic?" Does it matter what I stack? Does it matter what area of script logic? How would I know if I bypassed it successfully? Is there a way for it to be partially bypassed? How would I know? How else could I phrase the question my test is asking to help make these things more clear?


• It can also be helpful to ask if any given test should be broken out into smaller tests. Are there smaller and simpler questions that could be asked? What are they? Do they tell you more or less?


• Finally, the last thing you can do is ask yourself if you test might have multiple right answers. Does pass really mean pass? And does fail really mean fail? What conditions could take place that might make you change your answer? While this might not change this specific round of testing, it might give you ideas for other tests.