Today at the Workshop on Open Certification we came up with the following (non-ordered) heuristics that might be useful in test question creation:

1) Plausible buzzwords
2) True but irrelevant
3) Write but for the wrong reason
4) Some fool said it
5) My boss will believe it
6) Two conclusions from the same reason
7) Incomplete reason
8) More detail is typical in the correct answer
9) Confusion test techniques
10) Incorrect application of technique
11) Formally phrased answers
12) Read learning objectives first
13) Variations of the theme to make it more challenging
14) Any time you feel the need to mention a source, then try to reword so we do not need to mention the source
15) Invert the cause and effect
16) Avoid inappropriate or confusion humor
17) The correct tends to be similar to the incorrect answer

There is no context for this list. This is more for distribution purposes for attendees. Others will post details around this later.

The attendees included:

• Scott Barber


• Tim Coulter


• Zach Fisher


• Dawn Haynes


• Doug Hoffman


• Andy Hohenner


• Paul Holland


• Kathy Iberle


• Karen Johnson


• Michael Kelly


• Phil Kos


• Baher Malek


• Ben Simo

I'm not normally a fan of models like this, but I've actually found myself using these ideas on a regular basis now. And it's not only because I'm back in the corporate world. I also used this mnemonic at the last IWST to better understand the potential causes of a problem as well as the relative effectiveness of a suggested corrective action.



I recently read RoadMap: How To Understand, Diagnose, and Fix Your Organization by Dean Meyer. This book covers Meyer's five systems for an organization:

Culture: The behavioral patterns (habits and conventions) generally adopted within the organization.

Structure: The definition of jobs and the reporting hierarchy (organization chart), as well as the processes that combine people into teams as workflows across organizational boundaries.

Internal economy: The budgeting, priority-setting, pricing (chargebacks), project-approval, and tracking processes that determine how resources flow through an organization and to its clients.

Methods and tools: The procedures, methodologies, skills, and tools that people in an organization use.

Metrics and rewards: The feedback loops that let people know how they are doing so they can adjust their behavior, and the incentives for improving performance.

I've found a handy little way to remember it: ISCMM

Internal economy
Structure
Culture
Methods and tools
Metrics and rewards

I can remember that because IS reminds me of Information Systems (aka IT, the types or organizations that Meyer is talking about) and CMM reminds me of bloated process models that large companies grab on to (and this model has a lot of process and process-like aspects).

Now, armed with this, when I look at a problem and try to think of a solution, I quickly run down the list and try to make sure there's not something I'm missing from the larger picture.