Automation Bias

Yesterday at WHET #4, James Bach mentioned automation bias. A bit of research turned up Automation Bias in Intelligent Time Critical Decision Support Systems by M.L. Cummings.

While humans are typically effective in naturalistic decision making scenarios in which they leverage experience to solve real world ill-structured problems under stress, they are prone to fallible heuristics and various decision biases that are heavily influenced by experience, framing of cues, and presentation of information. For example, confirmation bias takes place when people seek out information to confirm a prior belief and discount information that does not support this belief. Another decision bias, assimilation bias, occurs when a person who is presented with new information that contradicts a preexisting mental model, assimilates the new information to fit into that mental model. Of particular concern in the design of intelligent decision support systems is the human tendency toward automation bias, which occurs when a human decision maker disregards or does not search for contradictory information in light of a computer-generated solution which is accepted as correct. Operators are likely to turn over decision processes to automation as much as possible due to a cognitive conservation phenomenon, and teams of people, as well as individuals, are susceptible to automation bias. Human errors that result from automation bias can be further decomposed into errors of commission and omission. Automation bias errors of omission occur when humans fail to notice problems because the automation does not alert them, while errors of commission occur when humans erroneously follow automated directives or recommendations.

Lots of good stuff in there:

• confirmation bias takes place when people seek out information to confirm a prior belief and discount information that does not support this belief


• assimilation bias occurs when a person who is presented with new information that contradicts a preexisting mental model, assimilates the new information to fit into that mental model


• automation bias occurs when a human decision maker disregards or does not search for contradictory information in light of a computer-generated solution which is accepted as correct