As I dug into the meaning behind the term automation bias, I became more horrified. The benign definition is as follows: “Automation bias is the propensity for humans to favor suggestions from automated decision-making systems and to ignore contradictory information made without automation, even if it is correct.” (1)

Horror #1 because this implies we are now in an age where we trust the machine more than we trust instinct and fact. We ignore gut feelings and certainties because we think the machine knows better.  And as James Bridle most recently put it, “Automation bias means that technology doesn’t even have to malfunction for it to be a threat to our lives” (2)

Horror #2 because this concept is easily abstracted out of our day-to-day lives and its social impact.   I have written about the age of abstraction and the dangers of putting complex things in a black box. Black boxes are scary because the info they house are viewable to a select audience. (control)

Horror #3 because when you couple automation bias with black box abstraction, the path to the final source of truth is obfuscated. No single person has a full view. it’s unclear if we would understand everything even if we could see all the parts anyway (oh joy) — but being abstracted out of both logic and code erases all possibility. 

We need to be in the era of certainty and clarity not doubt and abstraction.