Engineering Topics Special Reports Blogs Multimedia The Magazine Professional Resources Search Risk Factor
Are We Automating Ourselves Into a Corner?
By Robert Charette
Posted 15 Dec 2009 | 5:00 GMT
Most of the research into the ironies and especially the paradoxes of automation over the past thirty years or so have focused on areas where human safety has been an overriding concern and time to recover from a problem is short. Early research centered a lot of attention first to the automation used in aircraft, and then to other forms of transportation, and now to fields like weather forecasting and health care. My IEEE Spectrum on-line story, "Automated to Death," is about the ironies and paradoxes of automation. The ideas were articulated by researchers K.S. Bibby et al. in their 1975 paper, "Man's Role in Control Systems" (Proc. 6th IFAC Congress, Boston) and later expanded upon by Lisanne Bainbridge in her 1983 Automatica paper, "Ironies of Automation."
Quoting from Bainbridge's paper,
"The classic aim of automation is to replace human manual control, planning and problem solving by automatic devices and computers. However, as Bibby and colleagues (1975) point out : '... even highly automated systems, such as electric power networks, need human beings for supervision, adjustment, maintenance, expansion and improvement. Therefore one can draw the paradoxical conclusion that automated systems still are man-machine systems, for which both technical and human factors are important.' This paper [Bainbridge's] suggests that the increased interest in human factors among engineers reflects the irony that the more advanced a control system is, so the more crucial may be the contribution of the human operator."
For instance, the increased use of automation in health care such as cueing aids for cancer screening, e-prescribing systems, and electronic health records have also drawn the attention of researchers in the field of human-centered systems, as the same automation-operator issues that were encountered in the automation of cockpits are appearing here as well.
One research study found, for example, that screeners for breast cancer using computer-aid detection (CAD) software would sometimes use the results displayed not as attention cues (which indicated the screeners needed to examine the potential cancer mass that was detected further) but instead as a decision aid to help determine which patients should be called back for further screening or not. Using the CAD system as a decision aid was not an intended (or foreseen) use by the system's designers.
The over-reliance on automation is also sometimes called alternatively "automation bias" or "automation misuse" and refers to using automation in ways that its designers never intended.
The good news is that previous research in aircraft automation and the like can be applied to these new fields; however, there is also a cautionary note as well.
Charles Perrow, professor emeritus of sociology at Yale University, author of the landmark book, Normal Accidents: Living with High-Risk Technologies states that, "We've made strides in human factors but we're just barely keeping up, if not lagging behind, the engineering and technical strides being made. We've improved human factors but not faster than the need for human factors."
This is increasingly becoming a concern as what once were expensive automated systems become available to the general public. As these systems become more available, we can expect the ironies and paradoxes to become even more visible, along with their unpleasant side effects.
For instance, GPS has moved from use in military systems to cell phones. With its widespread introduction have come reports of people over-relying on what their GPS systems tell them.
In one case a few years back reported by Reuters, a German motorist drove his BWM straight into the Havel River in eastern Germany. The police reported that the driver drove his car past a stop sign, down a ferry ramp and into the river. The driver reported that he was following his car's navigation system, which apparent showed what appeared to be a path across the river, but the system failed to note that it required a ferry to do so.
More recently as I blogged about here, one GPS system told a driver to drive off a cliff, while another recommended that the driver turnaround into on-coming traffic.
Clifford Nass, a communication professor at Stanford University makes the point that humans have a tendency to follow the instruction of machines over their common sense: "Rather than trust our judgment of nature, we let technology tell us what's going on," he says.
In New York, Governor David A. Paterson is trying to fight this tendency the old fashion way: with tough, almost draconian, laws.
New York State officials point out that it is common knowledge that trucks and buses are not allowed on New York State Parkways. Yet for some reason, truckers in particular seem to ignore the law when their GPS units tell them to.
So a few months ago, Gov. Patterson announced legislation he plans to introduce in January 2010 that would require all large commercial trucks to use enhanced GPS devices that route them away from restricted roads. Some 81 percent of bridge overpass strikes by commercial vehicles in the state are caused by GPS (mis)guidance. By requiring the use of enhanced GPS, New York hopes to reduce its bridge repair costs. Drivers of commercial vehicle found by police not to be using an enhanced GPS system could not only receive a