← Lucian Cosinschi

Digest — A6: The philosophy of automation

Companion to A6-philosophy-automation.md.

The intellectual through-line

The problem is stated once, in 1983, and never really answered. Bainbridge’s Ironies of Automation is a process-control paper with flight-deck examples, not an aviation paper. Its logic is a chain of reversals. The designer who thinks the operator unreliable “still leaves the operator to do the tasks which the designer cannot think how to automate,” producing “an arbitrary collection of tasks” with no support designed for them. The operator’s skills then decay — “physical skills deteriorate when they are not used” — so “a formerly experienced operator who has been monitoring an automated process may now be an inexperienced one.” Yet take-over comes precisely when things are worst: the operator “needs to be more rather than less skilled, and less rather than more loaded, than average.” Monitoring cannot rescue this, because “the automatic control system has been put in because it can do the job better than the operator, but yet the operator is being asked to monitor that it is working effectively” — and where the machine’s decisions exceed human judgement, “the human monitor has been given an impossible task.” Her conclusion is a training conclusion: “it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training.”

Norman (1990) reframes the target. The pathology is not the quantity of automation but its feedback: automation “at an intermediate level of intelligence, powerful enough to take over control that used to be done by people, but not powerful enough to handle all abnormalities,” whose intelligence “is insufficient to provide the continual, appropriate feedback that occurs naturally among human operators.” This constrains what training can do — training can teach active interrogation of an under-communicative machine, but cannot manufacture information the machine withholds. His remedy permits more autonomy as well as less; citers routinely quote only half.

Billings systematises the response. Premise, axiom, corollaries: the human must be in command; to command, be involved; to be involved, be informed; must be able to monitor the automation; automation must therefore be predictable; automation must also monitor the human; every intelligent element must understand the others’ intent. Plus: automate only for good reason, and design automation “simple to train, to learn, and to operate.”

Dekker & Woods (2002) make the move that matters. Substitution-based allocation “cannot provide progress”; quantitative “who does what” fails “because the real effects of automation are qualitative: it transforms human practice and forces people to adapt their skills and routines.” The right question is “How do we make them get along together?” Woods (2002) supplies the mechanism: the Substitution Myth — the belief that machines can be swapped for people “preserving the system though improving the results” — and the law of stretched systems: “every system is stretched to operate at its capacity; as soon as there is some improvement… it will be exploited to achieve a new intensity and tempo of activity.” Hollnagel & Woods supply the unit of analysis: the joint cognitive system, the ensemble, not human-plus-machine.

Resilience engineering generalises it, and gets argued with. Safety-II moves from “as few things as possible go wrong” to “as many things as possible go right,” and names training explicitly as part of the Taylorist machinery for forcing work-as-done into work-as-imagined. Leveson — co-editor of the 2006 founding volume — later publishes against it. Hollnagel & Dekker (2024) turn the ironies argument on human factors itself, including training.

Dekker (2003) is the training pivot: “Discouraging people’s attempts at adaptation can increase the number of failures to adapt… Allowing procedural leeway without investing in people’s skills at adapting… can increase the number of failed attempts at adaptation.” Latitude without invested skill is worse than either extreme.

Leveson closes the loop with the clearest verbatim statement in the corpus: “training requirements go up (not down) in automated systems, and they change their nature”; operators need “a model of the automation and its operation”; they need “General strategies rather than specific responses”; and “Knowledge turns unidentifiable weak signals into identifiable strong signals. People need to know what to look for.”

The theory of learning implied

Five commitments run through the tradition:

  1. Competence is maintained, not acquired — the decay rate sets the recurrency schedule.
  2. It is situated and feedback-dependent; Bainbridge explicitly rules classroom instruction inadequate.
  3. The object is strategies, not responses — asserted independently by Bainbridge (1983) and Leveson (2011) in near-identical words.
  4. It requires a second mental model, of the controller as well as the controlled process.
  5. It is joint, so it cannot be certified in the individual alone — the trainable skill is coordination with a partner that cannot explain itself.

The unresolved tension the report must not paper over. This tradition demands more training (Bainbridge, Baxter, Leveson) while doubting training is the right lever (Woods’ stretched systems predicts the margin gets spent; Hollnagel & Dekker 2024 name training as a variability-suppression instrument). The resolution consistent with the sources: “training” is doing two incompatible jobs — procedural transmission, which they all attack, versus capacity to act in unspecified situations, which they all demand.

Strongest citations

  1. Bainbridge, L. (1983). “Ironies of automation.” Automatica 19(6), 775–779. DOI 10.1016/0005-1098(83)90046-8. Orig. IFAC/IFIP/IFORS/IEA Conf., Baden-Baden, Sept 1982. https://ckrybus.com/static/papers/Bainbridge_1983_Automatica.pdf — [VERIFIED, full text]
  2. Norman, D. A. (1990). “The ‘problem’ with automation: inappropriate feedback and interaction, not ‘over-automation’.” Phil. Trans. R. Soc. Lond. B 327(1241), 585–593. DOI 10.1098/rstb.1990.0101. PMID 1970904 — [abstract VERIFIED; full text not retrieved]
  3. Billings, C. E. (1991). Human-Centered Aircraft Automation: A Concept and Guidelines. NASA TM-103885, Aug 1991, 122 pp — [VERIFIED]
  4. Billings, C. E. (1996). Human-Centered Aviation Automation: Principles and Guidelines. NASA TM-110381. https://ntrs.nasa.gov/api/citations/19960016374/downloads/19960016374.pdf — [VERIFIED — open access, best source for the principles]
  5. Billings, C. E. (1997). Aviation Automation: The Search for a Human-Centered Approach. Lawrence Erlbaum. ISBN 0-8058-2127-9 — [SECONDARY]
  6. Dekker, S. W. A., & Woods, D. D. (2002). “MABA-MABA or Abracadabra? Progress on Human–Automation Co-ordination.” Cognition, Technology & Work 4(4), 240–244. DOI 10.1007/s101110200022 — [bibliographic + abstract VERIFIED; body SECONDARY]
  7. Woods, D. D. (2002). “Steering the Reverberations of Technology Change on Fields of Practice: Laws that Govern Cognitive Work.” Proc. Cognitive Science Society 24(24). https://escholarship.org/uc/item/4d1410cw — [VERIFIED — the citable primary source for both the law of stretched systems and the Substitution Myth]
  8. Woods, D. D., & Dekker, S. W. A. (2000). “Anticipating the Effects of Technological Change.” Theoretical Issues in Ergonomics Science 1(3), 272–282 — [SECONDARY; pagination unconfirmed]
  9. Hollnagel, E., & Woods, D. D. (1983). “Cognitive systems engineering: New wine in new bottles.” IJMMS 18(6), 583–600. DOI 10.1016/S0020-7373(83)80034-0 — [bibliographic VERIFIED; quotations SECONDARY]
  10. Hollnagel, E., & Woods, D. D. (2005). Joint Cognitive Systems: Foundations of Cognitive Systems Engineering. CRC Press — [SECONDARY]
  11. Woods, D. D., & Hollnagel, E. (2006). Joint Cognitive Systems: Patterns in Cognitive Systems Engineering. CRC Press — [SECONDARY]
  12. Hollnagel, E., Woods, D. D., & Leveson, N. (eds) (2006). Resilience Engineering: Concepts and Precepts. Ashgate — [bibliographic VERIFIED]
  13. Hollnagel, E. (2009). The ETTO Principle. Ashgate — [SECONDARY]
  14. Hollnagel, E., Wears, R. L., & Braithwaite, J. (2015). From Safety-I to Safety-II: A White Paper. https://www.era.europa.eu/system/files/2022-10/From%20Safety-I%20to%20Safety-II%20White%20Paper.pdf — [VERIFIED — use for verbatim Safety-I/II and WAI/WAD definitions]
  15. Dekker, S. W. A. (2003). “Failure to adapt or adaptations that fail: contrasting models on procedures and safety.” Applied Ergonomics 34(3), 233–238. DOI 10.1016/S0003-6870(03)00031-0 — [key quotation SECONDARY — re-verify before publishing]
  16. Leveson, N. G. (2011). Engineering a Safer World. MIT Press. Open access — [VERIFIED, full text; §12.6 is the training passage]
  17. Baxter, G., Rooksby, J., Wang, Y., & Khajeh-Hosseini, A. (2012). “The ironies of automation… still going strong at 30?” Proc. ECCE 2012, 65–71. DOI 10.1145/2448136.2448149 — [VERIFIED]
  18. Strauch, B. (2018). “Ironies of Automation: Still Unresolved After All These Years.” IEEE Trans. Human-Machine Systems 48(5), 419–433. DOI 10.1109/THMS.2017.2732506 — [VERIFIED via accepted manuscript; do not cite page numbers from that copy]
  19. Hollnagel, E., & Dekker, S. W. A. (2024). “The ironies of ‘human factors’.” Theoretical Issues in Ergonomics Science, online 20 Dec 2024. DOI 10.1080/1463922X.2024.2443976 — [VERIFIED]
  20. Cooper, M. D. (2022). “The Emperor has no clothes: A critique of Safety-II.” Safety Science 152, 105047 — [bibliographic SECONDARY; content UNVERIFIED]

Back-attributed coinages found

  • “Bainbridge’s four ironies” is a later invention. She numbers exactly two, both about the designer. The rest are “a more serious irony” and “perhaps the final irony.” Do not attribute a numbered list of three or four to her. [VERIFIED by inspection]
  • Billings did not coin “human-centered automation” and says so: “The term is not mine, and I have been unable to find out who first conceived it.” [VERIFIED]
  • “The law of stretched systems” — the name is Woods’; the attribution to Hirschhorn resolves to a citation of a quotation. Woods’ own reference reads: “Hirschhorn, L. (1997). Quoted in Cook, R. I., Woods, D. D. and Miller, C. (1998). A Tale of Two Stories…” — no independent Hirschhorn publication. Cite Woods (2002). [VERIFIED]
  • “The Substitution Myth” is credited by Woods to Woods & Tinapple (1999) — a HFES presidential address and multimedia production, not a paper. Citable print source is Woods (2002). [VERIFIED]
  • “MABA-MABA” is not Fitts’ term. Bainbridge (1983) and Woods (2002) both say “Fitts list.” Who first put “MABA-MABA” in print: [UNVERIFIED]
  • Norman (1990) is routinely half-quoted — his remedy permits more autonomy as well as less. [VERIFIED against abstract]
  • Resilience engineering is not a consensus. Leveson co-edited the 2006 founding volume and published against Safety-II in 2020. [co-editorship VERIFIED]

Genuine gaps

  1. Dekker & Woods (2002) body text. Only the abstract verified. Do not quote the body verbatim without retrieval.
  2. Hollnagel & Woods (1983) full text. Blocked. All three definitional quotations are [SECONDARY] with no page numbers.
  3. “The envisioned world problem” — widely attributed to Woods & Dekker (2000); could not confirm the phrase appears in it. Pagination unconfirmed. [UNVERIFIED]
  4. Dekker (2003) abstract and page numbers. Key quotation string-matched only against a secondary reading-notes source. Re-verify before publishing. The “two contrasting models” reconstruction is the agent’s wording, not Dekker’s, and was not put in quotation marks.
  5. Cooper (2022) Safety-II critique. Bibliographic record confirmed, but ResearchGate flags “TEMPORARY REMOVAL” while a live Elsevier DOI exists. Publication status must be settled before citing. Content [UNVERIFIED].
  6. Leveson’s Safety III (2020). Authorship, date and thrust confirmed; the quoted objection is [SECONDARY], not string-matched.
  7. Hollnagel’s “The Folly of Safety-III” rebuttal. URL located, retrieval failed. [UNVERIFIED]
  8. The “Safety-II is under-operationalised / not new / lacks empirical basis” critique. A real position, but no specific paper making it in those terms was retrieved, and the agent refused to manufacture a citation.
  9. Fitts (1951) not retrieved; [SECONDARY] throughout.
  10. Baxter et al.’s third training bullet lost across a PDF page break. [UNVERIFIED]
  11. Whether Leveson’s “General strategies rather than specific responses” deliberately echoes Bainbridge’s identical phrase — she does not cite her at that point. [UNVERIFIED]