← Lucian Cosinschi

Crew Resource Management: Six Generations, Three Technologies, and the Evidence Problem

Research memo — Stream A2 Scope: the CRM generational model; LOFT, LOSA and TEM as training technologies; behavioural markers as a methodological move; and a hard look at the evidence that any of it works.

Tagging convention: [VERIFIED] = I read the primary source text directly (downloaded PDF, publisher abstract, or bibliographic API record). [SECONDARY] = reliable secondary source, primary not obtained. [UNVERIFIED] = plausible but I could not confirm; treat as a gap.

1. The generational model

1.1 The canonical paper

Helmreich, Merritt & Wilhelm (1999) is the reference account. Full bibliographic detail confirmed via CrossRef and PubMed: The Evolution of Crew Resource Management Training in Commercial Aviation, The International Journal of Aviation Psychology 9(1), 19–32, DOI 10.1207/s15327108ijap0901_2, PMID 11541445. [VERIFIED]

A crucial practical point for the report: the FAA hosts a full-text manuscript version of this paper for free at https://www.faa.gov/sites/faa.gov/files/2022-11/crmhistory.pdf. This is what I worked from. It is a manuscript/preprint rather than the typeset journal article, so do not cite page numbers from it; cite the journal pagination (19–32) and use the FAA copy for text. [VERIFIED]

The paper’s own abstract states its thesis compactly: it describes “changes in the nature of Crew Resource Management (CRM) training in commercial aviation, including its shift from cockpit to crew resource management,” discusses “[v]alidation of the impact of CRM,” considers “[l]imitations of CRM, including lack of cross-cultural generality,” and presents “[a]n overarching framework that stresses error management.” [VERIFIED]

1.2 The five generations as originally published

The 1999 paper describes five generations. The sixth is a later addition (§1.3).

First generation — Cockpit Resource Management (1981 onward). Built on Blake and Mouton’s Managerial Grid and delivered with the help of management-consulting firms. Individual psychological testing, general leadership theory, intensive seminars, and non-aviation exercises such as “Lost on the Moon.” The target was individual deficiency: “lack of assertiveness by juniors and authoritarian behavior by captains.” Its reception created the reputational damage CRM never fully shed — pilots “denounced them as ‘charm school’ or attempts to manipulate their personalities.” [VERIFIED]

Second generation — Crew Resource Management (mid-1980s onward). Catalysed by a 1986 NASA workshop. The rename tracks a genuine theoretical shift: from fixing individuals to managing “cockpit group dynamics.” Content became modular and aviation-specific — team building, briefing strategies, situation awareness, stress management, decision-making, breaking the error chain. Resistance persisted in a new register: complaints of “psycho-babble,” with “synergy” singled out by participants “as representative of irrelevant jargon.” [VERIFIED]

Note for the report: the 1986 workshop also produced a prediction that has aged interestingly — that “explicit (or stand alone) CRM training would ultimately disappear as a separate component of training when it became imbedded in the fabric of flight training and flight operations.” Helmreich later publicly recanted this (§6.2). [VERIFIED]

Third generation — broadening the scope (early 1990s). Automation modules, human factors, advanced instructor training, and extension beyond the flight deck to cabin crew, dispatchers and maintenance. The authors flag the cost themselves: broadening “may also have had the unintended consequence of diluting the original focus on the reduction of human error.” [VERIFIED]

Fourth generation — integration and proceduralisation (1990 onward). Driven by the FAA’s Advanced Qualification Program (1990). CRM integrated into technical training, specific behaviours added to checklists, and Line Operational Evaluation (LOE) introduced as formal assessment. The authors’ own caveat is the seed of the whole evidence problem: “the fact that crews can demonstrate effective crew coordination while being assessed under jeopardy conditions does not mean that they practice these concepts during normal line operations.” [VERIFIED]

Fifth generation — error management (late 1990s). Grounded in James Reason’s work. The premise: “Human error is ubiquitous and inevitable—and a valuable source of information.” The operational form is the error-management troika: “The first, naturally, is the avoidance of error. The second is trapping incipient errors before they are committed. The third and last is mitigating the consequences of those errors which occur and are not trapped.” Requires an organisational precondition — a non-punitive stance toward error. Continental Airlines is named as the exemplar implementation. [VERIFIED]

1.3 The sixth generation and beyond — who extended the model

The sixth generation (Threat and Error Management, early 2000s onward) is not in the 1999 paper. It is a later attribution, applied retrospectively to Helmreich’s group’s own TEM work (§5). The most useful recent scholarly treatment is:

Pollitt, A., Vlaskamp, D., Blundell, J., & Landman, A. (2026). Whither CRM?—30 Years on: A Narrative Review and Position Paper on the Future of Aviation CRM Training. Aerospace 13(6), 500. DOI 10.3390/aerospace13060500. Citation confirmed via CrossRef; full text open access via Cranfield repository. [VERIFIED]

Pollitt et al. tabulate six generations, adding 6th (early 2000s onward): Threat & Error Management. On a putative seventh generation they are explicitly negative: attempts to define “CRM 7.0” “have not yet gained widespread acceptance,” and the “organization-wide” concept proposed by Terzioğlu “is one which has still not fully materialized to date.” They gesture at a resilience/systems-thinking successor but decline to name it a generation: “Although it has not yet solidified sufficiently to be put forward as a distinct new ‘generation’, this approach might yet come to characterize the next evolution.” [VERIFIED]

Absorption into competency frameworks. The real terminus of the generational story is not a seventh generation but dissolution into Competency-Based Training and Assessment (CBTA) and Evidence-Based Training (EBT), where CRM skills reappear as named competencies — communication, leadership and teamwork, situation awareness, workload management, problem-solving and decision-making — assessed alongside technical competencies rather than beside them. Pollitt et al. read this as “a major step forward towards achieving” Helmreich’s integration vision, while noting implementation “requires expertise, time, and resources.” [VERIFIED]

Gap: I could not obtain ICAO Doc 9868 (PANS-TRG) or Doc 9995 directly — the ICAO-hosted copy returned HTTP 403 and the documents are sold through the ICAO Store. The competency list above is from Pollitt et al., not from ICAO’s own text. Anyone citing the ICAO competency framework verbatim must obtain Doc 9868 itself. [SECONDARY]

2. Origins: the empirical case, United 173, and the 1981 programme

2.1 The simulator study that made the argument

Ruffell Smith, H. P. (1979). A simulator study of the interaction of pilot workload with errors, vigilance, and decisions. NASA TM-78482. NASA Ames Research Center. [VERIFIED] — full text at https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19790006598.pdf

This full-mission simulation established that error rates varied enormously between crews flying identical scenarios, and — the load-bearing finding — that decision quality “seemed related to the ability of captains to manage the resources available to them on the flight deck.” That sentence is the intellectual origin of the phrase “resource management.” [VERIFIED]

Caution: I could not extract the crew count or simulator type from the NTRS text layer. The commonly cited figures (20 crews, B747) should be verified against the report body before use. [UNVERIFIED]

2.2 United 173 and the regulatory trigger

NTSB (1979). Aircraft Accident Report — United Airlines, Inc., McDonnell-Douglas DC-8-61, N8082U, Portland, Oregon, December 28, 1978. NTSB-AAR-79-7, adopted 7 June 1979. [VERIFIED] — I downloaded and read the report.

Probable cause, verbatim: “the failure of the captain to monitor properly the aircraft’s fuel state and to properly respond to the low fuel state and the crewmember’s advisories regarding fuel state. This resulted in fuel exhaustion to all engines. His inattention resulted from preoccupation with a landing gear malfunction and preparations for a possible landing emergency.” Contributing: “the failure of the other two flight crewmembers either to fully comprehend the criticality of the fuel state or to successfully communicate their concern to the captain.” [VERIFIED]

The safety recommendation is the founding regulatory text of CRM. The Board recommended the FAA issue a bulletin directing inspectors to “ensure that their flightcrews are indoctrinated in principles of flightdeck resource management, with particular emphasis on the merits of participative management for captains and assertiveness training for other cockpit crewmembers.” [VERIFIED — text] The recommendation number is OCR-garbled in the scanned report; it is conventionally cited as A-79-47, which I could not confirm. [UNVERIFIED — number only]

2.3 The 1979 workshop and United’s 1981 course

Cooper, G. E., White, M. D., & Lauber, J. K. (Eds.) (1980). Resource Management on the Flight Deck: Proceedings of a NASA/Industry Workshop. NASA CP-2120. Workshop held San Francisco, 26–28 June 1979. [VERIFIED] — NTRS record and download confirmed.

Helmreich et al. trace CRM’s US roots to this workshop, where “the label Cockpit Resource Management (CRM) was applied to the process of training crews to reduce ‘pilot error’ by making better use of the human resources on the flightdeck,” on the basis of research identifying the human-error component of most crashes as “failures of interpersonal communications, decision making, and leadership.” [VERIFIED]

“The first comprehensive U.S. CRM program was initiated by United Airlines in 1981. The training was developed with the aid of consultants who had developed training programs for corporations trying to enhance managerial effectiveness.” [VERIFIED]

Analytical point worth making in the report: the consultants’ provenance explains the “charm school” reception. First-generation CRM was corporate management training with an aviation veneer; the resistance was not irrational conservatism but an accurate perception that the content had not been derived from flight operations.

2.4 The resisters who never converted

Helmreich et al. document a persistent non-converting minority: “From the earliest courses to the present, a small subset of pilots have rejected the concepts of CRM… known by a variety of names – Boomerangs, Cowboys, and Drongos to mention a few. Efforts at remedial training for these pilots have not proved particularly effective.” No percentage is given. [VERIFIED]

They also report attitude decay: “A disturbing finding from this research is a slippage in acceptance of basic concepts, even with recurrent training.” The FAA’s own AC 120-51E repeats this: “when there is no effective reinforcement of CRM concepts by way of recurrent training, improvements in attitudes observed after initial indoctrination tend to disappear.” [VERIFIED]

3. LOFT: the AC 120-35 lineage

3.1 The document chain (all dates verified from the ACs themselves)

AC Title Date Cancels
120-35 Line-Oriented Flight Training Programs 24 May 1978 —
120-35A Line-Oriented Flight Training Programs 11 Aug 1981 120-35
120-35B Line Operational Simulations: LOFT, SPOT, LOE 6 Sep 1990 120-35A
120-35C Line Operational Simulations (same subtitle) 27 Sep 2004 120-35B
120-35D Flightcrew Member Line-Operational Simulations: LOFT, SPOT, LOE 3 Mar 2015 120-35C

[VERIFIED] — I downloaded and read 120-35, 120-35A, 120-35B and 120-35D. The 120-35B revision is where the title broadens from “LOFT programs” to “Line Operational Simulations,” absorbing SPOT and LOE — the documentary trace of Helmreich’s fourth generation.

3.2 The finding that matters most: LOFT’s method predates CRM

AC 120-35 of 24 May 1978 — three years before United’s first CRM course, and seven months before United 173 — already contains the no-instruction rule verbatim:

“The instructor or check airman should not make any comments, give suggestions, or provide any input into the operation once the training has commenced. All comments or criticisms should be reserved for post flight debriefing.”

[VERIFIED] — read directly from the 1978 AC. The identical sentence reappears as paragraph 4.i of AC 120-35A (1981).

This is a genuinely important historical correction. LOFT is not a CRM training technology that CRM produced; it is a pre-existing simulator methodology that CRM colonised. For a report about transferring aviation’s methods to other fields, this matters: the pedagogically radical move (withhold instruction, let consequences run, teach in the debrief) was invented independently of, and prior to, the team-training content it later carried.

3.3 The mechanism: why LOFT is not manoeuvre training

AC 120-35A (1981) states the diagnosis explicitly. Conventional proficiency checks “have the tendency to isolate the crewmember from the normal operating environment during the evaluation period. Over a period of years, the items to be checked have become stereotyped, resulting in the pilot being evaluated knowing exactly what to expect.” [VERIFIED]

The design rules that follow are the operative distinction:

  • No improvisation. “The instructor or check airman should adhere to the syllabus and not improvise or add to it.” (4.f)
  • Consequences persist. “Once an abnormal or emergency condition has been induced, it should remain for the duration of the flight segment unless it is corrected by flightcrew actions.” (4.f)
  • No freeze, real time. “The instructor may not ‘freeze’ the simulator position or problem to allow the flightcrew to cope with a simulated problem. The flight should be representative of ‘real’ flight segment times.” (4.g)
  • Real crews. LOFT “shall be conducted with crewmember positions occupied by line crewmembers qualified and serving in the respective crew positions.” (4.b)

[VERIFIED — all four, read from AC 120-35A]

Taken together these define a distinct instructional genus: an unfalsified consequence environment. The trainee’s errors are allowed to compound in real time, and the learning is extracted afterwards. This is the transferable idea.

3.4 The facilitated debrief

AC 120-35D (2015) codifies the modern position. [VERIFIED]

  • Scenario integrity: “LOFT and LOE event sets should be allowed to continue to their logical conclusion. In qualification LOFT, if the instructor is certain that negative training is occurring, the scenario may be interrupted, but only in extreme situations.”
  • Role: “The facilitator’s role in LOFT or LOE should be viewed as that of communicator, observer, and moderator in the debriefing process. He or she is not an instructor in the traditional sense during the FSTD period.”
  • Debrief method: “The facilitator should not handle the debrief in a teacher-tell manner.” And the pedagogical justification: “Frequently, crews are more critical of themselves than the facilitator would ever be. Self-criticism and self-examination are almost always present in these situations, and in many cases they are much more effective than facilitator criticism.”
  • Priority of purpose: “Technical proficiency, while important, is not the driving force behind the addition of LOS scenarios to an otherwise technical curriculum.”

The AC also draws the clean contrast that makes the taxonomy usable: SPOT (Special Purpose Operational Training) “permits direct instruction and allows for interruption of the scenario by the instructor”; LOFT does not; LOE is the same environment under jeopardy. Three settings, one simulator, differing only in whether instruction and interruption are permitted and whether the outcome counts. [VERIFIED]

4. Behavioural markers: making teamwork gradeable

This is the deep methodological move, and the one with the greatest transfer value. The problem: CRM asserts that crew coordination affects safety, but coordination is a social process, not a parameter on a flight data recorder. To train it you must observe it; to observe it reliably you must decompose it into named, countable, non-inferential behaviours.

4.1 The University of Texas Line/LOS Checklist

The instrument used in LOSA and in UT’s line audits:

Helmreich, R. L., Klinect, J. R., Wilhelm, J. A., & Jones, S. G. (1999). The Line/LOS Checklist, Version 6.0: A checklist for human factors skills assessment, a log for external threats, and a worksheet for flightcrew error management. University of Texas Team Research Project Technical Report 99-01. [VERIFIED — citation, from the reference list of the primary TEM paper]

An earlier version is Helmreich, Butler, Taggart & Wilhelm (1994), NASA/UT/FAA Technical Report 94-02. [SECONDARY]

Significant gap: the UT Austin Libraries guide to the Helmreich reports states plainly that these reports “are not held in the UT Libraries nor at any location we have discovered.” The LLC instrument itself appears to be not publicly available. Anyone building on the LLC must reconstruct it from LOSA documentation or contact the successor custodians (the guide directs enquiries to Dr David M. Musson, McMaster University). [VERIFIED — the unavailability is documented]

4.2 NOTECHS and the JARTEL validation

The European system, built because JAR-FCL required non-technical skills assessment and airlines needed a defensible instrument.

Flin, R., Martin, L., Goeters, K.-M., Hörmann, H.-J., Amalberti, R., Valot, C., & Nijhuis, H. (2003). Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety 3(2), 95–117. [VERIFIED — citation via University of Aberdeen research portal]

Structure: four categories — Cooperation; Leadership and Managerial Skills; Situation Awareness; Decision Making — each decomposed into elements, each element illustrated by behavioural markers for good and poor performance. [VERIFIED]

The design principles are the interesting part, because they are what make the ratings defensible rather than impressionistic:

  1. Only observable behaviour. Personality and affect are excluded.
  2. Technical consequence required. An “unacceptable” rating requires that “flight safety [be] actually (or potentially) compromised.”
  3. Repetition required. A single instance of poor behaviour is insufficient; it must recur within the evaluation.
  4. Explanation required. The examiner must name the element, state the safety consequence, and write a standardised justification.

[VERIFIED — via full-text of the companion Flin paper, "The Development and Testing of NOTECHS"]

Principles 2 and 3 are the crux. They convert a soft judgement (“poor CRM”) into a claim that can be contested on evidence: which behaviour, how often, and what safety consequence. That is the move other fields most need to copy.

JARTEL (Joint Aviation Requirements — Translation and Elaboration of Legislation), established 1998, was the validation programme. Method: eight scripted video scenarios filmed in a B757 simulator, with actors performing pre-defined NOTECHS behaviours across a range from very poor to very good; 105 instructors from 14 airlines in 12 European countries rated them after a half-day briefing. Results: the majority of instructors were consistent, with “an acceptable level of accuracy,” and — a notable finding for the cross-cultural argument — national-cultural differences were less influential than other biographical variables such as English-language proficiency and prior non-technical-skills evaluation experience. [VERIFIED]

Caveat to flag: JARTEL’s acronym expansion varies across sources. I found “Joint Aviation Requirements — Translation and Elaboration of Legislation.” [SECONDARY]

4.3 The reliability problem nobody solved

Flin & Martin (2001), Behavioral Markers for Crew Resource Management: A Review of Current Practice, International Journal of Aviation Psychology 11(1), 95–118, DOI 10.1207/s15327108ijap1101_6 [VERIFIED — CrossRef], surveyed airline practice and found “wide range of practice in the design and implementation,” predicting that “content validity of marker systems and rater reliability are likely to become the focus of both researchers’ and pilots’ interest.” [VERIFIED — abstract]

They were right, and the news is not good. The most rigorous recent treatment is from the medical simulation literature, where behavioural marker systems were imported wholesale from aviation:

Smith, S. E., McColgan-Smith, S., Stewart, F., Mardon, J., & Tallentire, V. R. (2024). Beyond reliability: assessing rater competence when using a behavioural marker system. Advances in Simulation 9(1), Article 55. DOI 10.1186/s41077-024-00329-9. [VERIFIED]

Reported inter-rater reliabilities for established marker systems are poor: Medi-StuNTS ICC = 0.37 (0.26–0.52); NOTSS ICC = 0.29–0.66. In the authors’ own study, experienced faculty achieved ICC 0.60 and near-peers 0.38, and even among experienced raters only 5 of 9 met all five competence criteria. Their argument: “inter-rater reliability can provide some validity evidence for a tool, [but] it does not provide any evidence with regard to the competence of individual raters.” [VERIFIED]

This is a load-bearing qualification for the whole report. Behavioural markers make teamwork nameable; the evidence that they make it reliably measurable by ordinary trained assessors is weak. Any recommendation to export behavioural markers to another field must carry the rater-competence caveat, not just the instrument.

5. LOSA and TEM

5.1 The primary TEM source

Helmreich, R. L., Klinect, J. R., & Wilhelm, J. A. (1999). Models of threat, error, and CRM in flight operations. In Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 677–682). Columbus, OH: The Ohio State University. UT Human Factors Research Project publication 240. [VERIFIED — full text downloaded and read]

Definitions, verbatim:

  • Flightcrew error: “crew action or inaction that leads to deviation from crew or organizational intentions or expectations.”
  • Five error types: Intentional noncompliance (“conscious violations of SOPs or regulations”); Procedural (“slips, lapses, or mistakes in the execution of regulations or procedure. The intention is correct but the execution flawed”); Communication; Proficiency (“a lack of knowledge or stick and rudder skill”); Operational decision (“discretionary decisions not covered by regulation and procedure that unnecessarily increases risk”).
  • Three crew responses: Trap / Exacerbate / Fail to respond.
  • Three outcomes: Inconsequential / Undesired aircraft state / Additional error.
  • Undesired aircraft state responses: Mitigate / Exacerbate / Fail to respond, resolving to Recovery, Additional error, or Incident/accident.

Critically: “Definition and classification of errors and crew responses to them are based on the observable process without consideration of the outcome.” [VERIFIED] — this is what separates TEM from outcome-driven blame attribution and is the single most transferable design decision in the framework.

5.2 LOSA as a method

Klinect, J. R., Murray, P., Merritt, A., & Helmreich, R. (2003). Line Operations Safety Audit (LOSA): Definition and operating characteristics. In Proceedings of the 12th International Symposium on Aviation Psychology (pp. 663–668). Dayton, OH: The Ohio State University. [VERIFIED — FAA-hosted copy]

The ten operating characteristics: jump-seat observations during normal operations; anonymous and confidential data collection; voluntary crew participation; joint management/pilot-association sponsorship; safety-targeted data collection form; trusted and trained observers; trusted data collection site; data-cleaning roundtables; data-derived targets for enhancement; results feedback to line pilots. [VERIFIED]

The institutional guarantee is what makes the data honest: “all LOSA data will be confidential, anonymous and not used to discipline pilots,” and “in over 2200 LOSA flights, not one pilot to the knowledge of The LOSA Collaborative has ever been disciplined.” [VERIFIED]

Institutional history (FAA’s own account): LOSA began at UT Austin in 1991 with FAA funding, prompted by Delta’s request to assess its CRM programme; UT partnered with Delta in 1994 on jump-seat line audits, then with Continental in 1997, which shifted the focus to threat and error and produced TEM. [VERIFIED] ICAO published Doc 9803, Line Operations Safety Audit (LOSA), First edition, 2002. [SECONDARY — the document is sold through the ICAO Store; I did not read it.]

5.3 What LOSA found about normal flights

Two datasets, from different eras and taxonomies. They disagree, and the report should present both rather than picking the friendlier one.

Early (1999, three airlines). From Helmreich, Wilhelm, Klinect & Merritt, Culture, error and Crew Resource Management: 64% of flights contained at least one error; mean 1.84 errors per flight (range 0–14); 72% of flights had at least one external threat, mean 1.91. Error mix: intentional noncompliance 54%, procedural 29%, communications 6%, operational decision-making 6%, proficiency 5%. Consequentiality varied enormously by type: proficiency errors 69% consequential, operational decision errors 43%, procedural 23%, communications 13%, intentional noncompliance 2%. [VERIFIED]

Between-airline variation was extreme — threats per segment 3.3 / 2.5 / 0.4 and errors per segment 0.86 / 1.9 / 2.5 across three carriers — leading Helmreich et al. to warn that “the LOSA database assembled by the University of Texas project cannot be used to represent industry norms because of this variability.” [VERIFIED]

Mature (2002–2006 LOSA Archive, 4,532 observations, 25 airlines). From Merritt, A., & Klinect, J. (2006), Defensive Flying for Pilots: An Introduction to Threat and Error Management, The University of Texas Human Factors Research Project and The LOSA Collaborative [VERIFIED — full text]: 19,053 threats, 13,675 errors, 2,589 undesired aircraft states. Mean 4.2 threats per flight; only 3% of flights encounter none, 17% encounter seven or more. Mean about three errors per flight; about 80% of flights have one or more errors. Error mix: half procedural, one-third aircraft handling, one-sixth communication. About 45% of observed errors went undetected or unresponded to. About 25% of errors are mismanaged (6% producing an additional error, 19% producing an undesired aircraft state). One-third of all flights had an undesired aircraft state. 5% of flights involved an unstable approach, and of those only 5% resulted in a go-around. [VERIFIED]

The apparent contradiction (1.84 vs ~3 errors/flight; 64% vs 80% of flights) is taxonomic, not empirical: the 1999 scheme foregrounded intentional noncompliance as a category; the mature scheme is organised around procedural/handling/communication. Present it as evidence that the measurement instrument was still being built while the “findings” were being cited.

The single most quotable result is the unstable-approach figure: 5% of approaches unstable, 5% of those going around. That is a 95% continuation rate on a condition the industry formally treats as mandatory-go-around. It is the strongest available demonstration that trained, briefed, well-intentioned crews routinely do not do what training says.

5.4 TEM’s adoption and its critics

TEM’s ICAO adoption made it the industry’s default accident-analysis and training vocabulary. Its strengths: outcome-independent classification; a shared language spanning training, audit and investigation; and non-punitive framing that makes data collection possible at all.

The main conceptual critique, per Pollitt et al. (2026): “Critics of TEM argue that it implicitly assumes threats to be identifiable and largely predictable; an assumption that does not always reflect the reality of complex operational environments.” They situate TEM as a Safety-I framework and note that although CBT and EBT “signaled a shift toward a more Safety-II-informed training philosophy,” instructional practice “continue[s] to reflect predominantly Safety-I-oriented assumptions.” [VERIFIED]

6. THE EVIDENCE QUESTION

6.1 What the systematic reviews actually found

Three studies carry the weight. All citations confirmed via CrossRef and PubMed.

Salas, E., Burke, C. S., Bowers, C. A., & Wilson, K. A. (2001). Team Training in the Skies: Does Crew Resource Management (CRM) Training Work? Human Factors 43(4), 641–674. DOI 10.1518/001872001775870386, PMID 12002012. Reviewed 58 published accounts using Kirkpatrick’s four levels. Findings: CRM “generally produced positive reactions, enhanced learning, and promoted desired behavioral changes. However, we cannot ascertain whether CRM has an effect on an organization’s bottom line (i.e., safety).” They call for “systematic, multilevel evaluation efforts” and note that “[a]s many evaluations do not collect data across levels… the impact of CRM cannot be truly determined.” [VERIFIED — abstract verbatim]

Salas, E., Wilson, K. A., Burke, C. S., & Wightman, D. C. (2006). Does Crew Resource Management Training Work? An Update, an Extension, and Some Critical Needs. Human Factors 48(2), 392–412. DOI 10.1518/001872006777724444, PMID 16884057. Reviewed 28 published accounts across aviation, medicine, offshore oil, maritime and nuclear power. Findings: positive reactions; “the impact of training on learning and behavioral changes suggest mixed results across and within domains”; and, five years on, “as was found by Salas, Burke, et al. in 2001, we cannot ascertain whether CRM has had an impact on the organization’s bottom line (i.e., safety).” Three critical needs identified: “a mandate, access to data, and resources.” [VERIFIED — abstract verbatim]

O’Connor, P., Campbell, J., Newon, J., Melton, J., Salas, E., & Wilson, K. A. (2008). Crew Resource Management Training Effectiveness: A Meta-Analysis and Some Critical Needs. The International Journal of Aviation Psychology 18(4), 353–368. DOI 10.1080/10508410802347044. Full author list confirmed via CrossRef. Only 16 studies met a priori inclusion criteria for meta-analysis. Reactions positive (mean 4 on a 5-point scale); large effects on attitudes and behaviours; medium effect on knowledge. The authors call for researchers “to be more rigorous about the data included in research reporting CRM evaluation to allow effect sizes to be calculated.” [VERIFIED — via institutional repository record]

Caveat I must flag: I obtained the O’Connor effect-size descriptions (“large,” “medium”) from a repository record, not from the article’s own results tables. The numeric effect sizes and confidence intervals should be taken from the paper itself before publication. [SECONDARY]

6.2 Which Kirkpatrick levels the evidence actually reaches

The honest summary:

  • Level 1 (Reactions): abundant and consistently positive. Trainees like CRM. This is the most robust finding in the literature and the least informative.
  • Level 2 (Learning — attitudes and knowledge): substantial, positive, but decaying. Attitude change is well documented; so is its erosion without recurrent reinforcement (Helmreich et al. 1999; AC 120-51E).
  • Level 3 (Behaviour): real but thin and mixed. Salas 2001 found behaviour change; Salas 2006 found it “mixed across and within domains.” Only 16 studies in the whole literature were methodologically adequate for meta-analysis by 2008.
  • Level 4 (Organisational impact / safety): absent. Two independent reviews, five years apart, by the same lead author, reached the identical conclusion in near-identical words.

That last point deserves emphasis in the report. This is not a case of an unexamined literature. It is a case of a literature that has been examined repeatedly, by sympathetic reviewers, and has repeatedly failed to produce Level 4 evidence.

A note on the internal tension: Helmreich’s own group asserted in 1999 that “Good programs do have a measurable, positive effect on crew performance, and, hence, safety” — with “and, hence” doing enormous unearned work — while conceding in the same paragraph that “[s]ome are carefully designed and reflective of their organization’s culture, others are mere exercises in compliance with requirements.” [VERIFIED] The inference from crew performance to safety is exactly the step the Salas reviews say cannot be made.

6.3 The counterfactual problem

This is the intellectual honesty test, and the answer is uncomfortable.

The magnitude of what must be explained. Barnett, A., & Reig Torra, J. (2024), Airline Safety: Still Getting Better?, Journal of Air Transport Management [VERIFIED — citation via MIT News; DOI not obtained, see gaps], gives death risk per boarding by period: 1968–77, 1 per 350,000; 1978–87, 1 per 750,000; 1988–97, 1 per 1.3m; 1998–2007, 1 per 2.7m; 2007–17, 1 per 7.9m; 2018–22, 1 per 13.7m. Roughly a halving per decade, sustained for five decades. [SECONDARY — figures via MIT News summary of the paper]

Boeing’s own current data, read directly from the source: Statistical Summary of Commercial Jet Airplane Accidents, Worldwide Operations 1959–2025, 57th edition, April 2026, The Boeing Company. “Over the past two decades, this report documents a 35% decline in the total accident rate and a 60% decline in the fatal accident rate – all while departures have increased by more than 20%.” Broken out: accident rate −35%, hull-loss rate −58%, fatal accident rate −60%. [VERIFIED — read from the PDF text layer]

Why CRM cannot claim this. The same decades delivered TCAS, GPWS and then EGPWS/TAWS, FMS, glass cockpits, vastly more reliable turbofans, FOQA, ETOPS discipline, SMS, and non-punitive reporting (ASAP/ASRS). CRM is one intervention among a dozen, all deployed to the same fleet over the same period. There is no untreated control fleet.

Does anyone decompose it? Essentially, no — and the report should say so plainly.

  • The FAA does not credit CRM. Its own retrospective, Out Front on Airline Safety: Two Decades of Continuous Evolution, states that “the fatality risk for commercial aviation in the United States fell 83 percent from 1998 to 2008” and credits CAST, ASIAS, voluntary reporting, Infoshare, SAS, SMS, new aircraft and regulations, and TAWS. CRM is not named. [VERIFIED]
  • CAST does not decompose its own headline number. The 83% figure is presented as an aggregate outcome of a portfolio of safety enhancements; CAST’s public material describes tracking implementation but supplies no per-intervention attribution. [VERIFIED — I fetched CAST's own overview material and its history page returned no decomposition]

The one place a decomposition exists — and it is bad news for CRM. The CAST CFIT/ALAR Joint Safety Implementation Team produced ranked effectiveness estimates for 171 interventions. I downloaded and read the report (Results and Analysis — JSIT — Approach and Landing, FAA-hosted). Methodology: the JSAT assigned each intervention an Overall Effectiveness (OE) score 0–6 based on analysis of the selected ALAR accidents; the JSIT assigned Feasibility (F) 1–3 by working-group consensus; interventions were prioritised by the product OE × F, with a cutoff of 5.1. [VERIFIED]

Verified rankings, read from Appendix D of the report:

# Intervention OE F OE×F
134 Checklist design prioritising critical items 5.0 2.8 14.2
85 Synthetic vision capability 5.0 2.3 11.7
35 TAWS (EGPWS) fit and retrofit 5.0 2.2 10.8
142 Unstabilised-approach recognition and go-around gates 4.0 2.5 10.0
24 Appropriate crew pairing 3.5 2.5 8.8
131 Training/standardisation emphasis (team concept) 1.4 2.8 4.0
308 Formal CRM training emphasising management skills 1.3 2.8 3.7
227 Inter-crew communications emphasis 1.1 3.0 3.3
25 Establish a CRM training programme; regulator to require it 1.1 2.8 3.1

And the disposition table records the verdict: the project titled “Crew Resource Management (CRM) Training” was not implemented, because “OExF values for the interventions in this project were below the cutoff value selected by the CFIT/ALAR JSIT.” [VERIFIED — read from the report]

How to characterise this fairly. These OE scores are structured expert judgement derived from accident analysis, not measured effect sizes. The panel was assessing marginal effectiveness for preventing approach-and-landing and CFIT accidents specifically, at a time when CRM was already universally mandated — so a low marginal score partly reflects saturation, not worthlessness. And a separate project titled “Flightcrew Training” was implemented on high OE×F, so the panel was not anti-training; it was specifically unimpressed by generic CRM curriculum interventions relative to targeted training and hardware.

But the finding stands and should not be softened: when the US industry’s principal safety body ranked interventions by expected effectiveness against the accident category CRM was invented to prevent, formal CRM training scored 1.1–1.4 out of 6 and was dropped, while TAWS scored 5.0 and was implemented. In the most recent decade of Boeing data (2016–2025, 31 fatal accidents worldwide), CFIT accounts for 1 fatal accident [VERIFIED — read from the Boeing PDF]. CFIT was substantially engineered out of existence by a box, in the same era that CRM was being credited with the improvement.

The steelman for CRM. Three points of genuine defence, which the report should give fairly:

  1. Category shift. As CFIT collapsed, the remaining fatal-accident categories in 2016–2025 are LOC-I (6) and runway excursion (6) — precisely the categories where crew monitoring, intervention and decision-making are decisive and where no equivalent hardware fix exists. If CRM has a domain, it is the residual.
  2. Documented operational change. Continental’s LOSA-driven SOP intervention produced “a 70 percent reduction in non-conforming approaches” between its 1996 and 2000 audits [VERIFIED — Klinect et al. 2003]. This is a real Level 3/4 result — though note it was achieved by changing procedures in response to audit data, not by CRM classroom training as such.
  3. Adjacent-field controlled evidence. The strongest outcome study anywhere in the CRM family is medical, not aviation: Neily, J., Mills, P. D., Young-Xu, Y., Carney, B. T., West, P., Berger, D. H., Mazzia, L. M., Paull, D. E., & Bagian, J. P. (2010), Association between implementation of a medical team training program and surgical mortality, JAMA 304(15), 1693–1700, DOI 10.1001/jama.2010.1506, PMID 20959579. Retrospective study with contemporaneous control, 182,409 procedures, 108 VHA facilities. Trained facilities showed 18% reduction in annual mortality (RR 0.82, 95% CI 0.76–0.91, p=.01) vs 7% at not-yet-trained facilities (RR 0.93, 95% CI 0.80–1.06, p=.59); propensity-matched decline ~50% greater in the training group (RR 1.49, 95% CI 1.10–2.07, p=.01); and a dose-response relationship — 0.5 fewer deaths per 1,000 procedures per additional quarter of programme (95% CI 0.2–1.0, p=.001). [VERIFIED — abstract verbatim]

Point 3 is double-edged and must be presented as such. It is the best Level 4 evidence in the entire team-training literature — a large sample, a control group, and a dose-response gradient. But it is (a) not aviation, (b) observational rather than randomised, and (c) an intervention bundled with surgical checklists and mandatory briefings/debriefings, so the effect cannot be cleanly assigned to team training rather than to the checklist. It shows the class of intervention can move hard outcomes. It does not show that CRM-as-delivered-in-aviation does.

The counterweight in the same field. Verbeek-van Noord, I., de Bruijne, M. C., Zwijnenberg, N. C., Jansma, E. P., van Dyck, C., & Wagner, C. (2014), Does classroom-based Crew Resource Management training improve patient safety culture? A systematic review, SAGE Open Medicine 2, 2050312114529561, DOI 10.1177/2050312114529561, PMID 26770720. 22 manuscripts from 1,926 screened; only 6 controlled designs. Conclusion: “evidence of the effectiveness of CRM training in health care in terms of improved safety culture is scarce and the validity of most studies is limited.” And the decisive sentence: “Uncontrolled studies in our systematic review all found positive effects… Two controlled studies that used a control group found no training effects.” [VERIFIED]

That contrast — every uncontrolled study positive, controlled studies null — is the classic signature of a literature dominated by design artefact. It is the single most important methodological warning to carry into any recommendation about exporting CRM.

6.4 The bottom line

After forty-five years, roughly $billions of industry investment, and universal regulatory mandate, the defensible statement is:

CRM reliably changes what pilots say they believe, and probably changes what they do in observable ways under observation. There is no study establishing that CRM training reduced accidents. The safety improvement of the CRM era is real and enormous, but it is over-determined, and the one published expert decomposition of intervention effectiveness ranked formal CRM training near the bottom.

This is not a claim that CRM does not work. It is a claim that the profession never built the evidence base that would let it know, and that the industry’s own prioritisation exercises quietly reflect that.

7. CRM’s failures

7.1 Accidents with CRM training in place

Colgan Air 3407 (NTSB/AAR-10/01, adopted 2 February 2010). The crew had received CRM training (report §1.17.1.4). The probable cause was “the captain’s inappropriate response to the activation of the stick shaker”; contributing factors included “[t]he flight crew’s failure to monitor airspeed,” “[t]he flight crew’s failure to adhere to sterile cockpit procedures,” and “[t]he captain’s failure to effectively manage the flight.” [VERIFIED — via full-text report] Every named failure is a core CRM competency, in a crew that had been CRM-trained. The report elevated “pilot professionalism” to a distinct safety issue — which is, in effect, a concession that thirty years of CRM had not produced the behaviour it was designed to produce.

Johnson’s incident-based critique. Johnson, C. (1999, unpublished draft), Reasons For The Failure of CRM Training in Aviation, University of Glasgow Department of Computing Science [VERIFIED — read; note it is a self-archived draft, not peer-reviewed, and must be cited as such]. He found only two explicit positive examples of CRM preventing an incident across 400+ reviewed incident reports; cites accidents (Federal Express 1406, Puerto Plata) where crews had completed approved CRM courses weeks or months earlier; and argues CRM is structurally mis-targeted because “coordination is not necessary for most of the time that crews are in flight” while the training emphasises routine operation. He also makes the sharp methodological observation that CRM failures are far more visible in reporting systems than CRM successes, which biases the evidence base in the pessimistic direction — a point of intellectual honesty that cuts against his own thesis. [SECONDARY — unpublished]

7.2 The cultural critique

Helmreich’s group made this critique themselves, which is to their credit and complicates the “Anglo-American imposition” reading.

The empirical basis: Merritt, A. (2000), Culture in the Cockpit: Do Hofstede’s Dimensions Replicate?, Journal of Cross-Cultural Psychology 31(3), 283–301, DOI 10.1177/0022022100031003001 [VERIFIED — CrossRef]. Survey of 9,400 male commercial airline pilots in 19 countries. Hofstede’s dimensions replicated, with correlations: Individualism–Collectivism .96; Power Distance .87; Masculinity–Femininity .75; Uncertainty Avoidance .68. Conclusion: “national culture exerts an influence on cockpit behavior over and above the professional culture of pilots.” [VERIFIED] Methodological caveat from the abstract: the replication succeeded only after relaxing item-equivalence constraints with Hofstede’s original items and formulae — i.e. Hofstede’s instrument required modification to fit pilots. Present this honestly; it slightly weakens the “Hofstede replicates cleanly in the cockpit” story. [VERIFIED]

The consequences for training, from Helmreich et al.’s own writing:

  • “CRM did not export well… In many cases, the concepts presented were incongruent with the national culture of the pilots.” [VERIFIED]
  • “Exhortations to junior crewmembers to be more assertive in questioning their captains may fall on deaf ears in these cultures.” [VERIFIED]
  • “[T]he same CRM training will not work as well in Turkey as in Texas.” [VERIFIED]
  • Even domestically: “courses imported from other organizations had less impact than those that were developed to reflect the organizational culture and operational issues of the receiving carrier.” [VERIFIED]

The nuance that stops this becoming a simple story: Helmreich et al. note that collectivist cultures may accept the teamwork content more readily than individualistic ones, since “[i]ndividualists may cling to the stereotype of the lone pilot braving the elements,” and that high uncertainty-avoidance cultures may be more accepting of behaviourally-specified CRM while “[t]he U.S. is low in Uncertainty Avoidance, which is reflected operationally in… weaker adherence to Standard Operating Procedures.” [VERIFIED] So the critique is not “CRM is Anglo and doesn’t travel”; it is “CRM’s assertiveness content is culturally specific, while its teamwork content may travel better than it does at home.”

Counter-evidence to weigh: JARTEL found national-cultural differences in NOTECHS rating less influential than English proficiency and prior evaluation experience (§4.2). The cultural-generality problem may be sharper for CRM content than for CRM assessment. [VERIFIED]

7.3 CRM as compliance ritual

The critique is not external. Helmreich, Klinect & Wilhelm wrote in 1999: “Criticisms of CRM often fail to recognize the variability in programs. Some are carefully designed and reflective of their organization’s culture, others are mere exercises in compliance with requirements.” [VERIFIED]

Pollitt et al. (2026) find this has become the norm rather than the exception. Their section 3.2 is titled “Deficiencies in Training Delivery: ‘Checkbox’ CRM”:

  • “CRM training effectiveness has decreased because it is often reduced to a watered-down, compliance-driven, ‘checkbox’ exercise within broader training programs, delivered through passive classroom sessions or stand-alone e-learning.”
  • “Training frequently relies on didactic, lecture-heavy delivery with limited application of structured instructional design, experiential learning, or systematic practice and reflection.”
  • On video-conference delivery: it does “nothing to facilitate ‘intensive introspection done publicly in a seminar style format’, as it was originally conceived.”

[VERIFIED]

The irony worth naming in the report: CRM’s fourth-generation triumph — integration and proceduralisation, driven by regulatory mandate — is the same mechanism that produced its degradation into compliance. Mandating a thing guarantees it is delivered and guarantees nothing about how.

7.4 Helmreich’s recantation

Worth a line in the report because it reverses the field’s founding prediction. Having earlier endorsed the 1986 NASA view that CRM should eventually dissolve into technical training, Helmreich, Klinect & Wilhelm wrote in 1999: “Some have argued that CRM training should ultimately disappear, as it becomes fully integrated into technical training. We once supported this notion, but with hindsight we now realize that it is and should be a separate aspect of training.” [VERIFIED] The CBTA/EBT movement has since gone the other way.

8. Explicit gaps and unverifiable claims

Listed so nothing here is over-claimed.

  1. ICAO Doc 9803 (LOSA, 1st ed. 2002) and Doc 9868 (PANS-TRG) — not read. ICAO’s own PDF returned HTTP 403; the documents are sold through the ICAO Store. All ICAO competency-framework and LOSA-manual claims here are second-hand. [SECONDARY]
  2. The UT Line/LOS Checklist instrument — publicly unavailable. UT Austin Libraries state the Helmreich technical reports “are not held in the UT Libraries nor at any location we have discovered.” Citation verified; content not.
  3. O’Connor et al. (2008) numeric effect sizes — the “large/medium” descriptions come from a repository record, not the article’s results tables. Obtain the article before publishing any effect size.
  4. NTSB recommendation number for United 173 — the scanned report’s OCR garbles it. Text verified; the number A-79-47 is conventional but unconfirmed.
  5. Ruffell Smith (1979) study parameters — crew count and simulator type not extracted from the NTRS text layer. The commonly cited “20 crews / B747” is unverified.
  6. Barnett & Reig Torra (2024) — citation and figures obtained via MIT News, not the article. Volume, pages and DOI not confirmed; the death-risk series should be taken from the paper.
  7. Boeing per-decade accident rates — the chart values are not reliably extractable from the PDF text layer. I have used only the explicitly stated figures (−35% accident rate, −58% hull loss, −60% fatal accident rate over two decades; 2016–2025 fatal accidents by CICTT category), which are verified.
  8. JARTEL acronym expansion — sources vary. [SECONDARY]
  9. O’Connor et al. (2002) two Human Factors and Aerospace Safety papers on CRM evaluation methods (vol. 2, pp. 217–233 and 235–255) — located via CrossRef reference records only; not read. [SECONDARY]
  10. Johnson (1999) is an unpublished self-archived draft. Cite as such or drop it; do not present it as peer-reviewed.
  11. No CAST per-intervention decomposition of the 83% figure exists in the public material I could reach. If the report asserts that no such decomposition has been published, that is my finding but is a negative claim and cannot be proven exhaustively.

9. Consolidated bibliography

Primary — foundational

  1. Ruffell Smith, H. P. (1979). A simulator study of the interaction of pilot workload with errors, vigilance, and decisions. NASA TM-78482. NASA Ames Research Center. https://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/19790006598.pdf
  2. National Transportation Safety Board (1979). Aircraft Accident Report: United Airlines, Inc., McDonnell-Douglas DC-8-61, N8082U, Portland, Oregon, December 28, 1978. NTSB-AAR-79-7. https://www.ntsb.gov/investigations/AccidentReports/Reports/AAR7907.pdf
  3. Cooper, G. E., White, M. D., & Lauber, J. K. (Eds.) (1980). Resource Management on the Flight Deck: Proceedings of a NASA/Industry Workshop, San Francisco, 26–28 June 1979. NASA CP-2120. https://ntrs.nasa.gov/citations/19800013796

Primary — the generational model and TEM

  1. Helmreich, R. L., Merritt, A. C., & Wilhelm, J. A. (1999). The Evolution of Crew Resource Management Training in Commercial Aviation. The International Journal of Aviation Psychology, 9(1), 19–32. DOI 10.1207/s15327108ijap0901_2. Free full text: https://www.faa.gov/sites/faa.gov/files/2022-11/crmhistory.pdf
  2. Helmreich, R. L., Klinect, J. R., & Wilhelm, J. A. (1999). Models of threat, error, and CRM in flight operations. Proceedings of the Tenth International Symposium on Aviation Psychology (pp. 677–682). Columbus, OH: The Ohio State University. https://www.pacdeff.com/pdfs/models_of_threat_error.pdf
  3. Helmreich, R. L., Wilhelm, J. A., Klinect, J. R., & Merritt, A. C. (2001). Culture, error and Crew Resource Management. In E. Salas, C. A. Bowers & E. Edens (Eds.), Improving Teamwork in Organizations (pp. 305–331). Hillsdale, NJ: Erlbaum. https://www.raes-hfg.com/reports/18oct00-RAWG-1/culture.pdf
  4. Klinect, J. R., Murray, P., Merritt, A., & Helmreich, R. (2003). Line Operations Safety Audit (LOSA): Definition and operating characteristics. Proceedings of the 12th International Symposium on Aviation Psychology (pp. 663–668). https://www.faa.gov/sites/faa.gov/files/about/initiatives/maintenance_hf/losa/klinect_operatingcharacteristics2003.pdf
  5. Merritt, A., & Klinect, J. (2006). Defensive Flying for Pilots: An Introduction to Threat and Error Management. The University of Texas Human Factors Research Project & The LOSA Collaborative. https://skybrary.aero/sites/default/files/bookshelf/1982.pdf

Primary — regulatory

  1. FAA (1978). AC 120-35, Line-Oriented Flight Training Programs, 24 May 1978. https://rosap.ntl.bts.gov/view/dot/69947
  2. FAA (1981). AC 120-35A, Line-Oriented Flight Training Programs, 11 Aug 1981. https://rosap.ntl.bts.gov/view/dot/69948
  3. FAA (1990). AC 120-35B, Line Operational Simulations: LOFT, SPOT, LOE, 6 Sep 1990. https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC%20120-35B.pdf
  4. FAA (2015). AC 120-35D, Flightcrew Member Line-Operational Simulations, 3 Mar 2015. https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_120-35D.pdf
  5. FAA (2004). AC 120-51E, Crew Resource Management Training, 22 Jan 2004. https://www.faa.gov/documentLibrary/media/Advisory_Circular/AC_120-51E.pdf
  6. CAST CFIT/ALAR Joint Safety Implementation Team. Results and Analysis — JSIT — Approach and Landing. FAA. https://www.faa.gov/sites/faa.gov/files/2022-11/ALAR.pdf
  7. National Transportation Safety Board (2010). Loss of Control on Approach, Colgan Air, Inc., Bombardier DHC-8-400, N200WQ, Clarence Center, New York, February 12, 2009. NTSB/AAR-10/01. https://libraryonline.erau.edu/online-full-text/ntsb/aircraft-accident-reports/AAR10-01.pdf

Primary — the evidence question

  1. Salas, E., Burke, C. S., Bowers, C. A., & Wilson, K. A. (2001). Team Training in the Skies: Does Crew Resource Management (CRM) Training Work? Human Factors, 43(4), 641–674. DOI 10.1518/001872001775870386
  2. Salas, E., Wilson, K. A., Burke, C. S., & Wightman, D. C. (2006). Does Crew Resource Management Training Work? An Update, an Extension, and Some Critical Needs. Human Factors, 48(2), 392–412. DOI 10.1518/001872006777724444
  3. O’Connor, P., Campbell, J., Newon, J., Melton, J., Salas, E., & Wilson, K. A. (2008). Crew Resource Management Training Effectiveness: A Meta-Analysis and Some Critical Needs. The International Journal of Aviation Psychology, 18(4), 353–368. DOI 10.1080/10508410802347044
  4. Neily, J., Mills, P. D., Young-Xu, Y., Carney, B. T., West, P., Berger, D. H., Mazzia, L. M., Paull, D. E., & Bagian, J. P. (2010). Association between implementation of a medical team training program and surgical mortality. JAMA, 304(15), 1693–1700. DOI 10.1001/jama.2010.1506
  5. Verbeek-van Noord, I., de Bruijne, M. C., Zwijnenberg, N. C., Jansma, E. P., van Dyck, C., & Wagner, C. (2014). Does classroom-based Crew Resource Management training improve patient safety culture? A systematic review. SAGE Open Medicine, 2, 2050312114529561. DOI 10.1177/2050312114529561
  6. Barnett, A., & Reig Torra, J. (2024). Airline Safety: Still Getting Better? Journal of Air Transport Management. [DOI unconfirmed]
  7. The Boeing Company (2026). Statistical Summary of Commercial Jet Airplane Accidents, Worldwide Operations 1959–2025, 57th edition. https://www.boeing.com/content/dam/boeing/v2/safety/statsum.pdf
  8. FAA. Out Front on Airline Safety: Two Decades of Continuous Evolution. https://www.faa.gov/newsroom/out-front-airline-safety-two-decades-continuous-evolution

Primary — behavioural markers

  1. Flin, R., & Martin, L. (2001). Behavioral Markers for Crew Resource Management: A Review of Current Practice. The International Journal of Aviation Psychology, 11(1), 95–118. DOI 10.1207/s15327108ijap1101_6
  2. Flin, R., Martin, L., Goeters, K.-M., Hörmann, H.-J., Amalberti, R., Valot, C., & Nijhuis, H. (2003). Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety, 3(2), 95–117.
  3. Helmreich, R. L., Klinect, J. R., Wilhelm, J. A., & Jones, S. G. (1999). The Line/LOS Checklist, Version 6.0. University of Texas Team Research Project Technical Report 99-01. [Not publicly available]
  4. Smith, S. E., McColgan-Smith, S., Stewart, F., Mardon, J., & Tallentire, V. R. (2024). Beyond reliability: assessing rater competence when using a behavioural marker system. Advances in Simulation, 9(1), 55. DOI 10.1186/s41077-024-00329-9

Primary — culture and critique

  1. Merritt, A. (2000). Culture in the Cockpit: Do Hofstede’s Dimensions Replicate? Journal of Cross-Cultural Psychology, 31(3), 283–301. DOI 10.1177/0022022100031003001
  2. Helmreich, R. L., & Merritt, A. C. (1998). Culture at Work in Aviation and Medicine: National, Organizational and Professional Influences. Aldershot: Ashgate.
  3. Pollitt, A., Vlaskamp, D., Blundell, J., & Landman, A. (2026). Whither CRM?—30 Years on: A Narrative Review and Position Paper on the Future of Aviation CRM Training. Aerospace, 13(6), 500. DOI 10.3390/aerospace13060500
  4. Johnson, C. (1999). Reasons For The Failure of CRM Training in Aviation. Unpublished draft, University of Glasgow. https://www.dcs.gla.ac.uk/~johnson/papers/crm