← Lucian Cosinschi

Digest — A5: The regulatory and training response

Companion to A5-regulatory-training-response.md.

Findings

The core move. In 1990 the FAA did something structurally novel: it offered operators freedom from prescribed training content in exchange for proof. SFAR 58 (55 FR 40262, 2 Oct 1990, Docket 25-04) required only that an AQP provide “at least an equivalent means of compliance with current regulations.” Operators gained proficiency-based rather than hours-based qualification and wrote their own Terminal Proficiency Objectives via Job Task Analysis; in return they accepted permanent data-surrender obligations. SFAR 58 migrated into 14 CFR Part 121 Subpart Y (§§121.901–121.925; source credit Docket FAA-2005-20750, 70 FR 54815, 16 Sept 2005), with AC 120-54A (23 June 2006, Chg 1 31 Jan 2017) supplying the five-phase design machinery. AQP’s “First Look” — evaluating objectives before any briefing or practice — is the direct ancestor of the EBT evaluation phase. Notably, AQP’s chief documented evidential yield was as an input to EBT, not as a self-evaluation: rigorous published outcome evaluations of AQP are thin.

MPL arrived via ICAO Annex 1 Amendment 167, adopted by Council 10 March 2006, applicable 23 November 2006 — the first ICAO licence CBTA-compliant from the ground up, with TEM embedded from day one and a 240-hour minimum. The evidence is asymmetric: advocates cite training-system data (586 trainees, ~100% pass rates, average 105 sectors to line release; IATA end-2013: 800 graduates, ~30 airlines), with no control group. Critics — ECA (19 Dec 2013) and IFALPA — object that “the competency-based training concept was misunderstood or misinterpreted,” producing hours-reduction exercises wearing competency labels. The FAA does not issue MPLs.

What “evidence” means in EBT. The point the wider L&D world routinely inverts. Evidence-based does not mean “there is evidence the training works.” It means the curriculum content and its frequency were derived from operational data rather than tradition. ICAO Doc 9995: “Training and assessment based on operational data…” IATA: “The training topics and their associated frequency were defined during the EBT design phase, through the analysis of both safety and training data.” The evidence sits upstream of the curriculum, not downstream of the outcome.

The evidence base itself. The IATA Data Report for Evidence-Based Training is 1st Edition, August 2014 (ISBN 978-92-9252-228-5) — not 2013, correcting a widespread error. Doc 9995 (2013) preceded full publication of its own underpinning data. Seven data streams, 18 sources: LOSA Archive 9,000+ observations across 45 airlines; 3,045 accidents/incidents 1962–2010 (NTSB primary); 1.7m+ flights of FDA for Gen 3/4; 966 pilot survey respondents via IFALPA; 167 pilots in the Training Criticality Survey (161 three-part questions, 40 threats/errors, 51 types); 300+ evidence statements. Six aircraft generations (four jet, two turboprop). Threats scored on likelihood × severity × training benefit, yielding priority A/B/C over a three-year rolling programme.

Headline findings: manual aircraft control a factor in 52% of fatal accidents and 84% of events highly mitigable by training; pilots detect only ~40% of handling errors (commanders detect 39% of their FO’s, 9% of their own); 97% of unstable approaches end in landing, 90% uneventfully, while go-arounds from unstable approaches produce 85% more high-severity events; 28% of LOSA flights carry an automation error; procedures deficiency a factor in 49% of highly-trainable accidents. The cleanest causal argument in the corpus: flights rated outstanding for Captain Leadership and Communication average 2.3 errors versus 7.0 for poorly rated flights — at identical threat load (4.9 vs 5.0).

The ICAO pilot core competencies. Original set (Doc 9995 / EBT Implementation Guide 2013) = eight: Application of Procedures; Communication; Aeroplane Flight Path Management, automation; Aeroplane Flight Path Management, manual control; Leadership and Teamwork; Problem Solving and Decision-Making; Situation Awareness; Workload Management.

Revised set (ICAO 2020, PANS-TRG Amendment 7) = nine. Three changes, and precision matters: “Application of Knowledge” was added; “Application of Procedures” became “Application of Procedures and Compliance with Regulations”; “Situation Awareness” became “Situation Awareness and Management of Information.” An addition plus two renamings/expansions — not a split. Decisive internal evidence: the new competency’s observable behaviours are numbered OB 0.1–0.7, prepended to the existing 1–8 series.

The nine, with EASA abbreviations:

  • Application of Knowledge (KNO) — demonstrates knowledge and understanding of relevant information, operating instructions, aircraft systems and the operating environment (OB 0.1–0.7)
  • Application of Procedures and Compliance with Regulations (PRO) — identifies and applies appropriate procedures per published operating instructions and applicable regulations (OB 1.1–1.7; incl. 1.3 “Follows SOPs unless a higher degree of safety dictates an appropriate deviation”)
  • Communication (COM) — communicates through appropriate means in normal and non-normal situations (OB 2.1–2.10; incl. 2.7 escalation to resolve deviations)
  • Aeroplane Flight Path Management, automation (FPA) — controls the flight path through automation (OB 3.1–3.6)
  • Aeroplane Flight Path Management, manual control (FPM) — controls the flight path through manual control (OB 4.1–4.7)
  • Leadership and Teamwork (LTW) — influences others toward a shared purpose; collaborates to accomplish team goals (OB 5.1–5.11)
  • Problem Solving and Decision-Making (PSD) — identifies precursors, mitigates problems, makes decisions (OB 6.1–6.9; incl. 6.8 “Adapts when faced with situations where no guidance or procedure exists,” 6.9 “Demonstrates resilience when encountering an unexpected event”)
  • Situation Awareness and Management of Information (SAW) — perceives, comprehends and manages information and anticipates its effect (OB 7.1–7.7)
  • Workload Management (WLM) — maintains available workload capacity by prioritizing and distributing tasks (OB 8.1–8.9)

EASA’s rules. ED Decision 2015/027/R introduced mixed EBT. Commission Implementing Regulation (EU) 2020/2036 of 9 December 2020 (OJ L 416/24, 11.12.2020) inserted ORO.FC.231 (six modules over three years; minimum two per type-rating validity, ≥3 months apart), ORO.FC.146 (instructor standardisation), ORO.FC.232, and ARO.OPS.226. ED Decision 2021/002/R issued the AMC/GM under RMT.0599. Baseline EBT replaces the LPC and OPC; mixed EBT retains them as a safety net. Each module has three phases — Evaluation (diagnostic, uninterrupted, identifies individual training needs), Manoeuvres Training, Scenario-Based Training (largest). The evaluation phase is not a jeopardy check because jeopardy has been relocated to a programme-level competency judgement and decoupled from the diagnostic observation — that decoupling is the actual innovation. Grading is 1–5 across four metric levels (competent / competency / observable behaviour / TEM).

The instructor problem. Instructor becomes facilitator (IOB 3.9 “Encourages the trainee to self-assess”; 3.10 “Allows trainee to self-correct”). EASA mandates an Instructor Concordance Assurance Programme: standardisation material every ≤72 months, grading-data analysis every 12 months, Cohen’s/Fleiss’ kappa and ICC. It exists because the underlying problem is real and unsolved: Weber, Roth, Mavin & Dekker (2013) found one rater pair passing a captain whom two pairs failed, dimension scores differing by 2.5 points on a 5-point scale, and only 15–33% overlap in assessment topics — with the warning that “high IRR-scores do not imply that assessors made the same observations.” The one study reporting strong agreement (Sun et al. 2023, ρ = 0.947) achieved it by substituting a flight-data algorithm for the human rater.

Does EBT work? No independent peer-reviewed outcome evaluation of EBT versus the OPC was located. EASA’s own RMT.0599 safety material presents no accident statistics, no safety-benefit estimate, no OPC/EBT comparison. IATA’s 2024 white paper argues deductively (“should enhance,” “should lead to”). The strongest quantitative claims come from a vendor review (Salient, EATS Nov 2025: 90 airlines, check failures 3.4%→0.4% at one carrier) — no control, no peer review. EBT is evidence-based in curriculum derivation and largely faith-based in outcome validation.

UPRT. ICAO Doc 10011 (1st ed. 2014) defines upset as pitch >25° nose-up, >10° nose-down, bank >45°, or inappropriate airspeed within those bounds; it emerged from the RAeS ICATEE work after a June 2009 conference. US: PL 111-216 §208 (post-Colgan 3407) → 78 FR 67800, 12 Nov 2013, Amdt 121-366, §121.423 Extended Envelope Training (six tasks; the decisive shift from “approach to stall” to actual stall recovery) → 81 FR 18178, 30 Mar 2016, Amdt 60-4, requiring stall modelling to 10° AOA beyond stall identification. The FAA candidly concedes fidelity limits: type-specific validated behaviour beyond stall “may not be a reasonable goal,” permitting “type representative” modelling with documented Statements of Compliance. EASA: (EU) 2018/1974 + ED Decision 2019/005/R (RMT.0581) — advanced UPRT on aeroplane, −1g to 2.5/3g, FCL.745.A, FCL.915(e).

Manual flying. SAFO 13002 (4 Jan 2013), SAFO 17007 (4 May 2017), AC 120-123 Flightpath Management (21 Nov 2022) — the last explicitly proscribing blanket rules like “shall never manually fly at night,” and treating manual flight as cognitive + psychomotor + communication skill. All advisory; the operational and economic pressures run the other way.

CBTA generalises. PANS-TRG is structured by personnel group: Part II flight crew (MPL, EBT, UPRT), cabin crew and dispatchers [Reserved], Part III maintenance, Part IV ATM (controllers, ATSEP) — with Doc 10056, Doc 10098 and Doc 10002 as supporting manuals. Amendment 7 (2020) formalised expansion to all Annex 1 licensing and Annex 6 operator training.

The precondition. Every number above exists because reporting was made safe. FOQA (AC 120-82) with de-identification, gatekeeper protocol, 49 U.S.C. §40123 / 14 CFR Part 193 / §13.401; ASAP (AC 120-66C) with the MOU, tripartite ERC, the “Big Five” exclusions, and “no action against an employee who submits a report that is accepted”; ICAO Annex 19 2nd ed. (2016, applicable 7 Nov 2019) §§5.1.2/5.1.3 and Appendix 3’s prohibition on disciplinary/civil/criminal use with three narrow exceptions; Regulation (EU) 376/2014 Art 2(12) just culture, Art 15 confidentiality, Art 16 non-prejudice plus the obligation on every organisation to “adopt internal rules describing how ‘just culture’ principles are guaranteed.” Remove any of these and the data thins, at which point “evidence-based” reverts to expert opinion with a database attached.

The live frontier. IATA has flagged that CBTA creates a new class of individual performance data that Annex 13/19 protections do not cleanly cover, and has asked the ICAO Personnel Training and Licensing Panel to propose new Annex 1 and Annex 6 standards. Any organisation importing competency-based assessment inherits that problem on day one.

Strongest citations

  1. FAA, SFAR 58, Advanced Qualification Program, 55 FR 40262, 2 Oct 1990, Docket 25-04 — https://www.faa.gov/sites/faa.gov/files/2022-11/FR_55_40262%20SFAR%2058.pdf
  2. 14 CFR Part 121 Subpart Y, §§121.901–121.925 (Docket FAA-2005-20750, 70 FR 54815, 16 Sept 2005) — https://www.ecfr.gov/current/title-14/chapter-I/subchapter-G/part-121/subpart-Y
  3. FAA AC 120-54A, Advanced Qualification Program, 23 June 2006, Chg 1 31 Jan 2017
  4. Longridge, T. M., Overview of the Advanced Qualification Program, FAA
  5. ICAO Doc 9995, AN/497, Manual of Evidence-based Training, 1st ed. 2013, ISBN 978-92-9249-242-7 — https://skybrary.aero/sites/default/files/bookshelf/3177.pdf
  6. IATA/ICAO/IFALPA, Evidence-Based Training Implementation Guide, 1st ed., July 2013, ISBN 978-92-9252-191-2
  7. IATA, Data Report for Evidence-Based Training, 1st ed., August 2014, ISBN 978-92-9252-228-5
  8. ICAO Doc 9868, PANS-TRG, 2nd ed. 2015 (superseded 10 Nov 2016); 3rd ed. 2020
  9. IATA, Competency Assessment and Evaluation for Pilots, and Instructors/Evaluators — Guidance Material, 4th ed., 2025 (Appendix A = full nine-competency framework with all OBs)
  10. Commission Implementing Regulation (EU) 2020/2036 of 9 December 2020, OJ L 416/24, 11.12.2020 — https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32020R2036
  11. EASA, Appendix to Opinion No 08/2019 (A) (RMT.0599)
  12. EASA, Notes to EASA EBT Presentation (baseline vs mixed EBT, ICAP, frequency codes)
  13. EASA, Safety Material RMT.0599 — key negative finding: no quantified safety case — https://www.easa.europa.eu/en/downloads/131390/en
  14. ICAO Doc 10011, AN/506, Manual on Aeroplane Upset Prevention and Recovery Training, 1st ed. 2014
  15. FAA final rule, 78 FR 67800, 12 Nov 2013, Docket FAA-2008-0677, Amdt 121-366 (§121.423)
  16. FAA final rule, 81 FR 18178, 30 Mar 2016, Docket FAA-2014-0391, Amdt 60-4
  17. EASA, Explanatory Note to ED Decision 2019/005/R, 27 Feb 2019, RMT.0581
  18. ICAO Annex 19 — Safety Management, 2nd ed., July 2016 (applicable 7 Nov 2019)
  19. Regulation (EU) No 376/2014 of 3 April 2014 — https://eur-lex.europa.eu/eli/reg/2014/376/oj/eng
  20. FAA AC 120-66C (ASAP, 31 Mar 2020); FAA AC 120-82 (FOQA, 12 Apr 2004)
  21. FAA AC 120-123, Flightpath Management, 21 Nov 2022; SAFO 13002; SAFO 17007
  22. IATA, CBTA Expansion within the Aviation System, White Paper, March 2024
  23. Weber, Roth, Mavin & Dekker (2013), “Should we pursue inter-rater reliability or diversity?”, Aviation in Focus 4(2), 33–57; Weber (2016), JAAER 25(2), 39–70, doi 10.15394/jaaer.2016.1645; Sun et al. (2023), Cognition, Technology & Work 25(4), 345–356, doi 10.1007/s10111-023-00737-3
  24. ECA Position Paper on the MPL, 19 Dec 2013; ICAO Personnel Licensing FAQ (Annex 1 Amdt 167)
  25. UK CAA Paper 2004/10, Flight Crew Reliance on Automation, Dec 2004 (Simon Wood, Cranfield)

Gaps and unverified items

Corrections to the brief’s assumptions

  • The IATA Data Report is August 2014, 1st Edition — not 2013.
  • The 2020 competency change is not a split of Application of Procedures. One addition plus two renamings/expansions. OB numbering (0.1–0.7) confirms.
  • Commission Implementing Regulation (EU) 2019/1387 is NOT the UPRT instrument — it concerns landing performance and runway surface conditions.

Could not verify

  1. §121.901 substantive text (only section titles + source credit retrieved).
  2. Verbatim 2013 Doc 9995 wording of “Application of Procedures” and “Situation Awareness” — the count of eight and the three changes are solid; the exact 2013 strings are inferred.
  3. PANS-TRG 3rd Edition (2020) full text — framework sourced from IATA GM 4th ed. 2025, which mirrors it.
  4. Exact date of ED Decision 2021/002/R.
  5. Full text and title of ORO.FC.232.
  6. Which Commission Regulation inserted UPRT into Part-ORO (ORO.FC.220 / ORO.FC.230). Open.
  7. Date-of-application conflict in (EU) 2018/1974: EASA says 20 December 2019; EUR-Lex metadata says 31 January 2022. Unresolved.
  8. ICAO Doc 9803 (LOSA) edition and year — commonly cited as 1st ed. 2002, not confirmed.
  9. MPL four-phase nomenclature not confirmed against PANS-TRG Appendix 2.
  10. IFALPA MPL position paper number 15POS03 — inferred from a filename.
  11. EASA ATQP successor reference ORO.FC.A.245 unverified.
  12. Roth et al. (2015) Cognitive Science volume/pages (DOI 10.1111/cogs.12152 confirmed).
  13. Hwang & Kim, “Success factors of evidence-based training for pilots,” Management Learning — not retrieved.
  14. Hamman & Holt (1997) on Line Operational Evaluation — not retrieved.
  15. EASA RMT.0194 (CBTA into the Aircrew regulation) status.
  16. Doc 10056 / Doc 10098 / Doc 10002 — numbers and titles confirmed via catalogue; contents not read.

The most important gap is substantive, not bibliographic. No independent, peer-reviewed evaluation demonstrating that EBT produces better safety outcomes than the Operator Proficiency Check it replaces. This should be stated in the report rather than smoothed over — it is the sharpest available illustration of the distinction between “evidence-derived curriculum” and “evidence of training effectiveness.”