← Lucian Cosinschi

DIGEST — A10: AI Copilots, Skill Formation and Professional Education

Memo: /home/claude/research/A10-ai-copilots-formation.md Length note: the memo ran to ~13,000 words against a 5,000–7,000 brief. The overshoot is almost entirely the citation apparatus you asked for — full citation, design label, sample size, and verification tag on every source, plus the downgrade reasoning. Say the word and I will cut it to spec; the compressible parts are §5 (CBME background), §7 (higher ed) and §9 (learning products).

The core argument this stream supports

Assisted performance and unassisted capability dissociate, practitioners cannot detect the dissociation in themselves, and the destination fields are re-deriving automation complacency from scratch without citing aviation. That last point is the report’s justification: Dell’Acqua reaches for self-driving cars, the BCG paper reaches for Lebovitz et al. (2022) — neither cites Bainbridge, Wiener, Billings or Sarter & Woods.

Strongest citations, with design and N

1. Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö. & Mariman, R. (2024/25). “Generative AI Can Harm Learning.” SSRN WP 4895486. Cluster-RCT (classroom-randomised), pre-registered (AsPredicted 4DL_Q3J), N = 839 students, ~50 classes, Turkish high school, 4×90-min maths sessions. Working paper — not peer reviewed. Practice phase: GPT Base +0.137 (SE .031) = +48%; GPT Tutor +0.361 (SE .032) = +127%. Unassisted exam: GPT Base −0.054 (SE .022), p<.05 = −17%; GPT Tutor −0.004 (n.s.). Report both arms. The Tutor result is a null, not a win — safeguards prevented damage, they did not produce learning. Mechanism: GPT Base wrong 49% of the time; 31% of first messages were “what is the answer?”; logical errors did not transfer to exam problems (non-learning shown via a null). Metacognitive result: harmed students did not perceive reduced learning. Published critique exists: Tan & Rajaratnam, SSRN 4898213.

2. Liu, G., Christian, B., Dumbalska, T., Bakker, M. & Dubey, R. (2026). “AI Assistance Reduces Persistence and Hurts Independent Performance.” arXiv:2604.04721v2. 3 RCTs, Prolific, combined N = 1,222 (307/585/168). Preprint. Withdrawal-phase solve rates, d = −0.42 / −0.19 / −0.42. Novel contribution is persistence: skip rate d = 0.25 (Exp 1) and 0.42 (Exp 3). Costs concentrated in the direct-solutions subgroup (cross-sectional, authors say not necessarily causal). Exposure was 10–15 minutes — the “AI damages your brain in 10 minutes” framing is a distortion.

3. Dell’Acqua, F., McFowland III, E., Mollick, E. et al. (2026). “Navigating the Jagged Technological Frontier.” Organization Science 37(2), DOI 10.1287/orsc.2025.21838. Pre-registered field experiment, N = 758 BCG consultants. Now peer reviewed — most commentary still cites the working paper. Inside frontier: +12.2% tasks, 25.1% faster, >40% quality. Outside frontier: control correct 84.5%, AI conditions 60% and 70% — 19pp decrease. Skill compression: bottom-half +43%, top-half +17%. Correction for the brief: “falling asleep at the wheel” is not an argument in this paper — it appears only as a reference to Dell’Acqua’s own separate study.

4. Dell’Acqua, F. (2022). “Falling Asleep at the Wheel: Human/AI Collaboration in a Field Experiment on HR Recruiters.” HBS/LISH, pre-registered (OSF qp8et). RCT, N = 181 recruiters, 5,184 analysed evaluations, 4 arms (99%/85%/75%/no AI), PIAAC ground truth. Working paper. Recruiters with the worse (75%) AI outperformed those with the better (85%) AI (p = .05 on the 1–10 scale). Effort mechanism measured: Bad AI +8.77 sec per evaluation (p<.01); Good AI −1.53 (n.s.). Theorem 1: combined human–AI accuracy is U-shaped in algorithmic precision. Experienced recruiters gained from bad AI and were harmed by good AI. This is the formal statement of what aviation learned empirically — pair it with Parasuraman & Riley.

5. Shen, J. H. & Tamkin, A. (2026). “How AI Impacts Skill Formation.” arXiv:2601.20245v1. Pre-registered RCT (OSF pk6a5), N = 52 professional programmers. Preprint; both authors at Anthropic — discloses against interest. Withdrawal design: AI arm 17% lower on unassisted quiz, d = 0.738, p = .010. No significant speed gain — so not even a speed/learning trade-off. Mechanism: controls hit 3× more errors (median 3 vs 1), correlated with score. Six interaction patterns; AI Delegation = fastest completion, worst learning.

6. Scarfe, P., Watcham, K., Clarke, A. & Roesch, E. (2024). PLOS ONE 19(6):e0305354. Covert real-world field study, peer reviewed. 33 AI accounts, 5 modules, ~5% of submissions. 94% undetected (97% on stricter criterion). AI scored ~5pp above real-student medians; 83.4% probability of outscoring a random real submission. Authors note they submitted unmodified output — “the most detectable way possible.” Converts a contested claim into a measured fact.

7. Koedinger, K. R. & Aleven, V. (2007). “Exploring the Assistance Dilemma in Experiments with Cognitive Tutors.” Educational Psychology Review 19(3), 239–264. Peer-reviewed review/synthesis — by far the most established scholarship in this stream. “How should learning environments balance information or assistance giving and withholding?” The answer is conditional on prior knowledge, task complexity, and strategic fading of scaffolding over time. Fading = currency. Same construct, two fields, zero cross-citation.

8. Boyd, V. A., Whitehead, C. R., Thille, P., Ginsburg, S., Brydges, R. & Kuper, A. (2018). “CBME: the discourse of infallibility.” Medical Education 52(1), 45–57. Foucauldian critical discourse analysis — non-empirical. CBME proponents deflect conceptual criticism as implementation failure, insulating the framework. Reductionism; behaviourist foundations limiting “reflection, intuition, experience”; no method exists to assess the six ACGME competencies as independent constructs; “paucity of empirical evidence.” Corroborated by Alharbi (2024), BMC Med Educ 24:612 — systematized review, 38 articles: 84% didn’t report evaluation approach, 63% omitted standards.

Also load-bearing: ten Cate (2005), Med Educ 39(12):1176–7 — note it is a two-page commentary. Five-level entrustment–supervision scale (ten Cate et al. 2020, NCBI NBK619392) — the key move is prospective/decisional (“ready for more autonomy?”) vs retrospective/evaluative (“how good were they?”). TEQSA (Lodge, Howard, Bearman, Dawson, 2023): Proposition 5 — “security at meaningful points across a program” is the higher-ed restatement of the checkride. Kizilcec, Chirikov & Smirnov (2026), Science 392:818–820 — survey, N > 95,000 undergraduates, 20 universities: 37% monthly use, 9% admitted cheating, 26% of daily users vs 7% of monthly users (self-report; 9% is a floor).

Two things aviation has that nothing else does

Recurrent proficiency checking. CBME builds competence and has essentially no machinery for detecting its decay. A pilot re-demonstrates competence every six months for forty years; a doctor does it once. If AI erosion is gradual, initial certification is structurally incapable of catching it.

The simulator. The withdrawal condition — the highest-value evidence design — is only routinely available where safe simulation exists. If you cannot construct a setting in which the assistant is absent, you cannot assess or maintain unassisted capability. That is a product requirement, not a nicety.

Convergence worth using: aviation (“secure the checkride, free the practice”), TEQSA Proposition 5, and medicine’s programmatic assessment independently reached the same architecture — formative unsecured practice punctuated by few, secure, consequential gates. Three fields, no cross-citation, same answer.

Findings I downgraded

  • “Your Brain on ChatGPT” (Kosmyna et al., arXiv 2506.08872). N = 54 (18/group; only 18 completed the crossover). Published commentary (Stanković et al., arXiv:2601.00856): needs ~N=159 for power; FDR level never stated; 1,000 per-electrode ANOVAs with unstated selection criteria; 2–4 essays per group in some analyses; the Search Engine group showed no impairment and this was underplayed. Contributes nothing the better studies don’t.
  • METR 19% slowdown (Becker et al., arXiv:2507.09089; N = 16 developers, 246 issues). CI +2% to +39%. METR’s Feb 2026 post reports the follow-up found the opposite sign (−18%/−4%) and disowns both datasets for selection — 30–50% of developers withheld tasks they expected AI to help with. Cite only the perception gap: expected +24%, measured −19%, still believed +20% afterwards. That ~39-point gap survives the critique.
  • Peng et al. (2023) 55.8%. 95% CI [21%, 89%]; 95 randomised but only 70 analysed; authors at GitHub Inc. and Microsoft Research; single greenfield HTTP-server task; authors state it “does not examine code quality.”
  • “Fast and Forgettable” (Gardella et al., arXiv:2604.18538). N = 22, convenience sample. Key retention finding fails multiple-comparison correction (adj. p = .054); absolute retest difference null (p = .397). The robust part is the workload result (large drops in mental demand and effort).
  • Brynjolfsson, Chandar & Chen “Canaries.” ADP payroll, observational. 22–25-year-olds in AI-exposed roles 19% below trend, operating through reduced hiring not separations. Authors call these “descriptive indicators… rather than causal estimates,” note attenuation when controlling for education, and in Feb 2026 revised the timing — significant only after 2024 under firm–time fixed effects.
  • GitClear code-quality reports. Vendor, observational, non-peer-reviewed. Cite as an industry signal only. That the best quality evidence is vendor-produced is itself a finding.
  • Attribution corrections: Burning Glass The Expertise Upheaval is by Melissa DiMarzio, not Sigelman (Sigelman owns the “start careers in the middle” framing). The SSIR entry-level piece is Bruno V. Manno, “A New AI Career Ladder,” 30 Oct 2025.

Gaps — put these in the report

  1. No long-run withdrawal study exists. Max exposure anywhere: four sessions. Max retention interval: one week, N = 22, underpowered. Career-scale degradation is an extrapolation.
  2. Nobody has demonstrated the pipeline consequence — that juniors deprived of junior work fail to become seniors. Supported by mechanism and analogy, not outcome data. That test takes a decade.
  3. No study shows an AI assistant producing better durable capability than unassisted practice. The best available result is a null (Bastani’s Tutor arm). Any “AI tutors work if designed right” claim over-reads the evidence.
  4. Almost nothing is replicated, and two of the most-cited results were qualified by their own authors within a year.
  5. The market runs on engagement, not learning. Kucirkova et al. (2025), Hum Soc Sci Commun 12:947 (peer-reviewed Delphi): educational impact “has not traditionally been a primary consideration in… procurement”; investors “prioritised scaling and user acquisition”; 75 competing evaluation frameworks. Combined with Bastani’s metacognitive finding — learners rate the harmful condition favourably — a satisfaction-optimising product will optimise toward the GPT Base arm. This is Bainbridge’s irony restated commercially.
  6. Telemetry has no unassisted condition. It structurally cannot measure the thing that matters, so a product can look excellent on every metric while producing the harmful outcome.