ICLR 2026 · LLM Reasoning Workshop · with Igor Bogdanov & Chung-Horng Lung
What do LLM agents say right before they break a commitment?
Across 84,540 trajectories and 8 model families, we read what agents wrote in the moments before abandoning a goal, and built a seven-category taxonomy of their stated reasons, validated against human audit (κ = .83).
What I did
I authored the failure-rationale taxonomy and led the semantic analysis: a human pilot on 100 traces built the seven-category codebook, an LLM labeled all 14,025 pre-failure traces, and a stratified human audit of 200 verified the labels at κ = .83. The surrounding benchmark manipulates social visibility, persona stressors, and deliberation policy.
What I found
Failure has a temporal structure: early failures are impulse-driven, late ones are framed as fatigue and cost-benefit calculation. Public settings breed norm-based excuses. And among failures, longer deliberation correlates with more self-contradiction: more reasoning text ≠ more consistency.
Why it matters
Terminal success rates hide when and why agents drift. Failure fingerprints are model-specific, which makes this taxonomy a practical diagnostic lens for anyone evaluating agent reliability, including safety teams auditing rationales.
The seven things agents say when they give up
Share of 13,780 pre-failure rationales in each taxonomy category. Hover a bar for the category's definition.
245 further traces (1.7%) were flagged novel/ambiguous and excluded. Labels: LLM-assisted classification with the fixed schema; agreement with human audit κ = .83.
Failure reasons shift over the episode
Category share among early failures (steps 0–5) vs late failures (steps 14–19).