IR-Bench A Benchmark for LLM-Based Incident Response Reasoning

IR-Bench

A Benchmark for LLM Reasoning in Incident Response

Mudassir Syed · Updated June 2026

Partial Run

Claude Fable 5 completed 21 of 42 scenarios before Anthropic API credits ran out. Failed scenarios scored 0%, pulling the headline score to 45.5%. On completed scenarios only, Fable scored 91.8% — in line with o3 (91.5%) and Claude Sonnet 4.6 (91.4%). Run will resume when credits are restored.

42
Scenarios
6
Models
7
Reasoning Dimensions
9
Real Incidents

Model Comparison

Overall reasoning quality · Claude Fable 5 shown for 21 completed scenarios (*)

Claude Fable 5 — Partial Run

21/42 scenarios completed · API failures on remaining scenarios scored as 0%

Metric All 42 Scenarios 21 Completed Only
Overall 45.5% 91.8%
Hard scenarios 44.2% 91.5%
Medium scenarios 50.0% 92.9%
Scenarios with API response 21 / 42 (50%)

Dimension Breakdown

Score by reasoning dimension · includes Claude Fable 5 full run (42) and completed-only (21) rows

Model Decision Framing Tradeoff Articulation Uncertainty Ack. Conditional Rec. Second-Order Temporal Cal. Action Specificity

* Claude Fable 5 partial run: “all 42” includes API failures scored as 0%; “21 completed” reflects scenarios with a model response.

Performance Radar

Reasoning profile across 7 dimensions · Claude Fable 5 trace is 21 completed scenarios only

Hard vs. Medium Scenarios

Performance by difficulty · Claude Fable 5 bars reflect 21 completed scenarios (*)

About

Why Benchmark IR Reasoning?

Incident response is one of the hardest reasoning tasks in cybersecurity. When a production system is under attack, responders must make high-stakes decisions under time pressure, with incomplete information and competing tradeoffs. Should you roll back immediately or investigate first? How do you weigh customer impact against evidence preservation?

LLMs are increasingly being positioned as security copilots — tools that can assist during active incidents. But most evaluations test factual recall ("What is a SQL injection?"), not the multi-step reasoning that defines effective incident response.

IR-Bench fills this gap. Inspired by domain-specific benchmarks like TM-Bench for threat modeling, it evaluates whether LLMs can reason through the tradeoffs, uncertainties, and second-order consequences that real IR demands — not just recite the right answer.

Methodology

How It Works

Each model receives an open-ended incident response scenario grounded in a real-world security event, along with a detailed incident timeline. The model's response is then evaluated by an LLM judge (Gemini 2.5 Flash) against rubric-based criteria across 7 reasoning dimensions.

Dataset

42 scenarios across 9 real-world incident types: Cloudflare configuration errors, CrowdStrike sensor failures, HTTP/2 Rapid Reset DDoS, Heroku credential leaks, Colonial Pipeline ransomware, SolarWinds supply chain compromise, Log4Shell exploitation, MGM social engineering, and Cloudflare BYOIP outages. Scenarios are rated hard (0–3 point scale) or medium (0–2 point scale).

7 Reasoning Dimensions

decision_framing
Does the model frame the decision with appropriate urgency and identify the core question before diving into actions?
tradeoff_articulation
Does it explicitly name the competing concerns (e.g., speed vs. evidence, containment vs. business continuity)?
uncertainty_acknowledgment
Does it flag what is unknown and explain how that uncertainty affects the recommended approach?
conditional_recommendation
Does it provide branching logic — "if X then do Y, otherwise Z" — rather than a single fixed plan?
second_order_awareness
Does it anticipate downstream consequences of its recommendations, such as stakeholder impact or evidence destruction?
temporal_calibration
Does it sequence actions appropriately and distinguish between what to do now vs. what to defer?
action_specificity
Does it give concrete, actionable steps with thresholds and criteria, rather than vague directives?

Models Evaluated

o3 (OpenAI, reasoning-class) · Claude Sonnet 4.6 (Anthropic, reasoning-class) · Claude Fable 5 (Anthropic, reasoning-class — partial run, 21/42) · Gemini 2.5 Pro (Google, reasoning-class) · GPT-4o (OpenAI, general-purpose) · Mistral 7B (Mistral AI, open-source local)

Judging

All responses are scored by Gemini 2.5 Flash using pointwise rubric-based evaluation. Each dimension is scored independently with explicit rubric criteria. A dual-judge validation using GPT-5 (Azure) on an 18-scenario subset produced the identical model ranking, confirming scoring consistency.

Key Findings

What the Data Shows

Finding 01
Claude Fable: Partial Run, Frontier Performance
Claude Fable 5 scored 45.5% across all 42 scenarios, but only 21 completed before API credits expired — the rest scored 0%. On completed scenarios, Fable scored 91.8%, matching o3 and Claude Sonnet. Headline scores can misrepresent capability when runs are incomplete.
Finding 02
Three-Tier Performance Pattern
Frontier reasoning models (o3, Claude Sonnet 4.6) score ~91%, strong reasoning (Gemini 2.5 Pro) at 85%, and general-purpose models (GPT-4o, Mistral) at 46–54%. The gap between tiers is consistent across dimensions.
Finding 03
Second-Order Thinking Is the Hardest Skill
second_order_awareness is the lowest-scoring dimension for every model tested. Anticipating downstream consequences of IR decisions — evidence destruction, stakeholder cascades, regulatory triggers — remains challenging even for frontier models.
Finding 04
Reasoning Quality, Not Knowledge, Separates Models
All five models demonstrate awareness of the real incidents. The difference lies in how they reason through competing tradeoffs and structure their recommendations. GPT-4o knows the same facts as o3 but scores 38 points lower because it fails to articulate tradeoffs and conditional logic.
Finding 05
Architecture, Not Infrastructure
o3 and GPT-4o both run on Azure OpenAI. The 38-point gap between them confirms the performance difference is model architecture (reasoning-class vs. general-purpose), not platform latency, content filtering, or deployment differences.

Transparency

Limitations

  • Claude Fable partial run: Only 21/42 scenarios completed; remaining failures were API billing errors, not model refusals. Full-run score (45.5%) is not comparable to completed-run score (91.8%).
  • Dataset size: 42 scenarios show clear tier separation for complete runs, but are not statistically powered for fine-grained claims.
  • LLM-as-judge: Automated scoring has known biases. Mitigated by rubric-based pointwise scoring and dual-judge validation (GPT-5 on 18 scenarios produced identical ranking).
  • Judge-model overlap: Gemini 2.5 Flash (judge) is in the same model family as Gemini 2.5 Pro (tested). Disclosed; rubric anchors scoring to criteria.
  • Retrospective scenarios: All scenarios are based on past incidents. Real IR involves live, evolving information not captured here.