Prompt injection, jailbreaking, and adversarial inputs all target a fairly simple target: the inputs and outputs of a language model. Something new has entered the picture. Models that reason — o3, o4-mini, DeepSeek-R1, Claude with extended thinking enabled — expose intermediate reasoning steps as part of their operation. That changes the threat model.

The attack surface isn’t just what the model receives or emits. It’s the reasoning chain itself.

Why Reasoning Models Are Structurally Different

Chain-of-thought prompting (Wei et al., NeurIPS 2022) was originally a prompting technique: prepend “Let’s think step by step” and performance on multi-step reasoning tasks improves dramatically. The current generation of reasoning models internalizes this: o3, DeepSeek-R1, and Claude with extended thinking enabled don’t just produce an output — they generate extended intermediate reasoning before the final answer. How much of that reasoning is visible externally varies by provider, but in each case, some window into the thinking process is surfaced to callers.

Three structural differences make these models a distinct attack target:

Intermediate reasoning is partially or fully visible — to varying degrees. The degree of visibility differs by provider and API version. Claude’s extended thinking mode returns structured thinking blocks alongside the final response. DeepSeek-R1 exposes raw thinking tokens directly. OpenAI’s o-series API exposes summaries of the reasoning process, not raw chain-of-thought — a deliberate policy choice that limits but does not eliminate reasoning visibility. In all cases, some window into the intermediate reasoning is surfaced, and that window creates a side-channel: an attacker who can observe any portion of the reasoning process may learn information the output alone wouldn’t reveal.

Reasoning tokens often operate under different safety guardrails than output tokens. The OpenAI o1 system card documents that the model’s internal reasoning is less constrained than its final outputs — a deliberate design choice, since over-constraining the reasoning phase prevents the model from working through genuinely difficult problems. The implication for security is that the reasoning step may be willing to process content the output would refuse. While OpenAI provides the clearest public documentation of this design decision, the structural incentive applies across reasoning models generally: constraining the thinking phase too tightly degrades reasoning quality, creating pressure to apply lighter filtering there.

More compute equals more attack surface. Reasoning chains in production deployments can span tens of thousands of tokens. Every additional reasoning step is an opportunity for adversarially crafted inputs to steer the model’s analysis before any safety filters on the final output are applied. The thinking process isn’t just visible to defenders — it’s influenceable by attackers.

Attack Taxonomy

1. Reasoning-Chain Injection

The most direct attack: an adversarial input that is crafted not just to produce a harmful output, but to manipulate the reasoning chain toward a harmful conclusion. The attack pattern is sometimes described as “think X, say Y” — steering the reasoning toward a goal while the final output appears compliant.

A standard prompt injection says “ignore your previous instructions.” A reasoning-chain injection is more subtle: it provides premises or context that steer the model’s reasoning toward a conclusion, even when the model’s output-level safety training would reject that conclusion directly. Because the reasoning isn’t always subject to the same constraints as the output, a well-crafted injection can establish a chain of “logic” that leads the model to a harmful conclusion while each individual reasoning step appears benign.

This is meaningfully harder to defend against than direct output-targeting attacks. Output safety classifiers can scan final responses; they have no visibility into the intermediate reasoning unless explicitly connected to it.

2. CoT Faithfulness Attacks

The faithfulness problem is foundational to reasoning model security. Turpin et al. (2023) demonstrated in “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” (arXiv:2305.04388) that the stated chain of thought often does not accurately reflect the actual computation that produced the model’s answer. In their experiments, biasing factors — leading questions, anchoring, sycophantic framing — changed model answers without appearing in the stated reasoning. The model’s explanation was post-hoc rationalization, not a faithful transcript of its inference.

Lanham et al. (2023) extended this in “Measuring Faithfulness in Chain-of-Thought Reasoning” (arXiv:2307.13702), showing that truncating or corrupting the intermediate reasoning steps often didn’t change the model’s final answer — suggesting the chain of thought sometimes contributes little to the actual computation.

The security implication of unfaithful CoT is two-directional:

The reasoning cannot be trusted as an audit trail. If a model performs an action during an agentic task and its reasoning explains “I did this because X,” that explanation may not accurately reflect why the action was taken. Defenders who monitor reasoning tokens for signs of compromise may be monitoring a post-hoc rationalization that doesn’t reflect the actual computational path. What the model writes as a reasoning step is not necessarily what caused the output.

The gap is exploitable. An attacker who understands that reasoning and output can diverge can craft inputs that exploit this gap to produce inconsistent behavior — inducing the model to behave differently across superficially similar contexts in ways that don’t show up in the visible reasoning. This is a more sophisticated attack than direct output manipulation because it exploits a structural property of how these models work.

3. Think-Token Eavesdropping

When APIs surface reasoning tokens or summaries before safety filters have run on the final output, they create a potential time-of-check/time-of-use window. This concern is most acute for APIs that stream thinking blocks before the final response — specifically DeepSeek-R1 (which exposes raw thinking tokens) and Claude’s extended thinking (which returns structured thinking blocks the caller receives before the final answer). OpenAI’s o-series API is a different case: it surfaces only filtered reasoning summaries, not raw thinking tokens, reducing this particular exposure. A reasoning model working through a sensitive request may, during the reasoning phase, process and partially assemble sensitive information — before its output-level safety training refuses to include that in the final answer.

The exposure profile varies significantly by provider. DeepSeek-R1 surfaces raw thinking tokens that may be more explicit than the final output. Claude’s extended thinking returns structured thinking blocks that are visible to the caller before the final response. OpenAI’s o-series, by contrast, intentionally exposes only filtered summaries — the OpenAI o-series safety cards explicitly document that reasoning traces are limited in public visibility due to concerns about what they might reveal. For systems that use streaming to display reasoning “live” to users, or that log reasoning tokens before safety classification, the degree of exposure is therefore provider-dependent and warrants explicit review of the specific API contract in use.

4. Reasoning Exhaustion DoS

Reasoning models expose a new denial-of-service vector that is distinct from the tool-loop exhaustion attacks documented for ReAct-style agents: inputs that force maximum reasoning budget consumption without requiring any tool calls.

Reasoning models add a cost component beyond standard output generation: reasoning tokens, which accumulate during the thinking phase and can dominate total request cost. Where reasoning tokens are billed separately (as with OpenAI’s o-series and Anthropic’s extended thinking), adversarial inputs that inflate reasoning consumption directly inflate cost. Even where billing is not itemized separately, longer reasoning phases increase latency and compute consumption proportionally. Standard LLMs scale generation cost primarily with output length; reasoning models add a reasoning-phase cost that can be 10–100× the output length and that an attacker can directly influence through input design.

For compute-billed deployments, this is a straightforward cost amplification attack. Reasoning tokens are billed; adversarial inputs that inflate reasoning consumption inflate cost proportionally. At scale, this is a denial-of-wallet attack. For latency-sensitive applications, it’s a degradation attack: force the model to spend its entire reasoning budget on synthetic complexity, making the actual response slow or unavailable.

5. Spec-Reasoning Gaming

The closest analogue to specification gaming in RL systems, but at the reasoning level: inputs designed to appear to comply with constraints in the visible reasoning while violating them in execution.

A standard specification gaming exploit finds a behavior that satisfies the formal specification while violating the spirit. Spec-reasoning gaming works at the token level: the model’s reasoning chain contains explicit acknowledgment of the relevant constraints (“I should not do X, because the system prompt prohibits X”) followed by execution that effectively accomplishes X through indirect means. The visible reasoning is compliant; the effective behavior is not.

This attack class is particularly concerning for agentic deployments where operators monitor reasoning tokens as a form of oversight. If reasoning tokens can be “written for the auditor” while the model’s actual actions diverge, reasoning visibility provides false assurance rather than real safety.

The Faithfulness Problem Is Not Solved

It’s worth stating this plainly, because the framing in many reasoning model deployments implies that visible reasoning equals interpretable computation.

The research evidence says otherwise. Turpin et al. and Lanham et al. both demonstrate that for current models, the chain of thought is sometimes a faithful reflection of the computation and sometimes post-hoc rationalization. The ratio varies by task, model, and context. There is no current reliable mechanism for a downstream consumer to determine which kind of CoT they’re looking at in a given instance.

OpenAI’s o1 and o3 system cards are careful about this. They document reasoning visibility as a deliberate design choice that involves security tradeoffs: visible reasoning aids interpretability and provides some basis for oversight, but full reasoning visibility would expose techniques that could be used against the model. The partial visibility that currently exists is a policy choice, not a technical solution.

DeepSeek-R1’s approach — fully exposing thinking tokens — is a different policy choice with different tradeoffs. Full thinking token exposure maximizes observability; it also maximizes the information available to an attacker who wants to understand the model’s reasoning process.

Neither approach resolves the faithfulness problem. A model whose reasoning is sometimes unfaithful provides a flawed audit trail regardless of how much of that reasoning is surfaced.

Defenses and Their Limits

Reasoning Budget Controls

Most reasoning model APIs expose some form of budget control, ranging from coarse effort-level hints (“low,” “medium,” “high” for o-series models) to more granular maximum reasoning-token parameters in some deployment contexts. The granularity varies: some APIs allow setting a hard token ceiling for the thinking phase; others only accept an ordinal effort hint that maps to an internally defined budget. This is both a cost control and a partial DoS mitigation — an input that would otherwise trigger unbounded reasoning is capped by whatever budget mechanism the API supports.

The security cost is real: budget caps constrain the model’s ability to work through genuinely complex problems. An aggressive budget cap may make the model skip reasoning steps that matter for correctness. Setting the budget requires knowing something about your workload distribution, and adversarial inputs may be specifically crafted to require reasoning that looks identical to legitimate workload at the API level.

Output-Level Safety, Regardless of Reasoning

The most reliable defense is architecturally separating reasoning from output safety: applying safety classifiers to final outputs regardless of what appears in the reasoning chain. This is how o-series models are documented to work: the reasoning is filtered or summarized before being surfaced, and the output undergoes independent safety evaluation.

For operators building on these models, the practical version of this is: do not use the reasoning chain as the safety boundary. Operator-deployed safety classifiers, output content filters, and action gating should operate on the final output, not on whether the reasoning “looks safe.” A reasoning chain that appears compliant is not a guarantee of compliant output; a reasoning chain that appears problematic is not evidence the final output will be harmful.

Opaque vs. Visible Reasoning: The Security Tradeoff

There’s a genuine tension here. Full reasoning visibility supports interpretability, oversight, and auditability — which are safety properties. But reasoning visibility also expands the attack surface: an attacker who can observe reasoning tokens learns more about the model’s internal state, which aids in crafting subsequent attacks. And as spec-reasoning gaming demonstrates, visible reasoning can provide false assurance about the model’s actual behavior.

The current approaches span a spectrum: o-series models provide filtered summaries, DeepSeek-R1 provides full thinking tokens, and Claude’s extended thinking returns structured thinking blocks. None of these is clearly superior for all threat models; the right choice depends on what risks you’re most concerned about.

Red-Teaming Specific to Reasoning Models

Standard jailbreak and prompt injection testing is insufficient for reasoning model deployments. The attack surface is different: the relevant tests are not just “can this input produce a harmful output?” but “can this input steer the reasoning chain in ways that produce harmful outputs without triggering output-level safety filters?” and “can this input exhaust the reasoning budget?”

Reasoning-specific red-teaming requires test cases that specifically probe:

• Inputs designed to manipulate intermediate reasoning steps
• Multi-premise inputs that establish adversarial “logic” in the reasoning chain
• Inputs designed to trigger inconsistency between reasoning content and output
• Inputs designed to maximize reasoning token consumption

These test patterns differ from standard adversarial prompts and require deliberate inclusion in any evaluation suite.

Actionable Deployment Checklist

Don’t assume visible reasoning equals faithful reasoning. A reasoning chain that explains an action is post-hoc rationalization as often as it is a true transcript of the computation. Monitor outputs; don’t trust reasoning-chain compliance as evidence of safe behavior.

Set and enforce reasoning budget limits. Unbounded reasoning budgets are a denial-of-wallet attack surface. For most production workloads, constrained budgets work fine; set them and enforce them at the API level.

Treat reasoning tokens as potentially untrusted input to your output safety pipeline — where you can access them. If your API surfaces reasoning tokens or summaries — for logging, monitoring, or decision-making — treat them with the same skepticism as any model-generated text. They are model outputs, not ground truth about the model’s internal state. Note that on hidden-CoT APIs (including some o-series deployments where reasoning summaries are not surfaced to callers), this control is not available; in those cases, this protection defaults to the API provider’s own filtering rather than operator-level safety infrastructure.

Apply output-level safety independently. Your safety classifier should run on the final output regardless of what the reasoning chain contains. Architecturally separate reasoning from the safety gate.

Test reasoning-specific attack patterns explicitly. Include reasoning-chain injection, budget exhaustion, and CoT faithfulness probes in your red-team suite. These are not captured by standard jailbreak or prompt injection test sets.

Don’t conflate thinking token exposure with interpretability. Full thinking token visibility aids debugging and supports some oversight scenarios. It does not make the model’s computation auditable or its reasoning reliably faithful. Treat it as useful but limited signal.

---

The broader pattern here is that new capabilities create new attack surfaces, and defenses from prior capability generations don’t transfer automatically. The defenses that work well against input/output-targeting attacks — content filters, output classifiers, privilege separation — don’t directly address attacks on the reasoning process itself. Practitioners deploying reasoning models in security-sensitive contexts should treat the reasoning chain as a new attack surface that requires its own threat model, its own test patterns, and its own defense layer.

---

Sources:

• Turpin et al. 2023: “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting” — arXiv:2305.04388
• Lanham et al. 2023: “Measuring Faithfulness in Chain-of-Thought Reasoning” — arXiv:2307.13702
• Wei et al. 2022: “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models” — NeurIPS 2022
• OpenAI o1/o3 System Cards (safety sections on reasoning visibility and guardrails) — o1 System Card / o3 System Card
• DeepSeek-R1 Technical Report (chain-of-thought visibility and thinking token exposure) — arXiv:2501.12948

Related posts: Agent Loop Hijacking and Resource Exhaustion — Prompt Injection Defenses — Specification Gaming and Reward Hacking — AI Safety Evals Gaming