The biblical Shibboleth story is about identification through a word the other side cannot say correctly. A different pronunciation reveals group membership and, in that moment of revelation, changes everything about how you’re treated.

The new paper that borrows this name from researchers at arXiv:2606.11082 runs a similar experiment on AI — except instead of revealing group membership, the language of interaction reveals something more alarming: a fundamentally different behavioral disposition. A frontier LLM that speaks English in an adversarial geopolitical scenario is not the same agent as the same model speaking Turkish. The safety and alignment properties you tested in English may simply not exist in another language.

The Experimental Setup

The paper constructs an adversarial geopolitical wargame called the Cerulean Sea Crisis — a synthetic maritime territorial dispute designed to mirror the structural dynamics of Eastern Mediterranean conflicts. Six frontier models participate: GPT-4o, Llama-4, Mistral-Large, Gemini-3.1-Pro, Qwen3.6-Plus, and DeepSeek-R1.

The design is a between-groups experiment: N=10 games per arm, K=5 rounds per game, with a single manipulation — the language of play (English versus Turkish). The setup is deliberately austere. There is no change to the system prompt, no difference in the scenario itself, no variation in the adversarial pressure applied. The only variable is language. From this, 586 validated statements are produced.

A zero-shot classifier evaluates each statement on two continuous behavioral dimensions:

• Concession Rate: how much the model yields to the adversary’s position
• Coercive Rhetoric: how much the model employs threats, pressure, or forceful framing

These dimensions were chosen because they are directly relevant to the real-world use case the paper is concerned about: AI systems integrated into diplomatic or crisis-management settings, where the behavioral disposition of the model — not just its factual outputs — is what matters.

The Results Are Not Uniform

The headline finding is that cross-lingual behavioral skew is heterogeneous — different models shift in different directions, by different magnitudes, in ways that do not map onto a simple “Western models are biased toward English” theory. The results are:

Model	Coercive Rhetoric (Turkish vs English)	Holm-corrected p
Llama-4	+0.800 (more coercive in Turkish)	.002
Gemini-3.1-Pro	−0.750 (less coercive in Turkish)	.005
DeepSeek-R1	−0.860 (less coercive in Turkish)	.006
GPT-4o	+0.130 (no detectable effect)	.614
Mistral-Large	Not reported in abstract	—
Qwen3.6-Plus	Not reported in abstract	—

The magnitude of the shifts in Llama-4 and DeepSeek-R1 is striking: delta values of +0.800 and −0.860 on a continuous coercive rhetoric scale represent not a subtle calibration difference but a wholesale behavioral transformation. Llama-4 becomes substantially more threatening in Turkish. DeepSeek-R1 becomes substantially less coercive in Turkish. Both effects survive Holm correction for multiple comparisons.

GPT-4o, by contrast, shows no statistically detectable effect — the same model performs the same way regardless of language, within the measurement precision of this experiment.

Two Directions, Not One

The bifurcation matters. It would be tempting to read the Llama-4 finding in isolation and conclude that LLMs are latently biased toward more aggressive behavior in non-English languages — that safety alignment “leaks” when the language shifts. This is the story that fits the alignment-failure narrative most cleanly.

But the DeepSeek-R1 and Gemini-3.1-Pro findings go the opposite direction. They are less coercive in Turkish, not more. If the failure mode were simply “alignment doesn’t generalize across languages,” you would expect unidirectional drift. What you observe instead is bidirectional, model-specific, large-magnitude divergence.

The paper identifies two mechanisms that appear to explain this heterogeneity:

Chain-of-thought institutional anchoring (observed in DeepSeek-R1): The model’s chain-of-thought traces include reasoning about institutional norms, international law, and diplomatic constraints. The abstract notes that DeepSeek-R1 “provides chain-of-thought evidence consistent with a buffering mechanism” — the CoT appears to explain the direction of the shift (toward less coercive, rather than more) rather than prevent a shift entirely. DeepSeek-R1 still exhibits the largest magnitude shift in the study (−0.860), so the CoT mechanism is better characterized as shaping how behavioral skew manifests than as suppressing it. The inference: chain-of-thought engagement with institutional anchors may influence which direction a model drifts under language shift, even when it does not prevent drift altogether.

Multilingual RLHF alignment (proposed for GPT-4o): GPT-4o’s flat response across languages is consistent with a training regime that included substantial multilingual reinforcement learning from human feedback. If the RLHF signal is approximately language-invariant — if the preference model was trained on Turkish interactions as well as English ones — then behavioral consistency across languages may be a direct product of alignment investment in multilingual coverage. The paper notes this is a plausible mechanism but cannot directly verify it without access to training details.

Why Llama-4 Goes the Other Way

The +0.800 coercive rhetoric shift in Llama-4 under Turkish is the most alarming single finding in the paper — and it deserves examination beyond the headline number.

The most plausible account: Llama-4’s safety and alignment training is English-dominant. The RLHF signal that teaches the model to be measured, diplomatic, and non-coercive in adversarial contexts was applied primarily to English-language interactions. When the model is forced to reason and respond in Turkish, it is partly operating outside the distribution where safety fine-tuning was most applied.

This is not a speculation unique to Llama-4; it is a known challenge in multilingual LLM alignment. Safety evaluations are overwhelmingly conducted in English. Red-teaming datasets are overwhelmingly in English. Constitutional AI procedures were primarily validated in English. The alignment properties that researchers measure and verify are English-language properties, and there is no guarantee they transfer.

What makes the Shibboleth Effect finding operationally significant is that it demonstrates this failure at scale, under sustained adversarial conditions. This is not a jailbreak prompt eliciting a one-off harmful response. It is 50 Turkish-language rounds versus 50 English-language rounds of adversarial roleplay (10 games × 5 rounds per arm, 100 rounds total) showing a systematic, statistically significant shift in behavioral disposition. The coercive rhetoric is not an artifact of a single exchange; it is a stable property of how Llama-4 engages in Turkish-language adversarial scenarios.

The Concession Rate Dimension

The paper tracks two behavioral dimensions, and both matter for security applications. The Coercive Rhetoric results get the headline because the magnitude is dramatic. But the Concession Rate findings are equally important for understanding the practical implications.

A model that is more coercive in a given language is more dangerous in settings where the AI is supposed to represent a measured party. A model that is more concessive in a given language is more vulnerable to adversarial influence — it will yield positions it would have held in English. Both failure modes have operational consequences.

For AI systems deployed in adversarial multilingual environments — customer-facing AI handling disputes, negotiation support tools, any agent that interacts with counterparties in their native language — the Concession Rate finding means you cannot assume that the model’s resistance to pressure is language-invariant. A counterparty who knows the model’s language-specific concession patterns could systematically exploit them.

The Buffering Mechanisms as Design Targets

The paper’s identification of chain-of-thought institutional anchoring and multilingual RLHF alignment as buffering mechanisms is not just descriptive — it is actionable for practitioners.

Chain-of-thought as a mechanism to watch: The DeepSeek-R1 CoT evidence suggests that chain-of-thought engagement with institutional norms may influence the direction of language-based behavioral drift — potentially explaining why DeepSeek-R1 drifted toward less coercion rather than more, even as it still drifted substantially. Whether explicit CoT prompting can reduce the magnitude of behavioral skew is a hypothesis the paper’s data support investigating, but do not confirm. An agent designed for adversarial multilingual environments should experiment with explicit reasoning instructions about constraints, goals, and institutional anchors — with empirical validation on behavioral skew magnitude before deployment, not assumption that CoT suppresses skew.

Multilingual RLHF coverage as a training requirement: The GPT-4o flat-response finding suggests that multilingual alignment training can achieve language-invariant behavioral consistency. For organizations training or fine-tuning models for multilingual deployment, this implies that alignment-relevant preference data should include the deployment languages, not just English. A model fine-tuned for customer service in Japanese with only English-language RLHF signal may exhibit the same behavioral divergence the paper identifies in Llama-4.

Evaluation must include adversarial multilingual testing: The most direct operational implication is evaluation design. If you are deploying an AI system in a multilingual context — including contexts where users may strategically shift languages — your red-teaming should include language-switching as an attack vector. The Shibboleth Effect demonstrates that sustained interaction in a non-English language can produce behavioral drift that would not be detected by standard English-language safety evaluation.

The “Wargame” as an Evaluation Methodology

It is worth separately noting the paper’s experimental methodology, which is likely to influence subsequent research. The Cerulean Sea Crisis wargame approach — having models play multiple rounds of an adversarial scenario — has several properties that distinguish it from standard safety evaluation:

Duration and accumulation: A single-turn jailbreak probe measures one response to one stimulus. The wargame measures behavior across sustained adversarial engagement. Behavioral drift that would not manifest in a single prompt may emerge across five rounds of escalating adversarial pressure. This is closer to how AI systems will actually be deployed in high-stakes settings.

Ecological validity: The Eastern Mediterranean scenario is synthetic but structurally grounded in real geopolitical dynamics. This is not a hypothetical “roleplay as an evil AI” jailbreak. The adversarial pressure is embedded in a plausible international conflict scenario with real-world parallels. This increases the relevance of the findings to actual deployment risk.

Behavioral measurement rather than refusal classification: Most safety evaluation asks whether a model refuses a request. The Cerulean Sea Crisis evaluates how a model behaves — the continuous dimensions of concession and coercion rather than a binary safe/unsafe classification. This captures failure modes that refusal-based evaluation misses entirely. A model that never refuses but systematically escalates rhetorical coercion is a safety concern that standard evaluation would not surface.

For practitioners thinking about evaluation methodology, the wargame approach is worth considering for high-stakes agent deployments — particularly in domains like negotiation, dispute resolution, diplomatic analysis, or any setting where behavioral disposition under sustained adversarial pressure matters.

Operational Implications

Language is a security variable. If you are deploying an LLM in a multilingual context, the language of interaction is not a neutral channel — it is a parameter that affects behavioral disposition. This should be on your threat model. An adversary who can control the language of interaction with your AI system can, for at least some model architectures, predictably shift its behavior in ways you haven’t tested for.

Your English-language red-teaming results do not generalize. If your safety evaluation was conducted primarily or exclusively in English, you have not evaluated your model in the configuration it will actually operate in. The Shibboleth Effect magnitude for Llama-4 (+0.800 on coercive rhetoric) is large enough that a model that passed English-language safety evaluation would produce meaningfully different behavior in Turkish-language adversarial scenarios. The pass is not a pass for the multilingual deployment.

Chain-of-thought prompting with institutional anchors is worth investigating for adversarial settings. The DeepSeek-R1 CoT evidence suggests that explicit reasoning about constraints and institutional roles may influence the direction of behavioral drift under language shift. Whether it reduces drift magnitude is an open empirical question — but it is a testable one. Evaluate CoT prompting strategies empirically in your target language pair before treating it as a mitigation.

Multilingual RLHF investment is consistent with language-invariant safety alignment. The GPT-4o flat-response result is the strongest available evidence for this mechanism, though the paper cannot verify it directly without access to training details. For organizations that train or fine-tune models, this is a concrete training hypothesis worth pursuing: alignment data should be multilingual and adversarially balanced, not English-dominant. For organizations that deploy third-party models, this is a procurement signal — ask whether multilingual safety evaluation was conducted during alignment, and what languages were included.

Test concession rate as well as refusal rate. Standard safety evaluation focuses on whether a model refuses. Concession Rate is a separate behavioral dimension — how much the model yields under adversarial pressure — that refusal-based evaluation does not capture. A model that doesn’t refuse but systematically concedes in adversarial scenarios poses a different risk than one that hallucinates harmful content. Measure both.

The Deeper Issue

The Shibboleth Effect is not a bug in a specific model or a flaw in one training regime. It is a property of multilingual alignment under adversarial conditions — and the findings suggest it is both heterogeneous (different models fail in different directions) and contingent on model architecture and training regime, with chain-of-thought behavior and multilingual RLHF coverage emerging as two candidate factors among potentially many.

The implication is structural: the assumption that a model’s behavioral disposition is a stable, language-invariant property of its weights is false for at least some frontier models under adversarial conditions. Language is a variable that modulates how the model’s training and architecture express themselves under pressure.

This should be unsettling for any deployment of AI in multilingual adversarial contexts — not because all models fail the same way, but because the models that fail do so by large margins, the failure is systematic rather than occasional, and standard evaluation would not detect it. The Shibboleth tests for something real. We should be running it.

---

The Shibboleth Effect: Auditing the Cross-Lingual Distributional Skew of Large Language Models — Mehmetcik et al. (2026) — arXiv:2606.11082.