In March 2023, OpenAI disabled ChatGPT’s conversation history feature for several hours. The reason, disclosed in an incident report the following day: a bug in a Redis client library had allowed some users to see conversation titles — and in some cases, the first message of a conversation — belonging to other users. A small number of users also had payment information exposed in their subscription confirmation emails. This was not an adversarial attack, not a sophisticated zero-day, not an AI-specific failure mode at all. It was a caching library bug in a high-velocity SaaS product running shared infrastructure at scale.

That incident is the entry point for this post. The March 2023 ChatGPT episode illustrates the specific class of risk that comes with multi-tenant LLM deployments: not attacker ingenuity, but infrastructure failures that cause one customer’s data to surface in another customer’s session. This is a distinct threat class from adversarial prompt extraction (where an attacker crafts inputs to elicit your system prompt) and from training data memorization (where a model regurgitates content from its training corpus). What we are discussing here is isolation failure — the shared infrastructure beneath the LLM application failing to keep one tenant’s data from reaching another tenant’s view.

For enterprises evaluating or operating LLM services, understanding this threat class is prerequisite to making credible risk assessments. GDPR Article 32 requires “appropriate technical measures” to ensure data security; a cross-customer data exposure is a reportable breach under most frameworks. The questions this post concludes with are not academic — they are the ones that belong in a vendor security questionnaire.

The Incident Record: March 2023 ChatGPT Redis Bug

On March 20, 2023, OpenAI received reports from users who, upon logging into ChatGPT, saw conversation history entries that did not belong to them — titles of conversations they had never initiated. In the subsequent investigation, OpenAI identified the root cause: an open-source Redis client library, redis-py, contained a bug in its connection pool implementation that under specific concurrent load conditions could return data from one client’s connection to a different client’s request.

The specifics matter. OpenAI’s redis-py client used asyncio’s async Redis client. The bug was in how the library handled canceled async requests: when an asynchronous request was canceled mid-flight, the library returned the underlying connection to the pool before draining its pending response data. The next consumer of that pooled connection received the prior user’s buffered response data instead of its own. This is distinct from a generic “race condition” — it was specifically triggered by request cancellation under load, and the fix required correcting the connection state cleanup after async cancellation.

What was exposed?

• Conversation titles: The sidebar list showing a user’s conversation history pulled from Redis. Affected users saw conversation titles from other users’ histories in their own sidebar.
• Conversation content: OpenAI’s disclosure indicated that in some cases, potentially more than just the title of another user’s conversation could have been exposed through the cross-contaminated cache responses.
• Payment information: A small number of users received subscription confirmation emails that included payment card details belonging to a different user — specifically first and last name, email address, payment address, credit card type, last four digits, and expiration date. This occurred because the same connection bug affected the data pipeline populating those confirmation emails.

OpenAI’s incident report stated that approximately 1.2% of ChatGPT Plus subscribers who were active during a specific 9-hour window may have had their payment information exposed in another user’s subscription email. The scope of conversation-title exposure was reported separately and affected a broader (but similarly constrained by the active-session window) population. The company notified affected users, took the feature offline while the bug was patched, and upgraded their Redis client.

Several things make this incident instructive beyond its specifics:

It required no attacker. The data exposure happened through normal user activity — high concurrent load on a shared caching layer — without any adversarial action. This is the defining characteristic of infrastructure isolation failures: they are triggered by load patterns, race conditions, and configuration mistakes, not by malicious queries.

The affected data was not encrypted-in-use. User conversation data moving through a caching layer is typically decrypted in memory to be served efficiently. Once the isolation boundary failed, there was nothing preventing the mixing of plaintext records.

It was fast and hard to observe. The window during which cross-user data appeared was brief per session, and most users would not have known to screenshot or report it. Auditing the actual scope of exposure required OpenAI’s internal access to session logs — something external users and regulators could not independently verify.

The fix was a library patch, not an architecture change. Updating redis-py resolved the specific bug. But the underlying architectural question — whether conversation history for all users should flow through a shared in-memory caching layer — was not eliminated, only made conditionally safe by the corrected client behavior.

The Threat Landscape: Four Failure Modes

The March 2023 incident represents one instance of a broader class. Multi-tenant LLM deployments face four distinct architectural failure modes, each with different likelihood profiles and different mitigations.

1. Shared Session Storage Bugs (Confirmed Real Pattern)

The ChatGPT Redis incident is the canonical instance. Any time conversation history, user context, or session state passes through shared, high-performance storage — Redis, Memcached, in-process caches — the isolation model depends on both the correctness of the storage system and the correctness of the client code that accesses it.

This failure mode is not LLM-specific. Shared caching systems have experienced similar isolation failures in conventional web applications: session fixation bugs, cache poisoning, key collision under high load. What LLM deployments add to the mix is (a) longer, richer session state (multi-turn conversation histories carry more sensitive context than a typical web session cookie) and (b) higher velocity of context writes as users interact with longer AI responses.

The risk is highest in:

• High-velocity SaaS deployments where conversation context must be written and read at low latency across many concurrent users
• Shared infrastructure deployments where a single Redis or cache cluster serves multiple tenants without strong per-tenant namespacing
• LLM wrapper frameworks (LangChain, LlamaIndex, custom orchestration) where memory and history objects may inadvertently be shared across request handlers — a well-documented failure mode sometimes called “session bleed” in multi-threaded or serverless deployments

2. KV-Cache Timing Side-Channels (Plausible, Not Peer-Reviewed)

Modern LLM serving systems cache intermediate computation results — key-value (KV) attention activations — for shared prompt prefixes, reducing inference costs. When two requests begin with the same token sequence, the second request can skip recomputing those early layers by reusing the cached activations. This creates an observable timing differential: cached requests return the first token measurably faster than uncached requests.

The architectural risk: an attacker who can measure time-to-first-token across requests can construct a cache oracle — a probe that determines whether a specific, known prefix is currently cached. Within whatever namespace the attacker shares with a potential victim, a faster-than-baseline response on a test prefix indicates that some other caller recently used that prefix. This is not a general-purpose eavesdropping mechanism; it requires the attacker to already know or guess the prefix they’re testing for. But it enables configuration inference attacks: an attacker with a strong candidate set of system prompt templates can probe which one a target deployment is using.

The practical scope is constrained by namespace partitioning. Under per-account namespacing (the model described by OpenAI’s and Anthropic’s prompt caching documentation), a timing probe by one account cannot observe activity in another account’s namespace. The risk materializes where:

• Multiple deployments within the same account share a cache namespace
• A multi-tenant application platform multiplexes its users under a single LLM provider account without per-tenant cache namespace separation
• Provider isolation is implemented at the application layer rather than the storage layer, and a transient misconfiguration creates a cross-account bleed window

⚠️ Empirical status. Timing oracle attacks against production LLM KV caches have not been independently published in peer-reviewed literature as of this writing. The architecture is analogous to the well-demonstrated Flush+Reload class of CPU cache attacks; the practical signal-to-noise ratio under real-world network jitter and provider-side latency normalization is unconfirmed.

3. System Prompt Bleed (Confirmed Bug Class, Rare Production Reports)

System prompt bleed occurs when a tenant’s system prompt — or fragments of it — persists into a subsequent tenant’s session due to a session boundary failure. This is distinct from adversarial extraction (where an attacker deliberately elicits a system prompt through crafted user turns): bleed happens passively, through bugs in how the serving infrastructure resets state between sessions.

The mechanism: an LLM serving system maintains a constructed prompt for the active session, combining the system prompt, conversation history, and the new user turn. At session end, that constructed prompt should be discarded. If the session boundary is incorrectly implemented — or if an object holding constructed prompt context is reused across sessions rather than instantiated fresh — the next session may receive a context window that already contains prior session content.

This has been documented in:

• LLM application frameworks (LangChain’s ConversationBufferMemory and similar) when a memory object is instantiated once at module load rather than per-request in serverless or multi-threaded environments
• Custom LLM proxy implementations where request-level context is held in a mutable global rather than a per-request scope

What gets exposed in system prompt bleed? System prompts increasingly carry operationally sensitive content: business logic and rules the operator wants the model to follow, topic restrictions, persona definitions, and in some (inadvisable) configurations, credentials or API keys. The bleed recipient — the next user’s session — receives this context without being its intended audience, and depending on the LLM’s behavior, may be able to elicit it explicitly or may see its effects in unexpectedly shaped model responses.

For multi-tenant SaaS platforms built on top of a shared LLM backend — where each customer has their own system prompt but shares the serving infrastructure — this failure mode amounts to one customer’s proprietary configuration leaking to a competitor’s session.

4. Shared Batch Inference (Theoretical Risk Under Implementation Failure)

LLM inference at scale uses request batching: multiple prompts from different users are processed together in a single forward pass through the model. This dramatically improves GPU utilization. In correctly implemented batching, requests are isolated at the attention mask level — each request in a batch sees only its own tokens, enforced by masking that prevents attention heads from attending across request boundaries.

In principle, this isolation is tight. In practice, the correctness of batch isolation depends on the serving framework, the model architecture, and any efficiency optimizations applied at the operator level. Published serving frameworks (vLLM, TensorRT-LLM, Triton Inference Server) implement per-request attention masking as a core feature. What is not publicly documented is whether all production deployments of batched inference at scale apply these controls uniformly, and whether optimizations (speculative decoding, continuous batching variants) ever create conditions where request boundaries become ambiguous.

⚠️ Theoretical classification. Cross-request leakage via shared batch inference has not been demonstrated against production systems and would require a specific implementation failure in attention masking. This is a structural risk analysis, not a report of observed attacks.

Provider Isolation Models: What Is and Isn’t Documented

Major LLM API providers offer managed services that abstract away the inference infrastructure. Their documented isolation models provide a baseline for risk assessment — while also revealing significant verification gaps.

OpenAI (ChatGPT / API)

OpenAI’s API documentation states that prompt caching (automatic for prefixes ≥1,024 tokens) applies within per-organization namespaces. The March 2023 incident was a consumer product (ChatGPT) incident, not an API incident, and affected the conversation history storage layer rather than the prompt cache. OpenAI’s API terms state that API data is not used to train models by default and is retained for a limited period (30 days as of recent updates to their data policies). What is not documented: how cache namespace isolation is enforced at the storage layer, what the namespace partition granularity is (organization, project, region), and how isolation is maintained during infrastructure events.

Azure OpenAI Service

Azure OpenAI’s Provisioned Throughput Units (PTU) provide reserved model throughput capacity — your workload is served from capacity allocated to your organization, reducing contention with other tenants. This may reduce cross-tenant batch inference risks compared to on-demand shared compute, though Microsoft’s public documentation frames PTU as a throughput reservation, not as a guarantee of physically single-tenant serving hosts. Operators with strict isolation requirements should request an explicit architectural statement from Microsoft and ensure it is reflected in contract terms. For standard (shared on-demand) deployments, the same gap exists as with other providers: storage-layer isolation details are not publicly specified.

AWS Bedrock

Bedrock’s architecture documentation describes per-account isolation at the API level, with data not shared across accounts. AWS’s responsibility model places tenant isolation at the service layer — Bedrock is a managed service where AWS owns the inference infrastructure and makes isolation guarantees through its standard multi-tenant service model. Bedrock’s on-demand inference uses shared compute; Provisioned Throughput (analogous to Azure’s PTU) provides reserved capacity. AWS’s compliance documentation covers SOC 2, ISO 27001, and other frameworks but does not detail storage-layer KV cache isolation specifics.

Anthropic (Claude API)

Anthropic’s documentation for its cache_control API explicitly states that cache entries are namespaced per-account and are not shared across accounts. The documentation specifies TTL (5 minutes default; an extended 1-hour TTL is available in beta) and minimum token thresholds per model tier. On storage-layer isolation implementation: not documented publicly.

The shared gap. All major providers offer per-account isolation as a stated commitment. None publish storage-layer architecture documentation confirming how that isolation is enforced (routing layer, storage layer, or both), what happens during failover or cache migration, or whether cache hit status is available in audit logs. This is not unique to LLM providers — most multi-tenant cloud services make similar commitments without publishing storage-layer architecture — but it represents a verification gap that enterprise risk frameworks should account for.

GDPR and Enterprise Compliance Framing

Cross-customer data exposure in an LLM deployment is not just a security incident — it is a data protection compliance event.

GDPR Article 32 requires controllers and processors to implement “appropriate technical and organisational measures” to ensure a level of security appropriate to the risk, including “as appropriate… the ability to ensure the ongoing confidentiality, integrity, availability and resilience of processing systems and services.” Multi-tenant isolation failures that allow personal data from one user or customer to reach another are precisely the class of failure Article 32 is designed to prevent.

GDPR Article 33 requires controllers to notify their supervisory authority of a personal data breach within 72 hours of becoming aware of it, unless the breach is “unlikely to result in a risk to the rights and freedoms of natural persons.” A cross-customer leak of conversation history, payment information, or any user-identifiable data would typically meet the reportability threshold, though whether Article 33 reporting was legally required in any specific instance is a legal determination. The March 2023 ChatGPT incident attracted significant regulatory attention: the Italian Data Protection Authority (Garante) issued an emergency provisional order on March 31, 2023 temporarily suspending ChatGPT in Italy, citing multiple GDPR concerns including the March 20 data exposure alongside concerns about lawful basis for data processing and age verification — illustrating how a single data exposure event can trigger broader regulatory investigation.

Data Processor Agreements. For enterprises using LLM APIs to process personal data under GDPR, the provider may act as a data processor, in which case a Data Processing Agreement (DPA) is required. The specific GDPR role depends on the product and contractual arrangement — confirm with legal counsel. Where a DPA applies, it should specify the technical measures the provider implements for tenant isolation, including what isolation exists for inference infrastructure, prompt caching, and stored conversation data. A DPA that contains only high-level commitments (“we implement appropriate security measures”) without isolation-specific technical specifications is a gap that procurement and legal teams should flag.

Cross-customer exposure as a reportable breach. If an LLM deployment that processes personal data experiences a session bleed, Redis-style cache bug, or system prompt bleed incident, the exposure likely triggers notification obligations to supervisory authorities and potentially to the affected data subjects, depending on the sensitivity of the exposed data and the number of individuals affected. This is not hypothetical: the Italian DPA specifically cited the March 2023 ChatGPT incident in its enforcement action.

Defense Architecture: What You Can Control

The defenses fall into two categories: things operators can control directly, and things that depend on provider behavior.

What Operators Can Control

Application-layer session isolation (fully controllable). Any conversation state, session history, or context that your application maintains should be strictly scoped to the individual user session:

• Instantiate memory and history objects per-request, not per-process or per-module. This is especially critical in serverless and multi-threaded environments where module-level objects persist across invocations.
• Key stored session state (in Redis, DynamoDB, or any external store) to a session identifier that is unique per user conversation, not shared across users.
• In any LLM wrapper framework (LangChain, LlamaIndex, etc.), read the documentation specifically for ConversationBufferMemory or equivalent components and confirm they are not being shared across request handlers.
• Write a test: send two sequential requests with distinguishable user content and assert that the second response shows no knowledge of the first. In concurrent environments, extend this to overlapping requests.

Deployment-unique prefix differentiation (best-effort). Including a short, stable, deployment-unique identifier at the start of every system prompt ensures your deployments don’t share KV-cache entries with other deployments within the same provider namespace — even if they happen to use similar system prompt text. This is a low-cost structural control that eliminates intra-namespace timing oracle exposure at the cost of losing cache sharing across your own differently-configured deployments.

Reduce prompt caching exposure for sensitive system prompts. If your system prompt contains sensitive business logic, proprietary instructions, or configuration that you do not want inferable via timing side-channels, limit its cacheability. Anthropic’s API requires explicit cache_control markers — simply omitting them disables caching for your content. For OpenAI’s automatic caching, there is no explicit provider-side off switch; available mitigations include: (a) prepend a randomized per-session token to the system prompt — since OpenAI caches leading token sequences, a unique, unpredictable prefix prevents your system prompt from establishing a stable cached prefix that other sessions could probe; or (b) vary the system prompt content enough across sessions that a consistent cacheable prefix cannot form. Note that adding a randomized suffix alone does not prevent prefix caching of the identical leading content, and the 1,024-token threshold applies to the total cached prefix, not just the system-prompt segment in isolation. These are imperfect controls; the most robust mitigation for highly sensitive prompts is to use dedicated, single-tenant compute where possible.

Prefer dedicated compute for sensitive workloads. Azure OpenAI’s Provisioned Throughput Units and AWS Bedrock’s Provisioned Throughput both provide reserved capacity that reduces cross-tenant inference contention compared to shared on-demand compute. Neither provider publicly documents these tiers as guarantees of fully isolated serving infrastructure — control planes, storage layers, and other components may still be shared. Self-hosting via vLLM on single-tenant dedicated infrastructure is the only configuration where the full isolation model is verifiable by the operator, at the cost of operational overhead.

Audit logs and monitoring. Instrument your LLM wrapper to log, per request, which session identifier was loaded and which was written — using opaque session IDs rather than raw conversation content, to avoid creating a secondary store of sensitive data. An unexpected cross-user correlation in session ID reads (e.g., the same session ID appearing under two different user contexts) is a detection signal for session bleed bugs. Providers with audit log capabilities should be configured to retain per-request metadata at the session level for the duration required by your data retention policy.

What Depends on Providers

Storage-layer isolation verification. You cannot independently verify that a provider’s KV-cache or session storage is isolated at the storage layer rather than only at the routing layer. You can ask — and for enterprise contracts, you should ask — for:

• Architectural documentation describing where in the stack cache namespace isolation is enforced
• Specifics on namespace partition granularity (per-account, per-project, per-region)
• Description of how isolation is maintained during cache migration, failover, or maintenance events
• Whether cache-hit telemetry is available in audit logs

The absence of a substantive answer is itself information: a provider that cannot answer these questions for an enterprise customer operating under GDPR has a documentation gap that is an audit finding in its own right.

Data Processing Agreement specifics. For GDPR-regulated workloads, negotiate a DPA that includes specific language on inference infrastructure isolation — not just data storage retention. The inference layer, including batch processing and prompt caching, should be covered.

Questions to Ask Your LLM Provider

The following questions are appropriate for vendor security assessments of LLM API providers. They target the specific failure modes discussed in this post and are designed to elicit substantive architectural responses rather than generic security acknowledgments.

1. At what layer is tenant isolation enforced for prompt caches and session data? Is isolation applied at the routing layer, the storage layer, or both? Does this include the KV-cache for inference, or only stored conversation data?

2. What is the namespace partition granularity for shared infrastructure? Is cache isolation per-account, per-project, or per-region? Is there any scenario in which two accounts can share a cache namespace?

3. What isolation controls apply during infrastructure events? How is cache namespace isolation maintained during failover, cache migration, or rolling deployments? What is the incident history of isolation failures?

4. Is prompt caching active by default, and can it be disabled per-deployment? For workloads containing sensitive system prompts, what are the controls for disabling or scoping prompt caching?

5. What dedicated compute options exist, and what isolation guarantees do they provide? Does a dedicated tier eliminate shared-batch inference risks? Is dedicated compute in scope for the same security certifications as shared compute?

6. What audit logging is available at the inference layer? Is there per-request logging of which session state was accessed? Is this available for review, and for how long is it retained?

7. What is your Data Processing Agreement’s coverage for the inference layer? Does the DPA explicitly cover inference infrastructure (batching, caching, GPU memory) in addition to data storage and retention?

8. How were impacted customers notified in past isolation incidents? What is your incident notification process for cross-customer data exposures, including scope assessment methodology?

Risk Summary

Scenario	Likelihood	Evidence Base
Shared session storage bug (Redis / cache layer)	Confirmed real pattern	March 2023 ChatGPT incident; class of SaaS caching bugs is well-documented
Session bleed via application-layer memory objects (LangChain etc.)	Confirmed real pattern	Documented in framework advisories; preventable by correct implementation
System prompt bleed at session boundary	Confirmed bug class	Documented in framework bug reports; severity depends on operator system prompt content
KV-cache timing oracle (intra-namespace, known-prefix test)	Plausible/theoretical	Architecture analogous to Flush+Reload; LLM-specific not peer-reviewed
Cross-account cache leakage via provider misconfiguration	Low likelihood, unverifiable	Not supported under documented isolation models; transient events unverifiable
Shared-batch inference cross-request leakage	Speculative	Requires implementation failure in attention masking; no production reports

What Architects Should Do

Multi-tenant LLM deployments introduce an isolation model that most enterprise architects have not previously had to evaluate — because most SaaS services do not put every customer’s conversation through a shared, high-performance inference and caching stack. The March 2023 incident showed that even sophisticated providers operating at scale can experience isolation failures that require no attacker; that the scope of such failures can be difficult to audit retrospectively; and that regulators treat cross-customer data exposures as reportable breaches.

The immediate checklist:

1. Audit session state scoping in every LLM wrapper your organization deploys. This is the highest-probability and most preventable failure mode.
2. Disable prompt caching for system prompts containing sensitive business configuration, or add deployment-unique prefix identifiers to limit intra-namespace oracle exposure.
3. Request storage-layer isolation documentation from your LLM provider. For GDPR-regulated workloads, ensure your DPA explicitly covers the inference layer.
4. Evaluate dedicated compute tiers for workloads processing regulated personal data. The isolation guarantee is meaningfully different from shared inference.
5. Test for session bleed as part of your standard QA for any LLM application deployed to serverless or shared infrastructure.

The gap between the documented isolation commitments major providers make and the architectural verification available to operators is real and unlikely to close through public disclosure alone. Enterprise procurement, audit requirements, and regulatory attention are the levers most likely to produce more specific commitments and, eventually, independently verifiable ones.

---

Related reading: Cross-Tenant Contamination in LLM APIs covers the technical mechanics of KV-cache timing attacks and session bleed in detail. System Prompt Extraction Attacks covers adversarial extraction of system prompts — the attacker-driven complement to infrastructure isolation failures. Securing the AI Inference Stack maps GPU memory and self-hosted inference threats for organizations running their own LLM infrastructure.

Sources: OpenAI incident report, March 20 2023 (conversation history disclosure). Italian Data Protection Authority (Garante per la Protezione dei Dati Personali), ChatGPT investigation, March–April 2023. GDPR Articles 32, 33 (Regulation (EU) 2016/679). Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention,” SOSP 2023. Yarom and Falkner, “FLUSH+RELOAD: A High Resolution, Low Noise, L3 Cache Side-Channel Attack,” USENIX Security 2014. OpenAI Prompt Caching documentation. Anthropic Prompt Caching API documentation. Azure OpenAI Service data privacy documentation. AWS Bedrock security documentation.