Cloudflare discloses GLM-5.3 KV cache bug that leaked other customers' data into 155 Workers AI responses
Cloudflare has published a detailed post-incident report explaining how GLM-5.3, the Z.ai coding model it serves on Workers AI, returned fragments of other customers' requests inside 155 responses between August 27 and September 9, 2026. The report, released on October 26 as part of a coordinated disclosure with the maintainers of the open-source SGLang inference engine and the researcher who found the flaw, traces the exposure to two problems that only became dangerous in combination: a deployment whose KV cache pool on the prefill GPUs was smaller than the model's advertised context window, and an SGLang bug in disaggregated prefill/decode mode that let a rejected request keep running instead of failing.
The affected model was @cf/zai-org/glm-5.3, which Cloudflare announced in its Workers AI changelog on August 28 with a 1M-token context window, reasoning support and function calling. Every one of the 155 affected responses came from a prompt longer than 681,280 tokens, the exact size of the prefill KV pool. Instead of receiving an error, the clients that sent those prompts received an HTTP 200 response that looked coherent and reported the long prompt-token count, but whose content was partly derived from cached state left behind by other users' prompts and partial generations on the same model shortly before.
Cloudflare says "a small number of customers" received such responses, without giving an exact count, and that after excluding its own testing and the researcher's, the affected traffic was consistent in timing and volume with ordinary long-context use. The company states that it found no evidence of deliberate exploitation. No other Workers AI model was affected, according to the report, because every other disaggregated deployment has a KV pool larger than its context length on both the prefill and the decode side.
··········
HOW A REJECTED PROMPT BECAME A CROSS-CUSTOMER LEAK.
Workers AI runs GLM-5.3 with SGLang in disaggregated mode, an architecture in which one group of GPUs handles prefill, the phase that processes the entire prompt and builds the key-value (KV) cache, while a separate group handles decode, the token-by-token generation of the answer. The KV cache built on the prefill side is transferred to the decode GPUs over RDMA using the Mooncake transfer engine, which copies data directly from one node's memory to another's. Cloudflare notes that KV cache pages are not zeroed between requests, a choice it describes as necessary for the best performance, which means the engine must track precisely which request owns each page.
The first problem was configuration. GLM-5.3 was deployed with a context window of 1,048,576 tokens, but the prefill KV pool could hold only 681,280 tokens. SGLang sizes that pool at startup from the GPU memory left after loading the model weights, and nothing in the engine forces the pool to match the context length that the service advertises. Cloudflare says it knew about the mismatch before launch. Its long-context benchmarks returned successful HTTP status codes and streamed bodies with no error frames, so the team assumed the engine was handling the difference gracefully. The report calls this a critical gap in its testing: the team validated status codes and performance at long contexts, not whether the output was correct or coherent.
The second problem was in SGLang itself. When a prompt exceeded the pool, the engine rejected it through Req.set_finish_with_abort, which records an error for the client and replaces the prompt with a one-token placeholder. On a single node this path works as intended: the placeholder runs and the client gets an error. In disaggregated mode, however, the prefill and decode admission queues checked whether a request was finished rather than whether it had been rejected, so the aborted request still looked live to the rest of the pipeline.
Cloudflare's report walks through the resulting sequence with an illustrative example. The router told the decode side the original prompt length, and decode reserved eight KV pages for it. Prefill admitted the rejected request, ran a forward pass on the one-token placeholder, transferred a single page of KV to decode and signalled completion. Decode accepted the handoff and attended over all eight reserved pages, although only the first page had been written for this request. The other seven still held data from earlier requests, and the model generated its answer from that mixture. The client saw a normal-looking response, with HTTP 200 instead of HTTP 400.
The report adds that every request above the threshold went through this path rather than failing with an error. The underlying SGLang behaviour had already been reported upstream as issue #30233 on July 6, 2026; Cloudflare says it found that issue while investigating its own incident, and that it had been flagging related classes of problems to the SGLang project since version 0.5.10.
··········
DETECTION, RESPONSE AND THE INCIDENT TIMELINE.
The flaw was found from outside the company. It was reported on September 8 at 23:11 UTC through Cloudflare's HackerOne program by Gadi Evron of Knostic, who observed that very long prompts sometimes returned text that belonged to other users' sessions. Cloudflare began verifying the report the following morning, declared a security incident at 13:22 UTC on September 9, and its Product Security team reproduced the leak with two independent accounts within 25 minutes.
........
Date and time (UTC) | Event |
Aug 27, 2026 | GLM-5.3 deployed with a prefill KV pool smaller than its context window |
Sep 8, 23:11 | Report received via HackerOne from Gadi Evron (Knostic) |
Sep 9, 13:22 | Security incident declared; leak reproduced with two accounts |
Sep 9, 14:00 | Model set to private; requests return HTTP 400 |
Sep 9, 14:35 | Root cause identified: prefill pool smaller than context length |
Sep 9, 14:52 | All prefill nodes moved to the corrected configuration |
Sep 9, 19:04 | Deployment rebuilt with corrected configuration and engine patch |
Sep 9, 19:40 | GLM-5.3 restored to public access |
Sep 16 to Oct 12 | Upstream SGLang merges admission fix, checksums and other hardening |
Oct 2 and Oct 19 | SGLang v0.5.21 and v0.5.22 released |
Oct 26 | Coordinated public disclosure |
........
The model was taken offline about 40 minutes after the incident was declared, and Cloudflare confirmed the root cause by running a canary prefill node with a larger pool that was not susceptible. Before restoring access, it validated both the configuration change and its engine patch against the researcher's reproduction script and against its own canary-marker tests. Between September 11 and 19, additional hardening was rolled out to all of Cloudflare's production SGLang deployments, not only the one serving GLM-5.3.
The report does not say how or when the affected customers were individually informed, and it does not characterize the leaked content beyond describing it as other users' prompts and partial generations held in stale KV pages. It also does not state whether personal data was involved. Customers who sent long-context requests to GLM-5.3 during the window therefore have no public list of which responses or which source requests were involved.
··········
WHAT CLOUDFLARE AND SGLANG CHANGED.
The immediate fix had two parts. Cloudflare aligned the GLM-5.3 configuration with the one it uses for GLM-5.2, which freed GPU memory and raised the prefill KV pool to about 2.04 million tokens, roughly twice the 1,048,576-token context length. It also made the rejection path safe in the engine: a request carrying a pending rejection is now retired at the disaggregation admission queue, with its original error, before any bootstrap, page allocation or KV transfer takes place. Cloudflare proposed this change upstream as SGLang PR #38704, and the report also describes a check on the decode side that verifies the number of transferred tokens against the number reserved.
Upstream, two changes address the specific failure. PR #36988, shipped in SGLang v0.5.20, retires aborted prefill requests before their result reaches decode; it was merged to the main branch on September 6 for a related failure path, and Cloudflare says it stops the exposure seen in this incident. PR #38935, in v0.5.21, rejects such requests at admission, following the approach of Cloudflare's own patch.
........
Area | Change | Status |
Admission | Rejected requests retired before decode (#36988, #38935) | Released in v0.5.20 and v0.5.21 |
KV lifetime | Deferred KV release on both roles (#37454, #41023) | Upstream, on by default |
Prefill pages | Upstream hardening | |
Router | Both halves cancelled on disconnect or prefill failure (#39461, #41612) | Upstream hardening |
Transfers | Bounded Mooncake batches (#32758); failed sends no longer reported as drained (#43719) | Upstream hardening |
Integrity | Per-transfer KV checksum, --disaggregation-enable-kv-checksum (#39500) | Merged, awaiting release |
Page reuse | Zeroing KV pages on release (draft #46435) | Pending |
........
Cloudflare also changed its own load balancer so that a client disconnect no longer aborts a request in the middle of a KV transfer; the connection is now held until prefill finishes and the first token is produced. Over the longer term, the company says it is building page-level integrity tags that check every KV read and is applying formal verification to its RDMA-based KV transfer protocol. It also commits to checking KV pool size against context length directly from the running engine, and to adding that check to every change to a disaggregated deployment.
··········
WHAT IT MEANS FOR WORKERS AI CUSTOMERS AND SGLANG OPERATORS.
For Workers AI users, the practical situation is that the exposure is closed. GLM-5.3 was restored on September 9 with the corrected configuration, and Cloudflare says no other model on the platform was affected. When Cloudflare launched GLM-5.3, it listed the model at $1.40 per million input tokens, $0.26 per million cached input tokens and $4.40 per million output tokens, available on the Workers Paid plan or with prepaid AI Gateway credits through the Workers AI binding, the REST API, an OpenAI-compatible endpoint and AI Gateway. The incident report gives no customer-specific action list and invites further security reports through Cloudflare's HackerOne program. Only prompts longer than about 681,000 tokens sent to GLM-5.3 between August 27 and September 9 could have produced an affected response. The other side of the exposure is wider: since the stale pages held whatever other users had recently sent to the same model, any customer using GLM-5.3 during that window could in principle have had fragments of their own requests surface in someone else's response, and the report does not identify which source requests were involved.
For teams that run SGLang themselves in a disaggregated prefill/decode setup, the report is more prescriptive. Cloudflare recommends measuring the KV pool size on every prefill and decode node and comparing it with the configured context length after every relevant change; upgrading to SGLang v0.5.22 or later, which includes both #36988 and #38935; running with deferred KV release on both roles and with bounded Mooncake transfer batches; and enabling corruption detection where supported, through --disaggregation-enable-kv-checksum and --kv-canary. The report cautions that these detection options do not yet cover every architecture, naming GLM-5.3, GLM-5.3 Flash and DeepSeek V4 as models they do not currently protect. It also advises configuring load balancers to avoid unnecessary aborts before the first token.
The broader lesson for anyone serving long-context models is the one Cloudflare draws about its own process. An inference stack can return successful status codes and plausible text while silently computing on the wrong memory, so long-context validation has to check output correctness and memory ownership, not only latency and HTTP results. That applies with particular force to the growing number of open-weight models advertised with 1M-token windows, such as MiniMax-M3.1-Flash, where the gap between the context a provider advertises and the cache it actually allocates can be large. The incident also adds to a run of supply-chain and infrastructure problems in the AI tooling layer, after the LiteLLM malicious release traced to a leaked master key.
··········
WHAT IS ESTABLISHED AND WHAT REMAINS UNCLEAR.
The established facts come from Cloudflare's own account: 155 responses from GLM-5.3 on Workers AI contained data derived from other customers' requests; all were triggered by prompts longer than 681,280 tokens; the exposure ran from the August 27 deployment until September 9; the cause was the combination of an undersized prefill KV pool and an SGLang admission bug in disaggregated mode; and the model was taken offline within about 40 minutes of the incident being declared and restored the same day with a corrected configuration and an engine patch. The upstream fixes are public in SGLang v0.5.20 through v0.5.22, with further hardening still pending.
Several points remain open. Cloudflare has not given the number of affected customers, has not described how they were notified, and has not said whether the exposed fragments included personal or otherwise sensitive data. The statement that there is no evidence of deliberate exploitation rests on Cloudflare's analysis of traffic patterns. And because the SGLang issue had been reported upstream in early July, other operators running similar disaggregated deployments with undersized pools before v0.5.20 may have been exposed to the same failure; the report does not assess that wider population, and the checksum and canary safeguards that would detect such corruption do not yet cover every model architecture.
··········
FOLLOW US FOR MORE.
·····
DATA STUDIOS
·····
datastudios.org
