Cloudflare published an incident report on October 26 saying that 155 responses from its hosted GLM-5.3 model on Workers AI contained data from other customers' requests. The cause was a mix of Cloudflare's own configuration and a bug in SGLang, the open-source inference engine it runs. The report drew a long Hacker News thread and a steady stream of replies and quote posts on X. Several people praised the write-up. Others used it to ask how much they should trust shared AI hosting.
What Cloudflare disclosed
According to Cloudflare's report, a security researcher, Gadi Evron from Knostic, reported through its HackerOne program on September 8 that very long prompts sent to @cf/zai-org/glm-5.3 sometimes returned text from other users' sessions. Between August 27 and September 9, 155 responses were affected, all to prompts longer than 681,280 tokens. Cloudflare says it found no evidence of deliberate exploitation and that no other Workers AI model was affected.
The mechanism, in plain terms:
- Cloudflare offered GLM-5.3 with a 1,048,576-token context window, but the KV cache pool on its prefill GPUs only had room for 681,280 tokens. The KV cache is the GPU memory where the model keeps its working state for a prompt. Cloudflare says it knew about the mismatch before launch and expected the engine to return an error for prompts that did not fit.
- In SGLang's disaggregated mode, where one set of GPUs reads the prompt and another writes the answer, a rejected request was swapped for a one-token placeholder but still handed over as if it were live. The decode side then read memory pages that still held earlier requests' data.
- Pages are not zeroed between requests, a choice Cloudflare calls common and necessary for performance. Every affected reply came back with HTTP 200, not an error.
Cloudflare says it took the model offline within 40 minutes of declaring an incident on September 9 and restored it the same day with a larger pool and a patched engine. The fix is in SGLang v0.5.20, with follow-ups in v0.5.21 and v0.5.22. Cloudflare recommends that all SGLang operators update to v0.5.22 or later, and the SGLang team tells operators of disaggregated deployments to upgrade. Cloudflare's X post had about 15,700 views and the LMSYS post for SGLang about 15,100 as of 6:25 a.m. PT on October 27. The Hacker News thread stood at 149 points and 48 comments at the same time.
Praise for the write-up
A common reaction was credit for the disclosure itself. On Hacker News, kami23 wrote: "I like this RCA a lot more than most. I appreciate the ownership and clear explanation of why their testing did not catch this bug." They also said they were impressed that Cloudflare could pull the model quickly.
On X, @kaluops replied: "KV cache is just a cache, and cross-tenant leaks in shared caches are an old bug class. Nice to see a clear writeup instead of a vague status page note". @wuguanzheng2 praised Cloudflare for publishing the stale-page diagram and added: "The scariest line is that every leaked response came back as HTTP 200."
The HTTP 200 problem and the testing gap
A detail several people repeated was that the leak looked like success. @catcowbug put it in two lines: "it failed open with an http 200, and launch testing only checked status codes." and "a 200 just means somebody's answer came back."
Several replies focused on the decision to ship with a known mismatch. @BenjieMalinao wrote: "I'd want a test that proves the reject path actually rejects." On Hacker News, SillyUsername said: "So they knowingly deployed a configuration with an overflow risk, much like writing code with a buffer overflow exploit." segmondy asked: "Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?"
Not everyone thought the setup was reckless. hedgehog replied that, as they read it, the limit applied to a single prefill and that "people try pretty hard to avoid a cache miss on sessions that long and rarely do standalone requests that big so the situation triggering this condition is rare."
Practical fixes people suggested
Engineers used the thread to trade checks. @Justrunme suggested: "Cap max context at the gateway to what the KV pool can actually hold, so an over-length prompt gets a 4xx before it ever reaches prefill." @cdiamond pointed to a detail in the report, that leaked replies arrived too fast: "The tell was timing: the leaked replies came back faster than a prefill of that size should take." They floated a minimum time-to-first-token per prompt length as a check, with the caveat that prefix cache hits could blur it.
@eliasprunaire told self-hosters: "For those self hosting disaggregated SGLang, the fix is not backported: you need to upgrade to 0.5.21 (Mooncake, NIXL) or 0.5.22 (MORI)." That is their reading. SGLang's post has the official operator guidance.
There was also some engine rivalry. @JoeGuglielmucci replied to Cloudflare: "Bro use vLLM come on". An account named Kai @ SGLang Summit, @unseenmars_, answered: "I appreciate your honesty. Also would love to know what are things your hate about SGLang? We want to do better"
The title fight
Part of the Hacker News thread argued about the headline. jacquesm wrote: "This doesn't really have anything to do with GLM-5.3." They proposed "Cloudflare configuration error leaked data between user sessions" instead. robrenaud said: "The title is extremely misleading."
A commenter posting as kflansburg, which matches the name of one of the report's authors, Kevin Flansburg, replied: "At no point do we claim that it is a bug with the model." and "We name the model in the title because our customers need to know which of our endpoints was affected." Others sided with that view. srdjanr wrote: "And it's nice to know from the title that you could only be affected if you used this model."
Bigger worries about shared inference
The wider debate was about multi-tenant hosting. @aaaljaz wrote: "good on cloudflare for publishing this" and then "your prompt sits in the same gpu memory as everyone elses and the only proof of isolation is a blog post after it broke". On Hacker News, 0xbadcafebee wrote: "Enterprises with highly sensitive data should really be on non-shared hosts." nitishr added: "I'm just surprised more serverless providers haven't found (or publicly disclosed) similar issues, yet."
A few commenters said they had once received what looked like someone else's answer from other AI services. Those are personal accounts with no evidence attached, and Cloudflare's report covers only its own GLM-5.3 deployment. Some replies, including @mustafaakin and atrokhymchuk on Hacker News, joked about "Cloudbleed", a nickname for Cloudflare's February 2017 parser bug that leaked memory.
Why it matters
For people who send long documents or codebases to hosted models, Cloudflare's report says the KV cache on a GPU is a shared pool of pages and that the inference engine is responsible for tracking who owns each page. The incident only touched prompts over 681,280 tokens on one model, and Cloudflare says it found no deliberate exploitation. If you used @cf/zai-org/glm-5.3 on Workers AI between August 27 and September 9 with very long prompts, the report is the place to start. If you run SGLang yourself in disaggregated mode, the SGLang post lists the versions to upgrade to for each KV transfer backend.
Sources
- Cloudflare: Cross-request KV cache exposure in the GLM-5.3 Workers AI model, and our remediation with SGLang
- SGLang team (LMSYS): SGLang security update: improving request isolation
- Hacker News discussion
- Cloudflare on X
- LMSYS Org on X
- Cloudflare: Incident report on memory leak caused by Cloudflare parser bug (2017)
- Cloudflare: Quantifying the Impact of "Cloudbleed" (2017)
