Cloudflare has explained some of the changes it has introduced in Workers AI to run large open models such as Kimi K2.6 from Moonshot AI and GLM 5.2 from Z.ai, more efficiently. The company claims to have reduced GPU memory consumption and inference costs through new optimization techniques, while keeping the models’ accuracy virtually intact.
Key points of Cloudflare’s optimizations in 20 seconds
- Workers AI incorporates new techniques to run large open models with less GPU memory.
- Cloudflare combines KV cache in FP8, compressed weights in INT4, and new memory protection mechanisms.
- The company states that nearly the same accuracy as the original models is maintained.
- The goal is to increase the number of concurrent requests and reduce inference costs.
The publication offers an unusual perspective on the work behind running large-scale language models, where the challenge is no longer just about having more powerful GPUs but making better use of every available gigabyte of memory.
The real bottleneck is memory
Cloudflare explains that models like Kimi K2.6 or GLM 5.2 stand out for their ability to handle very long contexts and Mixture of Experts (MoE) architectures, but these very features make deployment especially demanding in terms of memory.
In many cases, the main limit isn’t the model’s parameters themselves but the so-called KV Cache, a structure that stores attention keys and values generated during conversation to avoid recalculating the entire context for each new response.
With long conversations, this cache can occupy more memory than the model itself.
Halving the cache with FP8
One optimization involves storing the KV Cache using FP8 precision instead of BF16.
According to Cloudflare, this change roughly halves the space needed for cache storage. In the case of Kimi K2.6, the maximum context that can be held in memory increases from about 686,000 tokens to nearly 1.37 million tokens.
Although per-request performance slightly decreases due to the necessary conversions between numeric formats, the improvement becomes apparent as concurrency increases.
With BF16, the system exhausted memory with 32 simultaneous requests, while FP8 allows up to 64 concurrent requests, reaching approximately 2,192 tokens per second, about a 41% increase over the maximum achieved with BF16.
Cloudflare adds that their internal tests detect no appreciable differences in accuracy between the two formats on benchmarks like GSM8K, MMLU, or ARC.
GLM also reduces model size with INT4
The second technique directly affects the model weights.
For GLM 5.2, Cloudflare uses INT4 quantization, reducing the model size from approximately 705 GB to 421 GB, about 40% less.
This reduction frees up memory on each GPU for greater context capacity and also speeds up text generation.
Published data show performance improvements especially notable under low concurrency. In a single request, GLM’s generation speed increases from about 60 tokens per second with FP8 to approximately 92 tokens per second with INT4.
The company acknowledges that this technique isn’t as beneficial during the initial prefill phase, which continues to use FP8 since it’s mainly limited by compute capacity rather than memory bandwidth.
More efficiency demands more security mechanisms
Increasing the number of requests sharing GPU memory also raises the risk of errors during KV Cache reuse.
To minimize this, Cloudflare has developed an integrity verification system that checks each request to ensure it accesses only its designated memory pages.
If inconsistencies are detected, the request is canceled before potentially returning incorrect results.
According to published data, this mechanism adds less than 1% impact on performance and latency, so the company considers the cost acceptable to improve system reliability.
The current challenge is to serve more models at the same cost
Cloudflare states it will continue to expand these optimizations across its infrastructure.
Upcoming initiatives include expanding FP8 usage for KV Cache, new compression techniques like NVFP4 based on NVIDIA’s Blackwell architecture, and widespread implementation of memory integrity checks.
Beyond technical details, the article reflects a growing industry trend: the race is no longer only about building larger models but about running them more efficiently to serve more users without skyrocketing infrastructure costs.
Frequently Asked Questions
What models is Cloudflare optimizing?
The company predominantly describes improvements to Kimi K2.6, developed by Moonshot AI, and GLM 5.2 from Z.ai, both available in Workers AI.
What advantage does using FP8 for the KV Cache provide?
It roughly halves cache memory consumption, enabling more simultaneous requests without significantly affecting model accuracy.
What does INT4 quantization achieve?
It compresses model weights to use less GPU memory and speeds up text generation during decoding.
Is there a loss of accuracy when reducing model size?
According to Cloudflare’s published benchmarks, the differences compared to original versions are minimal and not significant in the tests used by the company.
