The race to expand the context window of language models has been one of the main technical battles in the development of generative artificial intelligence. Companies like OpenAI, Google, Anthropic, and Meta are competing to offer models capable of processing larger amounts of text simultaneously. However, new research titled RULER: What’s the Real Context Size of Your Long-Context Language Models?, developed by researchers from NVIDIA and collaborators, questions the actual performance of these models when faced with tasks that require maintaining coherence and accuracy in genuinely extensive contexts.
What is the context window?
In the realm of language models (LLMs), the context window defines the maximum number of tokens—text fragments such as words, punctuation marks, or characters—that the model can process, analyze, and remember simultaneously. This means it determines how much text a model “has in mind” when generating its responses.
This parameter is key in advanced applications like code generation, document analysis, business assistants, or scientific research. The larger the window, the more information can be processed coherently without losing track.
RULER: a more demanding test
The RULER study (Real Use-case Long-context Evaluation and Ranking) aimed to measure not only the maximum amount of text that models claim to handle but also their actual capacity to maintain performance in long contexts. To achieve this, the team designed a synthetic and configurable test bed with tasks more challenging than simple information retrieval exercises.
In total, 17 open-source and commercial models were evaluated across 13 tasks grouped into four categories: retrieval, variable tracking, data aggregation, and multi-hop questions. The goal was to measure their effective performance across different context ranges, from 4,000 to 128,000 tokens.
Main results: many promises, few realities
The analysis reveals a striking conclusion: most models experience a significant drop in performance before reaching the context length they claim to support. Only a handful maintain performance over 85 percent when surpassing the 64,000-token barrier.
Below is a selection of the most notable results:
| Model | Declared Window | Effective Window | Average Performance (%) |
|---|---|---|---|
| Jamba-1.5-large | 256,000 tokens | Over 128,000 | 96.0 |
| Gemini 1.5 Pro (Google) | 1,000,000 tokens | Over 128,000 | 95.8 |
| Jamba-1.5-mini | 256,000 tokens | Over 128,000 | 93.9 |
| GPT-4 Turbo | 128,000 tokens | 64,000 tokens | 91.6 |
| Llama 3.1 (70B) | 128,000 tokens | 64,000 tokens | 89.6 |
| Mistral-Large-2411 | 128,000 tokens | 64,000 tokens | 86.0 |
| Qwen2 (72B) | 128,000 tokens | 32,000 tokens | 85.9 |
In contrast, some models that claim to handle contexts of up to a million tokens hardly exceed 16,000 in practice.
Marketing outpaces engineering
Researchers warn that the promotion of inflated figures regarding context size can mislead businesses and developers seeking reliable models for real-world use cases. Often, models are capable of “seeing” all the text, but not “reasoning” about it effectively beyond a certain threshold.
The RULER test introduces a paradigm shift: it’s not enough to remember a keyword buried in lengthy text; models must perform complex cognitive operations—such as tracking variables or synthesizing dispersed information—across the entire length of the context.
Implications for the industry
In business, legal, or scientific environments, where precision and consistency are essential, a drop in performance in long contexts can lead to costly mistakes or misinterpretations. This report underscores the need to evaluate models beyond their technical specifications and under conditions that simulate real-world use cases.
Moreover, it highlights the importance of independent and open benchmarks to assess model capabilities. Tools like RULER allow for objective comparisons of models from different providers, bringing transparency to an expanding market.
Conclusion
The race to expand the context window will continue to be a key factor in the development of LLMs. However, RULER’s results make it clear that the promise of handling millions of tokens is still far from being effectively realized. Meanwhile, technology decision-makers must choose their models based on actual performance and not marketing claims.
Memory is important, but what one does with it is even more so. For now, only a few models are demonstrating lasting comprehension when the text extends beyond the conventional.
Source: AI News