Databricks published one of the most interesting benchmarks of the moment for coding agents, not because it uses a public set of exercises, but because it relies on its own codebase. The company has evaluated models and harnesses of agents on real tasks extracted from internal pull requests, involving changes over a codebase with millions of lines and multiple languages, including Python, Go, TypeScript, Scala, Rust, and Java.
The most striking result is that GLM 5.2, an open-source model from Z.ai, appears in the higher capability group and is statistically tied with Claude Opus 4.8 in quality within Databricks’ internal test. The main difference lies in cost: GLM 5.2 comes in at $1.28 per task, compared to $1.94 per task for Opus 4.8 in the evaluated configuration.
The takeaway isn’t that one model automatically replaces another across all scenarios. A more meaningful insight is that, in real engineering tasks, the efficiency frontier is no longer owned by a single provider. According to Databricks, the Pareto front of their tests includes models from OpenAI, Anthropic, and open source, pointing to a future where companies won’t choose “the best model” fixed, but different routes based on cost, complexity, and task type.
The benchmark companies should build themselves
Databricks explains that public benchmarks like SWE-bench or TerminalBench are useful but don’t resolve internal questions. The reason is clear: public tasks can become part of training data over time and don’t always reflect the reality of a specific enterprise codebase. That’s why the company built its evaluation based on real, recent, and reviewed PRs, with quality tests and changes scoped to specific modules.
This approach makes a difference. A public benchmark measures general capability. An internal benchmark assesses whether an agent helps solve the company’s real problems, with its own patterns, frameworks, conventions, build systems, and historical decisions.
Databricks filtered out changes generated by bots, service accounts, fully AI-generated commits, and auto-generated code. The selected PRs were then turned into tasks: summarizing intent, removing clues about the solution, separating tests, and evaluating whether the agent could reproduce a correct implementation. The company also avoided using an LLM judge to decide if the response was valid, believing that such an approach could favor convincing answers even if they weren’t correct.
Another key technical detail was controlling internal leaks. In initial experiments, Databricks found that some solutions seemed “too good,” because the agent could retrieve the correct implementation from Git history. To fix this, they sealed the history during each task execution so the agent couldn’t access previous commits to find the answer.
This makes the experiment an insightful signal for any engineering team: many companies already have their own benchmark without realizing it. It’s in their closed PRs, tests, resolved issues, and human-reviewed changes.
Model matters, but the harness matters almost as much
The most technical aspect of the study is the separation between the model and the harness. In a coding agent, the model reasons, proposes changes, and writes code. The harness is the layer that enables operation: searching files, executing commands, reading outputs, managing context, deciding what information to send at each turn, and keeping the task under control.
Industry tends to compare models as if they function in isolation. Databricks shows that this is insufficient. In their tests, running the same model with the same reasoning effort across two different harnesses caused the cost per task to vary more than twofold in some cases, even while quality remained consistent. The main culprit was the amount of context each harness sent back to the model at each turn.
Pi, the standout internal harness in the analysis, sent about three times less context per turn. According to Databricks, it managed the workflow better, maintained a tighter window, and completed tasks in fewer executions.
This shifts the cost conversation. It’s not enough to look at price per million tokens. If an agent reads too much, re-sends unnecessary context, iterates excessively, or can’t maintain focus, it could end up more expensive—even if the model’s rate is lower.
| Element | Decision in a coding agent |
|---|---|
| Model | Reasoning, code generation, instructions understanding |
| Harness | Sent context, available tools, commands, file reading |
| Tests | Actual verification of the solution |
| Routing | Choosing the model based on difficulty and cost |
| Internal pipeline | Security, traceability, permissions, repeatability |
Token cost can be misleading
Databricks provides an example that should appear in any enterprise AI adoption analysis. Sonnet 5 was about 1.7 times cheaper per token than Opus 4.8, but in their tasks, it ended up costing more per task: $2.09 vs. $1.94, with a lower completion rate: 81% vs. 87%. The reason was that Sonnet 5 used around 1.9 times more tokens to reach the result.
Therefore, the key metric isn’t the unit cost per token but the total cost to successfully solve a task. In coding agents, that includes input tokens, output tokens, reasoning, tool calls, retries, runtime, context reads, and failures.
This nuance will become increasingly important for companies moving from testing pilots to operating agents at scale. An apparent cost saving per token can vanish if the model needs many more retries or if the harness feeds it excessive irrelevant information.
GLM 5.2 and the new role of open source
GLM 5.2 does not appear in this benchmark as a cheap curiosity but as a competitive option for high-level tasks within a real environment. Z.ai presents GLM 5.2 as an open-source model with an MIT license geared toward long-horizon tasks, enhancing its appeal for companies wanting to evaluate open models alongside closed ones.
This doesn’t mean that all teams can just download and run it without operational costs. Large open models require infrastructure, GPUs, memory, inference engineering, security, observability, and maintenance. But it does shift the strategic balance: a company can incorporate open models into its routing system, use them for certain tasks, retain closed alternatives for others, and measure all against its own results.
The enterprise takeaway is clear: the idea of a single-provider solution diminishes as quality, cost, and efficiency vary depending on task, model, and harness. A mature AI development architecture should enable testing, routing, and swapping models seamlessly.
Databricks signals this direction by expressing interest in using smart routing capabilities in Unity AI Gateway and Omnigent to assist developers in selecting more suitable agents while maintaining efficiency and control.
Lessons for engineering teams
The Databricks study offers several practical lessons. First, public benchmarks are just a starting point. They help explore the market, but the real decision should be based on internal tasks, custom tests, and specific metrics.
Second, the entire agent system is the evaluation unit. Model, harness, tools, permissions, context, tests, and review flow form a system. Comparing models alone can lead to poor decisions.
Third, routing will become a core component. Not all tasks require the most expensive model. Databricks observed that about a quarter of the evaluated tasks were low complexity and around 60% medium complexity, yet costly models tended to be used by default.
Fourth, open source cannot be ignored anymore. If an open model can achieve high quality at a lower task cost, it should at least be evaluated. It won’t always win, but ignoring it will be harder to justify over time.
AI-assisted engineering is entering a more mature phase. It’s no longer enough to ask which model writes better code in a demo. You need to measure which agent solves real tasks better, at what cost, with what supervision, what risks, and how it fits within your development architecture.
The future of code agents won’t be dominated solely by the most powerful model. It will be shaped by those who can effectively combine models, harnesses, tests, and routing based on proprietary data. Databricks has just demonstrated a serious way to get started.
Frequently Asked Questions
What exactly did Databricks test?
They evaluated coding agents on real tasks derived from internal pull requests, within a codebase containing millions of lines, with their own tests to verify functionality.
Why is GLM 5.2 important in this benchmark?
Because it ranks high in capability, is statistically tied with Opus 4.8 in quality, but costs less per task according to Databricks’ evaluation.
What is a harness in coding agents?
It’s the layer that manages tools, context, file searches, terminal commands, test outputs, and the interaction between the model and the repository.
Why isn’t token price alone enough to compare models?
Because a cheaper per-token model can end up more expensive per task if it consumes more tokens overall or requires more iterations. The key metric is the total cost of completing a correct task.
Should companies build their own benchmarks?
Yes, especially if they operate coding agents at scale. Their historical PRs, tests, and real changes provide a more accurate performance measure over tasks that reflect their engineering reality, not generic problems.
