Google’s TPUs are starting to compete with NVIDIA’s Blackwell GPUs on one of the metrics that matters most to anyone running large AI models: how much it costs to generate every million tokens. The first independent benchmarks from SemiAnalysis on the TPUv7 Ironwood show cost advantages over B200 and B300 in numerous scenarios, while Google is preparing TorchTPU to remove one of the biggest barriers holding back adoption: native PyTorch support.
TPUs vs. NVIDIA in 20 seconds
- SemiAnalysis compared TPUv7 Ironwood against NVIDIA B200 and B300 on large-model inference.
- At 100 tokens/s per user, Ironwood costs 19% less than B200 and 34% less than B300.
- At certain points on the curve, it reaches up to 50% more performance per dollar.
- TorchTPU will let developers use PyTorch natively on TPUs.
The result doesn’t mean Google has built an accelerator that’s always faster than Blackwell. In fact, NVIDIA keeps significant advantages in certain scenarios, and TPUv7 lacks native FP4 compute. What matters here is a different metric: the relationship between performance, latency, and cost, which has become one of the major battlegrounds of AI infrastructure.
Ironwood enters the cost-per-token battle
SemiAnalysis has published on InferenceX its first independent inference benchmarks for TPUv7 Ironwood, comparing it at equal FP8 precision against NVIDIA’s B200 and B300.
Results shift quite a bit depending on the concurrency level and the latency required, so reducing the comparison to a single percentage would be misleading.
At a generation speed of 100 tokens per second per user, SemiAnalysis calculates a cost of roughly $0.181 per million tokens for Ironwood, versus $0.222 for B200 and $0.276 for B300. That’s an advantage of about 19% over B200 and 34% over B300.
| Scenario | TPUv7 Ironwood | NVIDIA B200 | NVIDIA B300 |
|---|---|---|---|
| 100 tokens/s/user | $0.181/M tokens | $0.222/M | $0.276/M |
| Difference vs. TPU | — | TPU ~19% cheaper | TPU ~34% cheaper |
| 20s median response time | ~$0.098/M | ~$0.106/M | ~$0.132/M |
| Precision compared | FP8 | FP8 | FP8 |
There are points on the curve where the gap widens even further. At 20 tokens per second per user, SemiAnalysis measured 9,364 tokens/s per chip on Ironwood, versus 8,903 on B200 and 8,925 on B300.
Combining that performance with its estimated hourly cost, the analysis calculates 50.4% more tokens per dollar than B200 and 96% more than B300 at that specific point. Precision matters here: those differences can’t be automatically extrapolated to any model, latency, or configuration.
There are also scenarios where Blackwell keeps the edge. SemiAnalysis acknowledges that B200 outperforms Ironwood on part of the curve, and that disaggregated GB200/GB300 NVL72 configurations remain competitive while TPU’s external software for this type of inference continues to mature.
The first consequence is that the debate can no longer be reduced to which accelerator offers more FLOPS or more tokens per second. For a provider serving billions of tokens, a few cents of difference per million tokens can end up having a significant effect on total infrastructure cost.
TorchTPU targets one of NVIDIA’s biggest advantages
Hardware is only part of the problem. NVIDIA has spent years building an edge around CUDA and a software environment widely used by developers, researchers, and inference providers.
Google needed to lower that barrier.
That’s where TorchTPU comes in, a new backend designed to make TPUs behave like a native device for PyTorch. Google explains that a developer will be able to work with ordinary PyTorch tensors on a tpu device, keeping familiar interfaces and using torch.compile when needed.
Underneath, the TPU stack’s own technologies are still there, including XLA, StableHLO, and specialized kernels. So “native PyTorch” doesn’t mean Google has removed its software architecture — it means the company wants to make it far less visible to the developer.
The change could have consequences beyond a simple convenience improvement.
vLLM and SGLang are two of the most widely used projects for serving language models in production. If both can treat TPUs as first-class PyTorch devices, Google reduces the work needed to adapt new open models to its hardware.
Google already confirmed in April that TorchTPU uses PyTorch’s PrivateUse1 interface and that it’s working on deep integrations with projects like vLLM. The company is also preparing a public repository and improvements for dynamic shapes, custom kernels, and distributed execution.
SemiAnalysis notes that TorchTPU remains in private beta and expects its open-source release around October. It also points to Qwen3.5-397B as one of the first large models used to fine-tune the new environment.
TPUv8 will add FP4 and raise the pressure on Blackwell
Ironwood still has one notable limitation compared with Blackwell GPUs: it has no native FP4 compute.
That matters because B200 and B300 can use FP4 to significantly boost inference performance, although dropping precision from FP8 can affect model quality. That’s why SemiAnalysis runs its main comparison at FP8 against FP8.
That will change with the eighth generation of TPUs.
Google has already unveiled two different variants: TPU 8t, aimed mainly at training, and TPU 8i, designed specifically for inference, post-training, and reasoning workloads. TPU 8i will pack 288GB of HBM, 384MB of integrated SRAM, and FP4 compute with an announced peak of 10.1 PFLOPS.
It will also debut Boardfly, a new interconnect topology that replaces the 3D torus design used by Ironwood, for this inference-focused model. Google says it reduces the maximum number of hops from 16 to seven in a comparable 1,024-chip configuration. The company attributes to TPU 8i an improvement of up to 80% in performance per dollar compared with Ironwood for certain low-latency inference scenarios.
TPU 8i is still listed as “coming soon,” so those figures come from Google itself and aren’t equivalent to the independent results SemiAnalysis has now published for Ironwood.
The other important piece is opening up the hardware beyond Google’s strictly internal infrastructure. SemiAnalysis argues that Ironwood marks a new stage in which TPUs can be rented and can also reach certain customers through direct purchase. If that distribution expands, and PyTorch stops being a barrier while Google gets vLLM and SGLang running competitively, NVIDIA could find itself facing a much more serious alternative than the traditional TPU locked inside Google Cloud.
That wouldn’t be the end of NVIDIA’s dominance. Blackwell still has very mature software, a huge installed base, and technical advantages across several workload types. But inference economics are changing the buying criteria: having the fastest accelerator is no longer enough; what increasingly matters is how many tokens it can produce per dollar spent and per watt consumed.
Frequently asked questions
Are Google’s TPUv7 chips cheaper than NVIDIA’s B200 and B300?
In the benchmarks published by SemiAnalysis, Ironwood achieves a lower cost per million tokens at numerous points on the curve analyzed. The advantage depends on concurrency, latency, and configuration, so there’s no single percentage that applies to every workload.
What is TorchTPU?
TorchTPU is the new environment built to let Google’s TPUs work as native PyTorch devices. It aims to shrink the gap between developing for GPUs and for TPUs, and to make integration with tools like vLLM easier.
Can TPUv7 use FP4?
Ironwood has no native FP4 compute. That’s why SemiAnalysis uses FP8 to make an equivalent comparison against B200 and B300. TPU 8i will include FP4 acceleration.
When will TPU 8i be available?
Google has announced TPU 8i and detailed its architecture, but as of September 9, 2026, it is still listed as coming soon for Google Cloud customers.
Source: newsletter.semianalysis. Image: Google.