The big race in AI is no longer just about training larger models. Increasingly, the real bottleneck appears in inference: how long it takes a system to start responding, the latency accumulated when multiple agents interact with each other, and the cost of maintaining that speed at scale. In this context, NVIDIA has introduced Groq 3 LPX, a new rack-scale accelerator for the Vera Rubin platform designed specifically for low-latency inference workloads and very long contexts—two increasingly important ingredients in what’s called agential AI.
They position it as a complementary piece to Vera Rubin NVL72, not a replacement for their general-purpose GPUs. The idea is to distribute the workload: Rubin GPUs will remain the flexible engine for training, prefill, attention, and high-throughput serving, while LPX will handle the most latency-sensitive part of decoding, where every millisecond begins to matter in code assistants, copilots, tool-using agents, or multi-agent systems.
On paper, the figures are impressive. NVIDIA speaks of a system based on 256 Groq 3 LPUs, with 315 PFLOPS of FP8 inference, 128 GB of total SRAM, 40 PB/s on-chip SRAM bandwidth, and 640 TB/s of scale-up bandwidth per rack. They also describe it as the “seventh chip” in the Vera Rubin platform, emphasizing that this is not just a simple variant of a GPU but a new class of processor within their AI factory architecture.
An architecture designed for interactive AI
The most interesting part of the announcement isn’t just the raw compute volume but the kind of use cases NVIDIA aims to target. Their thesis is that inference is bifurcating into two worlds. On one side are throughput-oriented workloads, such as embeddings, moderation, batch pipelines, or large-scale services where maximizing tokens per GPU or per watt is paramount. On the other side are scenarios where latency rules: conversational assistants, autonomous agents, voice, translation, interactive reasoning, or systems that chain inference, retrieval, tools, and new model calls.
In these cases, optimizing the entire pipeline for a single operating regime involves trade-offs. Hardware tuned for high throughput with large batches isn’t always the best for generating tokens quickly and stably with small batches. Conversely, hardware optimized for instant response isn’t necessarily the most efficient during the most intensive pipeline phases. NVIDIA proposes solving this dilemma with a heterogeneous architecture: Rubin for the heavy lifting and LPX for the latency-sensitive decoding, especially in components like FFN and MoE.
This division also rests on a very different design from a traditional GPU. The core of LPX, the Groq 3 LPU, prioritizes deterministic execution, SRAM-first memory, explicit data movement, and close coordination between compute and communication under compiler control. NVIDIA details that each LPU incorporates 500 MB of on-chip SRAM, 150 TB/s internal bandwidth, and high-speed chip-to-chip links to reduce jitter and make token timing more predictable. In other words, the product isn’t marketed for its flexibility but for its ability to maintain stable response times when user experience depends on it.
More useful tokens, not just more tokens
NVIDIA connects this approach with a broader shift in the AI economy. The company argues that as models approach speeds of 1,000 tokens per second per user, interactions cease to resemble turn-based chat and instead become more like continuous collaboration, with agents reasoning, simulating, consulting tools, and reacting in real-time. This is the narrative that justifies Groq 3 LPX: creating a new inference category where it’s not enough to serve more requests, but to serve them with greater immediacy and less variability.
To operationalize this heterogeneity, NVIDIA supports deployment through Dynamo, their orchestration software for distributed inference. They present it as the layer that classifies requests, routes prefill to GPUs, coordinates activation exchanges between Rubin and LPX during decoding, and helps keep queue latency in check under variable traffic conditions. They also propose LPX as a very suitable component for speculative decoding, acting as a draft engine while Rubin GPUs verify and accept tokens with the primary model.
However, the most aggressive figures in the announcement should be approached with caution. NVIDIA claims that the combination of Vera Rubin NVL72 + LPX can deliver up to 35 times more inference throughput per megawatt and up to 10 times more revenue opportunity for billion-parameter models compared to previous systems—especially in highly interactive premium services. These are manufacturer metrics, useful for understanding positioning but still requiring practical validation once this architecture reaches real deployments.
What Groq 3 LPX clearly indicates is NVIDIA’s strategic direction. The company no longer wants the next-gen AI infrastructure to be measured solely by how many tokens a rack can produce, but by how it combines throughput, latency, and economic value per megawatt. In this scenario, agential AI is no longer just about models but also depends on a new layer of specialized hardware for interactive inference.
Frequently Asked Questions
What exactly is NVIDIA Groq 3 LPX?
It’s a new rack-scale inference accelerator introduced by NVIDIA for their Vera Rubin platform, designed for low-latency workloads, long contexts, and agential systems.
What role will it play alongside Vera Rubin NVL72?
NVIDIA positions it as a complement. Rubin will continue handling training, prefill, decode attention, and general-serving, while LPX accelerates the most latency-sensitive decode components like FFN and MoE.
What specifications has NVIDIA announced for LPX?
The company mentions 256 LPUs per rack, 315 PFLOPS FP8, 128 GB of total SRAM, 40 PB/s of on-chip SRAM bandwidth, and 640 TB/s of scale-up bandwidth.
Why does this launch matter for agential AI?
Because agential AI demands faster responses, more stable latency, and better behavior in inference loops, tools, and reasoning. NVIDIA aims to position LPX precisely at that market point.