NVIDIA has moved its Groq 3 LPX inference accelerator into full production, introducing a specialized system designed to accelerate token generation for AI agents and other latency-sensitive workloads. NVIDIA says the Groq 3 LPX, paired with its Vera Rubin NVL72 platform, delivered a median 3,431 output tokens per second on Artificial Analysis’ 100,000-token context benchmark using Google’s Gemma 4 31B model. The result was about four times faster than the fastest public endpoint measured in the same benchmark.
The announcement highlights a shift in AI infrastructure from simply maximizing computing throughput toward reducing the time users wait for generated responses. Agentic AI systems repeatedly generate tokens while reasoning, calling tools and interacting with other software, meaning delays can accumulate across many steps. NVIDIA designed Groq 3 LPX specifically for this decode-heavy portion of inference, complementing the broader Vera Rubin platform rather than replacing its GPUs.
NVIDIA Groq 3 LPX Enters Full Production
NVIDIA announced on August 24 that Groq 3 LPX had entered full production as an inference accelerator for the Vera Rubin platform. The system is designed to handle workloads where fast and predictable token generation is more important than simply maximizing aggregate computing capacity.
The accelerator is based on technology from Groq, the AI inference company NVIDIA acquired. NVIDIA is integrating the technology into its Vera Rubin architecture to create a heterogeneous system in which different processors handle different stages of AI workloads.
Groq 3 LPX At A Glance
| Metric | NVIDIA Groq 3 LPX |
|---|---|
| Primary purpose | AI inference |
| Production status | Full production |
| Platform | NVIDIA Vera Rubin |
| Rack-scale accelerator count | 256 Groq 3 LPU accelerators |
| Total SRAM | 128 GB |
| SRAM bandwidth | 40 PB/s |
| Scale-up bandwidth | 640 TB/s |
| FP8 inference compute | 315 PFLOPS |
| Key benchmark | Artificial Analysis |
| Benchmark model | Gemma 4 31B |
| Long-context test | 100,000 tokens |
| Median output speed | 3,431 tokens/second |
NVIDIA’s March technical description lists 256 interconnected Groq 3 LPUs in the rack-scale configuration, alongside 128GB of total SRAM and 315 PFLOPS of FP8 compute.
3,431 Tokens Per Second On A 100K-Token Context
The headline performance figure comes from an Artificial Analysis benchmark using Gemma 4 31B with a 100,000-token input context. NVIDIA reported a median output speed of 3,431 tokens per second.
The same benchmark found the fastest public endpoint at 870 tokens per second, meaning the Groq 3 LPX result was approximately four times faster in that particular test. NVIDIA said the testing showed no loss of precision or model quality.
Benchmark Comparison
| Test | NVIDIA Groq 3 LPX | Fastest Public Endpoint |
|---|---|---|
| Model | Gemma 4 31B | Gemma 4 31B |
| Context | 100,000 tokens | 100,000 tokens |
| Output speed | 3,431 tokens/sec | 870 tokens/sec |
| Relative performance | 1.0x | About 0.25x |
| Benchmark provider | Artificial Analysis | Artificial Analysis |
The result should be interpreted as a benchmark for a particular model, context length and infrastructure configuration. It does not mean every AI model or application will generate 3,400 tokens per second.
Why Token Generation Speed Matters For AI Agents
Traditional AI infrastructure often focuses on throughput: how many requests or tokens can be processed across a large system.
Agentic AI introduces another constraint. An AI agent may generate a response, call a tool, examine the result, reason about the next step and generate another response. This process can repeat dozens or hundreds of times.
When each stage requires sequential token generation, even small delays can accumulate.
AI Agent
↓
Reason
↓
Generate Tokens
↓
Call Tool
↓
Receive Result
↓
Reason Again
↓
Generate More Tokens
↓
Repeat
This makes decode latency increasingly important. NVIDIA says Groq 3 LPX is designed specifically to accelerate this token-generation phase, allowing agents to complete multi-step tasks more quickly.
5,000 Tokens Could Be Generated In About 1.5 Seconds
NVIDIA’s technical analysis provides an illustration of the practical difference.
At 3,431 tokens per second, generating 5,000 tokens would theoretically take roughly 1.5 seconds. At 100 tokens per second, the same output would take approximately 50 seconds.
Illustrative Generation-Time Comparison
| Output | At 3,431 tokens/sec | At 100 tokens/sec |
|---|---|---|
| 1,000 tokens | ~0.3 sec | 10 sec |
| 5,000 tokens | ~1.5 sec | 50 sec |
| 10,000 tokens | ~2.9 sec | 100 sec |
These are simple calculations based on the benchmark rates and illustrate generation time only. Actual application latency can also include prompt processing, networking, tool calls, scheduling and other overheads. NVIDIA itself notes that the comparison is illustrative rather than a complete measure of end-to-end agent performance.
Performance Remains High At 10,000-Token Context
The advantage was not limited to the 100,000-token benchmark. Artificial Analysis also tested Groq 3 LPX at a 10,000-token context length.
The system produced a median 3,382 output tokens per second, compared with 1,402 tokens per second for the fastest public endpoint in that test.
Long-Context Performance
| Context Length | Groq 3 LPX | Fastest Public Endpoint |
|---|---|---|
| 10K tokens | 3,382 tokens/sec | 1,402 tokens/sec |
| 100K tokens | 3,431 tokens/sec | 870 tokens/sec |
The relatively small difference between the 10K and 100K Groq 3 LPX results is significant because long context can create additional memory and communication demands for inference systems.
NVIDIA says the architecture’s SRAM and deterministic execution approach are intended to maintain predictable performance as context length increases.
Coding Workloads Show Even Higher Speeds
NVIDIA also evaluated Groq 3 LPX using the open-source SPEED-Bench benchmark for coding tasks.
On the Gemma 4 model, the system achieved a median output speed of 4,767 tokens per second. Its 80th-percentile result was 5,520 tokens per second, meaning 20% of the benchmark tasks generated responses above 5,500 tokens per second.
Coding Benchmark Results
SPEED-Bench
Median
4,767 tokens/sec
│
├──────────────
│
P80
5,520 tokens/sec
Coding agents are a particularly relevant application because they can read large codebases, generate substantial amounts of code and repeatedly interact with tools.
How Groq 3 LPX Works With Vera Rubin
Groq 3 LPX is not positioned as a standalone replacement for NVIDIA’s Rubin GPUs. Instead, NVIDIA describes Vera Rubin and LPX as a heterogeneous inference architecture.
Vera Rubin NVL72 can handle broad AI workloads, including training and inference, while Groq 3 LPX is optimized for fast token generation. The system can divide workloads between the two types of processors.
This approach is intended to use the right processor for the right stage of inference.
Heterogeneous AI Architecture
| Component | Primary Role |
|---|---|
| Vera Rubin NVL72 | General AI computing and inference |
| Rubin GPUs | Broad compute workloads |
| Groq 3 LPX | Low-latency token generation |
| Groq 3 LPUs | Decode acceleration |
| NVIDIA Dynamo | Workload orchestration |
| High-speed interconnect | Processor-to-processor communication |
NVIDIA says the architecture can support configurations including prefill-decode disaggregation, attention-FFN disaggregation and speculative decoding.
The Architecture Prioritizes Predictable Latency
One of the central features of Groq’s technology is deterministic execution.
Instead of allowing the hardware and software stack to make as many dynamic scheduling decisions as possible during runtime, Groq’s architecture relies heavily on compiler-scheduled execution and explicit data movement.
That approach is designed to reduce variation in latency and make token generation more predictable.
Key Architecture Specifications
| Specification | Groq 3 LPX Rack |
|---|---|
| LPU accelerators | 256 |
| Total SRAM | 128 GB |
| On-chip SRAM bandwidth | 40 PB/s |
| Scale-up bandwidth | 640 TB/s |
| FP8 compute | 315 PFLOPS |
| AI workload focus | Inference/decode |
| System design | Rack-scale |
NVIDIA says the system’s high-bandwidth SRAM and chip-to-chip communication are central to its ability to maintain fast token generation at scale.
NVIDIA Is Targeting The Agentic AI Market
The launch comes as the AI industry shifts toward agentic applications capable of performing multi-step tasks rather than simply answering individual prompts.
Agents may browse information, write and execute code, interact with enterprise systems, analyze documents or coordinate with other AI agents. Each additional step creates more opportunities for latency to accumulate.
For businesses deploying AI at scale, faster inference can therefore improve user experience and potentially increase the number of tasks that can be completed using the same infrastructure.
Traditional Chatbot
One Prompt → One Response
Agentic AI
Prompt
↓
Reason
↓
Tool
↓
Reason
↓
Tool
↓
Reason
↓
Final Response
More Steps = Greater Value From Low-Latency Inference
This is the market NVIDIA is targeting with the Vera Rubin and Groq 3 LPX combination.
Groq Becomes An Important Part Of NVIDIA’s Inference Strategy
The production launch also demonstrates how NVIDIA is incorporating Groq’s technology into its broader AI infrastructure strategy.
Groq announced that it will be among the first adopters of NVIDIA Groq 3 LPX and plans to deploy the technology with Vera Rubin NVL72 in its dedicated AI inference cloud. Groq is working with Dell Technologies on the deployment.
This gives NVIDIA another route into inference infrastructure, an area where competition is intensifying as AI companies look beyond training workloads.
NVIDIA Faces A Broader Inference Competition
The AI chip market is increasingly separating into specialized segments.
Training requires enormous computing capacity for building models, while inference requires serving those models efficiently to users. As inference volumes grow, companies have greater incentives to use architectures optimized for specific workloads.
Groq 3 LPX is therefore part of a broader trend toward specialized inference hardware.
AI Infrastructure Is Splitting Into More Specialized Layers
| Layer | Main Requirement |
|---|---|
| Model training | Maximum compute and scalability |
| Prompt processing | High memory bandwidth |
| Token generation | Low latency and predictable execution |
| Agent orchestration | Fast tool and model interaction |
| Networking | High-bandwidth, low-latency communication |
| Cloud serving | Cost-efficient high utilization |
NVIDIA’s strategy is to connect these layers within a single AI-factory architecture rather than relying on one processor design for every workload.
What 3,400 Tokens Per Second Does And Does Not Mean
The headline number is impressive, but it needs context.
It does not mean an AI chatbot will always respond at 3,400 tokens per second. Real-world applications include input processing, network communication, model scheduling, safety checks, database retrieval, tool calls and other operations.
It also does not mean Groq 3 LPX is universally four times faster than every competing AI accelerator. The fourfold comparison comes from a specific Artificial Analysis benchmark using Gemma 4 31B and a 100,000-token context.
The more important takeaway is that NVIDIA is optimizing specifically for the interactive decode phase of AI inference, where latency can become a major constraint for agentic workloads.
The Bigger Picture
NVIDIA’s Groq 3 LPX launch signals a major shift in AI infrastructure priorities. As AI systems evolve from simple question-and-answer applications into agents that reason, call tools and execute long chains of tasks, the speed at which models generate individual tokens becomes increasingly important.
The reported 3,431 tokens per second on a 100,000-token context benchmark shows the potential of specialized inference hardware. More importantly, the integration with Vera Rubin demonstrates NVIDIA’s strategy of combining general-purpose accelerated computing with specialized processors instead of expecting a single architecture to handle every AI workload.
Looking Ahead
The next test for Groq 3 LPX will be adoption outside NVIDIA’s own benchmark environment. Groq is among the first adopters, and other infrastructure providers are expected to evaluate the platform for large-scale agentic AI deployments. The industry’s focus will be on whether the benchmark performance translates into lower latency and better economics across real production workloads with multiple users and complex tool chains.
For NVIDIA, the broader opportunity is to make inference another major growth engine for its full-stack AI infrastructure business. If agentic AI adoption accelerates, specialized low-latency accelerators could become increasingly important alongside GPUs. Groq 3 LPX gives NVIDIA a way to address that requirement while keeping Vera Rubin at the center of the overall AI-factory architecture.
Get the day’s top stories in your inbox
One concise email. No spam, unsubscribe anytime.
