GB300 NVL72 vs H100: Why Rack-Scale Architecture Is the Future of AI Compute
For years, the NVIDIA H100 defined what a top-tier AI accelerator looked like. But the frontier has moved. The GB300 NVL72 represents a different kind of machine entirely — not a faster card, but a rack that behaves as one. Here is what that shift actually means.
From single cards to a rack as one GPU
The H100 is a single-GPU part: 80 GB of HBM3, connected to its neighbours through NVLink and a switch. It is excellent, but it is fundamentally a server part. The GB300 NVL72 is a rack-scale system — 72 Blackwell Ultra GPUs and 36 Grace CPUs linked into a single NVLink domain of 130 TB/s. From the software's point of view, the whole rack behaves like one enormous accelerator.
The numbers that matter
| Specification | NVIDIA H100 SXM | GB300 NVL72 (per GPU) |
|---|---|---|
| GPU memory | 80 GB HBM3 | 288 GB HBM3e |
| Memory bandwidth | 3.35 TB/s | 8 TB/s |
| Interconnect | NVLink 4 | NVLink 5 (130 TB/s rack domain) |
| FP4 compute | — | 15,000 TFLOPS |
| Highlight | Baseline | Up to 10× inference throughput vs Hopper |
Why memory capacity is the real story
Modern frontier models — especially mixture-of-experts architectures — are memory-hungry before they are compute-hungry. A 288 GB GPU can hold model weights that would otherwise need to be sharded across three or four H100s. Fewer GPUs means fewer interconnect hops, lower communication overhead, and dramatically better throughput per dollar on memory-bound workloads.
Built for the age of AI reasoning
The GB300 NVL72 is engineered specifically for test-time scaling inference — the reasoning-heavy workloads behind today's most capable models. NVIDIA reports up to 10× higher tokens-per-second per user and 5× higher throughput per megawatt compared to Hopper, which compounds into a claimed 50× improvement in AI-factory output. That is the kind of generational leap that changes what is possible to deploy economically.
The bottom line: Choose H100-class instances for standard training, fine-tuning, and enterprise inference where cost-per-GPU-hour is the priority. Choose GB300 NVL72 when you are running trillion-parameter or MoE models, need long-context reasoning, or want to serve frontier models at production scale.