Sizing Infrastructure for Trillion-Parameter Models: A Practical Framework
Trillion-parameter models are no longer a research curiosity — they are increasingly the default for frontier AI. But sizing infrastructure for them is a different discipline from sizing for a 7B or 70B model. This framework walks through what actually matters.
The memory problem comes first
Before you think about FLOPS, think about whether the model fits in memory. A trillion-parameter model in FP8 needs roughly a terabyte just for the weights — before optimiser states, gradients, activations, and KV cache. Memory, not compute, is the first bottleneck you will hit.
Why GPU memory per card decides everything
- More memory per GPU means fewer GPUs to hold the model.
- Fewer GPUs means less communication overhead during training and inference.
- A 288 GB part like the GB300 can hold weights that would span three or four 80 GB cards.
Mixture-of-experts changes the math
Most trillion-parameter models today are mixture-of-experts (MoE) architectures, where only a subset of experts is active per token. This changes the requirement: total memory must hold the entire model, but compute per token stays far lower. The result is a workload that is profoundly memory-bound — which is exactly why the highest-memory GPUs win here.
Interconnect and storage are not afterthoughts
At trillion-parameter scale, communication can dominate. A single NVLink domain — where 72 GPUs share one high-bandwidth fabric — removes the cross-node hops that otherwise throttle training and inference. Storage must keep pace too, feeding data and checkpoints without idling the GPUs.
A practical sizing sequence
- Estimate total memory footprint (weights + optimiser + gradients + KV cache).
- Divide by usable memory per GPU to get the minimum GPU count.
- Add 20–30% headroom for batch size and future growth.
- Verify interconnect and storage can sustain the workload.
- Confirm availability — a perfect plan is useless if the GPUs are not deliverable.
The bottom line: For trillion-parameter models, memory capacity per GPU is the single most important variable. Rack-scale systems with 288 GB-class GPUs, unified by high-bandwidth interconnect, are what make these workloads economically deployable at all.