From Training to Inference

Abstract

Training built the models. Inference will serve them. The two workloads are superficially similar. Both run on the same GPU hardware. Both demand high-bandwidth memory and fast interconnect. Both scale to thousands of GPUs in production. They are very different things to design infrastructure around. This paper examines what changes when token throughput becomes the unit by which AI infrastructure is measured. Cost per token. Power per token. Latency per token. We walk through the cost economics across legacy GPU generations and current Vera Rubin and AMD Helios systems and sketch what the rack-scale platforms of 2028 require from the data center underneath them. The argument: the inference workload is qualitatively different from training and demands a different infrastructure design philosophy. Operators who continue to design for the training profile will find themselves serving inference customers on infrastructure that is overprovisioned in some dimensions and underprovisioned in others.

Contents

1. 011. The workload of the decade
2. 022. Tokens as the unit of measure
3. 033. What this changes about infrastructure design
4. 044. The compounding effect of efficiency
5. 055. Per-platform throughput data
6. 066. Cost build-up worksheet
7. 077. India market sizing detail
8. 088. References

Key findings

• Inference is the workload of the next decade. It is qualitatively different from training, not just quantitatively bigger.
• Token throughput, token cost, and token efficiency are the new units of measure. They make small infrastructure improvements economically significant at scale.
• Approximately 45 percent of the cost per token is in the operator hands, not the GPU vendor. This is the structural opportunity for data center operators in the inference era.
• The infrastructure design choices that matter most for inference workloads are rack power architecture, cooling architecture, and capacity elasticity. Two of these are the subjects of HN-RP-002 and HN-RP-006.
• The Indian inference market will need 8 to 25 GW of equivalent capacity by 2030. The operators who build with the right architecture between now and 2028 will capture the long-term margin.