Liquid Cooling at 600 kW per Rack

Abstract

Air cooling stops working as a primary thermal medium somewhere around 50 kilowatts per rack. From that crossover up to the 600 kilowatts that the Vera Rubin Ultra NVL576 will draw, every kilowatt has to be moved by liquid. This paper covers the mechanics of doing that at production scale. We work through the cold plate design, the rack-level coolant distribution, the row-level coolant distribution unit (CDU), the secondary coolant loop, and the heat exchanger that hands the heat off to the building heat-rejection plant, which rejects to air through dry coolers rather than evaporative towers. We use propylene glycol coolant in the primary loop, not water, so the cooling chain consumes effectively no freshwater. We explain why. We cover the failure modes specific to high-density direct-to-chip cooling (leaks, hot spots, pump failures, glycol degradation, biological growth) and the control strategy that keeps the loop within tolerance when the load swings by a factor of three between idle and peak inference. The paper is based on the cooling architecture HyperNext is deploying at Kakinada with our cooling-system partners.

Contents

1. 011. The crossover from air to liquid
2. 022. Glycol, not water
3. 033. The flow path
4. 044. The failure modes
5. 055. What we are still figuring out
6. 066. Glycol coolant chemistry deep dive
7. 077. Heat transfer calculations
8. 088. Pump and pipe sizing methodology
9. 099. References and standards

Key findings

• Air cooling stops working around 50 kW per rack. From there to the 600 kW per rack of Vera Rubin Ultra NVL576, everything is liquid.
• The HyperNext architecture uses single-phase propylene glycol coolant (25 percent PG, 75 percent demineralised water with corrosion inhibitor and biocide) in the primary IT loop. That charge is sealed and recirculated, not consumed. The secondary loop carries the heat to dry coolers that reject it to air, so there is no evaporative freshwater loss. The CDU heat exchanger is the boundary between the loops.
• Glycol gives us biological resistance, corrosion control, stable conductivity, and freeze protection in stagnant lines. The 8 percent pump power penalty is worth it.
• The five-stage flow path has approximately 53 degrees Celsius of total delta budget. Each stage contribution is engineered separately and the control loop holds case temperatures within 1 degree Celsius of setpoint through load transients.
• Failure modes that matter: leaks (handled by negative pressure design and isolation valves), hot spots (rack-level temperature monitoring), pump failures (N+1 redundancy), glycol degradation (chemistry monitoring and 4 to 5 year replacement cycle), biological growth (biocide dosing and side-stream filtration).
• Liquid cooling is not limited to the GPU racks. At HyperNext every piece of equipment in the hall is liquid cooled, down to the storage, firewalls and switches, and the data hall rejects its heat through dry coolers alone, with no evaporative cooling towers.