This week, at the 2025 IEEE International Electron Devices Meeting (IEDM), imec, world-leading research center in advanced semiconductor technologies, presents the first thermal system-technology co-optimization (STCO) study of 3D HBM-on-GPU (high-bandwidth memory on graphical processing unit), a promising compute architecture for next-gen AI applications. By combining technology and system-level mitigation strategies, peak GPU temperatures could be reduced from 140.7°C to 70.8°C under realistic AI training workloads - on par with current 2.5D integration options. The result demonstrates the strength of combining cross-layer optimization (i.e., co-optimizing the knobs at all the different abstraction layers) with broad technological expertise, a combination that is unique to imec.

Integrating high bandwidth memory (HBM) stacks directly on top of graphical processing units (GPUs) offers an attractive approach for building next-gen compute architectures for data-intensive AI workloads. This 3D HBM-on-GPU promises a huge leap forward in compute density (with four GPUs per package), memory per GPU, and GPU-memory bandwidth compared to current 2.5D integration options where HBM stacks are placed around (one or two) GPUs on a silicon interposer. However, the aggressive 3D integration approach is prone to thermal issues because of higher local power density and vertical thermal resistance.