First, it was the CPU; after came the GPU; then all parallel workloads started running on GPUs, integrated or hyperscale. Similarly, the large language models we use today have adapted to the parallel nature of data processing on the GPU. The scale at which AI workloads operate is much larger than most of us can comprehend, and today we are exploring the new computing unit that was established as a byproduct of the need for much faster systems: the AI token factory. The AI token factory is a system or a combination of systems that work in tandem for a single purpose—the maximum throughput of AI tokens per second.

When large-scale AI systems are designed for training runs and later inference, the amount of compute resources is extensive. For example, xAI's Colossus 1 system has 100,000 NVIDIA H100 GPUs, while the next-generation Colossus 2 will utilize over 550,000 NVIDIA GB200 and GB300 GPUs. This is an insane amount of compute all for a single purpose: training and inferencing next-generation AI models. The "currency" of these AI models is tokens, and pumping out as many tokens as possible is what gives these data centers their core specifications. This includes the final token/s throughput that they are able to achieve on any hardware. First, it was inferencing on CPUs, then GPUs, then entire systems like NVIDIA NVL72, and now the latest unit of computing is an entire AI token factory. The AI token factory is the final form of AI systems, designed from the ground up for maximum efficiency in token creation, with a primary focus on throughput.