I visited AMD's office here in Taipei, Taiwan, during Computex 2023 to have a conversation with David McAfee, the company's Corporate VP and General Manager of the Client Channel Business. Our conversation was far-ranging on a number of topics that I'll write about separately, but I also had a chance to observe AMD's Ryzen XDNA AI engine at work in a laptop demo. McAfee also discussed the steps that AMD is taking to prepare the operating system and software ecosystem for the burgeoning AI use cases that will run locally on the PC, which we'll dive into further below.

After following the AMD-codenamed inspired hallway map, I found my way to the demo room to see AMD's latest tech in action.

AMD's demo laptop was an Asus Strix Scar 17 that comes powered by AMD's 4nm 'Phoenix' Ryzen 9 7940HS processor paired with Radeon 780M graphics. These 35-45W chips come with the Zen 4 architecture and RDNA 3 graphics. AMD also had an Asus ROG Zephyrus G14 running the same demo.

The XDNA AI engine, a dedicated accelerator that resides on-die with the CPU cores, is the object of our interest. The overall goal for the XDNA AI engine is to execute lower-intensity inference workloads, like audio and video processing, at lower power than you could achieve on a CPU or GPU while delivering faster response times than online services, thus boosting performance and saving battery power.

This engine can handle up to 4 concurrent AI streams, though it can be rapidly reconfigured to handle varying amounts of streams. AMD claims this engine, a progeny of its Xilinx IP, is faster than the neural engine present on Apple's M2 processors. It can handle up to 4 concurrent AI streams, though it can be rapidly reconfigured to handle varying amounts of streams. The engine is plumbed directly into the chips' memory subsystem, so it shares a pool of coherent memory with the CPU and integrated GPU. I've included AMD's slides from the launch in the above album -- they give an overview of how the engine works.

First, I popped open the task manager to see if the AI engine would enumerate itself as visible cores with utilization metrics, but the XDNA AI engine doesn't show up as a visible device. As you can see in the above album, I found the AI engine listed as the 'AMD IPU Device' in the device manager. However, we couldn't observe the load or other telemetry from the cores during the tests.

Here we can see the XDNA AI engine crunching away at a facial recognition workload. To the right of the screen, we can see a measurement of the latency for each step of the workload, and while the bars are impressively low and the workload ran quickly through a series of images as the AI engine crunched through the inference workload, we don't have any context on just how those figures compare to other types of solutions.

AMD's demo did have a button to test its onboard AI engine against the online Azure ONNX EP service, but the demo team told us they had encountered issues with the software, so it wasn't working. Naturally, we would expect the in-built Ryzen AI engine to have lower latency than the Azure service, and that is logically what AMD was trying to demonstrate here. Unfortunately, we were left without a substantiative comparison point for the benchmark results.

However, the benchmark does show that AI is alive and breathing on AMD's Ryzen 7040 processors, and the company is also well underway in bolstering the number of applications that can leverage its AI engine.

AMD announced last week at Microsoft's Build conference that it had created a new set of developer tools that leverage the open-source Vitis AI Execution Provider (EP), which is then upstreamed in ONNX runtime, to ease the work required to add software support for the XDNA AI engine. McAfee explained that the Vitis AI EP serves as a sort of bare metal translation layer that allows developers to run models without having to alter the base model. That simplifies integration.

AMD isn't providing performance metrics for its AI engine yet, but McAfee noted that its hard to quantify the advantages of an onboard AI engine with just one performance metric, like TOPS, as the other advantages of higher power efficiency and lower latency are all parts of the multi-faceted advantages of having an AI engine. AMD will share figures in the future, though.

McAfee reiterated the company's plans to continue to execute its XDNA AI roadmap, eventually adding the engine to other Ryzen processors in the future. However, the software ecosystem for AI on the PC is still in its early days, and AMD will continue to explore the tradeoffs versus the real-world advantages.

Much of the advantage of having an inbuilt AI engine resides in power efficiency, a must in power-constrained devices like laptops, but that might not be as meaningful in an unconstrained desktop PC that can use a more powerful dedicated GPU or CPU for inference workloads, but without none of the battery life concerns. I asked McAfee if those factors could impact AMD's decision on whether or not it would bring XDNA to desktop PCs, and he responded that it will doil down to whether or not the feature delivers enough value that it would make sense to dedicate valuable die area to the engine. AMD is still evaluating the impact, particularly as Ryzen 7040 works its way into the market.

For now, AMD isn't confirming any of its future plans, but McAfee said that while AMD is committed to the AI engine being a part of its future roadmaps, it might not come to all products. On that note, he said there could conceivably be other options for different types of chips, like desktop PCs, that leverage AMD's chiplet strategy. Other options, like add-in cards, are also conceivable solutions.

One thing is for sure -- we'll continue to see the scalable integrated XDNA AI engine make an appearance in many of AMD's products in the future. Hopefully, next time we'll see a better demo, too.