Intel open sources NPU Acceleration software for developers

Developers seeking to improve their software for next-gen Intel processors should consider looking into a new library released by the company. The software called NPU Acceleration Library is open source, and it is now available for download.

The library is written in Python, and it is designed to help developers boost their applications through Intel’s newest NPU (Neural Processing Unit) as available in Meteor Lake chips. Currently, no other Intel CPU features NPU, but by the end of this year Intel is also expected to launch Arrow Lake and Lunar Lake with built-in NPU units.

As per Tony Mongkolsmai , who serves as Intel’s Software Architect and Technical Evangelist, this library allows for the seamless local execution of Gemma-2b or TinyLlama compact language models. In theory, this enables developers to easily create local AI chat applications without running into significant challenges. Intel demonstrated this feature during the Meteor Lake launch.

Tony has presented this feature by running the AI model on MSI Prestige 16 AI laptop powered by Intel Core Ultra processor:

NPU Acceleration Library in action, Source: Tony Mongkolsmai

Intel is yet to provide documentation and extend the Library support for other popular AI formats and technologies:

• ✅ 8-bit quantization
  🔘 4-bit Quantization and GPTQ
  🔘 NPU-Native mixed precision inference
  ✅ Float16 support
  🔘 BFloat16 (Brain Floating Point Format)
  ✅ torch.compile support
  ✅ LLM MLP horizontal fusion implementation
  ✅ Static shape inference
  🔘 MHA NPU inference
  🔘 NPU/GPU hetero compute
  🔘 Paper

Worth adding that the software is not intended for end users, those should refer to the tools powered by DirectML and OpenVINO.

Source: Intel (Github)