NVIDIA Open Sources Audio2Face for Real-Time Character Animation
NVIDIA does something good.
NVIDIA isn’t the company we know for open sourcing its work. That’s despite alternatives appearing over time (upscaling/framegen), but they finally made an exception without much competition, one should add.
NVIDIA has announced that its Audio2Face technology is now available under an MIT open-source license. The tool, part of the company’s generative AI portfolio, produces real-time facial animation and lip syncing from audio input. By analyzing phonemes, intonation, and emotional tone, Audio2Face generates data streams that can be applied to 3D characters, making them more expressive and lifelike.
The model is already supported through plugins for Autodesk Maya and Unreal Engine 5, with NVIDIA also releasing the SDK, training framework, and example datasets. Developers will be able to fine-tune existing models or adapt them for their own projects, extending the use of Audio2Face beyond current integrations. Available models include both regression and diffusion variants for lip syncing, as well as Audio2Emotion networks that learn emotional state from voice.
Game developers have already applied this technology in production. The Farm 51 used Audio2Face in Chernobylite 2: Exclusion Zone to accelerate animation workflows and deliver more natural characters. Other partners include Codemasters, GSC Game World, NetEase, Perfect World, Convai, Inworld AI, Reallusion, Streamlabs, and UneeQ. The system’s ability to bypass traditional performance capture makes it an attractive option for studios seeking faster production pipelines.
NVIDIA describes Audio2Face as part of its wider RTX Kit of AI-powered tools for developers, which also includes neural texture compression and global illumination SDKs. By releasing it under open license, the company is inviting developers, researchers, and students to adapt and expand the technology for a broader range of applications, from video games to digital humans in customer service.
| Audio2Face SDK and plugins | |
|---|---|
| Package | Use |
| Audio2Face SDK | Libraries and documentation for authoring and runtime facial animations on-device or in the cloud |
| Autodesk Maya plugin | Reference plugin (v2.0) with local execution that allows users to send audio inputs and receive facial animation for characters in Maya |
| Unreal Engine 5 plugin | UE5 plugin (v2.5) for UE 5.5 and 5.6 that allows users to send audio inputs and receive facial animation for characters in Unreal Engine 5 |
| Audio2Face Training Framework | Framework (v1.0) to create Audio2Face models with your data |
| Audio2Face models and training data | |
|---|---|
| Package | Use |
| Audio2Face Training Sample Data | Example data to get started with the training framework |
| Audio2Face Models | Regression (v2.2) and diffusion (v3.0) models to generate lip-sync |
| Audio2Emotion Models | Production (v2.2) and experimental (v3.0) models to infer emotional state from audio |
Source: NVIDIA