NitroGen can play Witcher 3, Cyberpunk or even Rocket League
NVIDIA researchers and MineDojo collaborators have published NitroGen , a vision to action model trained on 40,000 hours of gameplay video spanning 1,000+ games . They are also releasing the dataset, evaluation suite, model weights, and code.
The key idea is turning public gameplay videos into labeled controller data. The team collected videos where creators show an on-screen gamepad overlay. They locate that overlay using template matching against a library of controller templates, using features like SIFT and XFeat, then crop the controller region for parsing.
Source: NitroGen
A segmentation model is trained to recover joystick positions and button states from the cropped overlay, and the overlay area is masked in the main frame so the policy cannot “cheat” by reading the overlay directly. The paper reports an average joystick correlation of 0.84 R² and button frame accuracy of 0.96 for the extraction pipeline.
Source: NitroGen
With the controller inputs reconstructed, NitroGen is trained with behavior cloning. It learns to map gameplay pixels to gamepad actions.
The team says the action model follows a GR00T-based policy. Here, GR00T refers to NVIDIA’s robotics work, where a “policy” is the model that outputs control signals. In NitroGen, that same policy style outputs joystick movement and button presses instead of robot motor commands. Their public results include a 500M-parameter model trained on the full dataset. For testing, they built a Gymnasium-style wrapper that runs game binaries through one API and records rollouts. On games held out from training, they report up to a 52% relative gain in task success after fine-tuning NitroGen weights compared to training from scratch.
Is it only for controller games. Mostly yes. NitroGen predicts gamepad actions , and the dataset is built around gamepad overlays. The public code path is also framed around running on Windows game processes, and it does not ship game binaries, so you need your own copies.
So keyboard warriors are mostly safe, unless someone adds a gamepad-to-keyboard bridge and starts training on that.
And yes, this feels like GeForce GTX G-Assist turning into a research project. The training data was pulled from public videos, not from my PC, so there is a chance this bot will still beat the game.
Source: Jim Fan (LinkedIn) , NitroGen
