NVIDIA’s DLSS 5 follow-up answers raise questions about earlier messaging
© NVIDIA NVIDIA’s Jacob Freeman responded to a query about DLSS5 technology posed by YouTuber Daniel Owen . In terms of how the technology actually functions, it answers one of the most important questions, and it’s not the answer gamers expected.
NVIDIA has now confirmed that DLSS 5 takes a 2D rendered frame plus motion vectors as input . That clarification came in follow-up answers to questions about how the new technology actually works. The answer is direct, and it does not fully match the earlier impression created by NVIDIA’s own marketing language around geometry, materials, and scene understanding.
DLSS 5 does not appear to read 3D geometry, depth, or artist-authored material data directly from the game engine. When asked whether the model is effectively taking a single 2D frame with motion vectors to create the output, the answer was: “ Yes, DLSS 5 takes a 2D frame plus motion vectors as input .” NVIDIA also said the model understands scene semantics such as hair, fabric, skin, and lighting conditions “ all by analyzing a single frame.”
NVIDIA had earlier described DLSS 5 in a way that suggested a deeper understanding of the scene. The follow-up answers paint a narrower picture. When asked whether the model reads PBR (Physically Based Rendering) properties from the engine, NVIDIA said: “DLSS 5 only takes the rendered frame and motion vectors as inputs. Materials are inferred from the rendered frame.” In other words, the model is not reading metallic, roughness, normal maps, or other underlying material properties directly.
DLSS5 in Starfield, Source: NVIDIA/Daniel Owen example
That may explain why some preview images raised concerns. In one example, a character appears to gain hair detail in an area where it was not visible before. In another, facial details appear altered enough (like the nose) to raise questions about whether the model is changing the look of the character rather than only improving lighting. NVIDIA’s response was that “the underlying geometry is unchanged,” while also adding that this is “a very early preview of the tech.” That answer addresses the engine-side model data, but it does not change the fact that the final displayed image can look different.
NVIDIA’s answers also suggest that developer control is still fairly limited . The company says developers can adjust intensity, color grading, blending, contrast, saturation, gamma, and apply masks to exclude specific objects or areas from enhancement. However, the answers do not point to any direct way for developers to correct a specific generated facial detail, makeup-like effect, or other scene interpretation beyond reducing, masking, or disabling the effect. If only NVIDIA allowed developers to supply lets say higher quality renders for faces that would be DLSS5 added, that would probably put some complaints to rest, but it seems that faces are still fully AI-generated.
Here is the full exchange between Daniel and Jacob:
1: My biggest question is whether or not this model is effectively taking a single 2D frame as an input (with motion vectors) to create the output frame? This then relates to some clarifying questions about the information you provided, which I ask below:
Answer: Yes, DLSS 5 takes a 2D frame plus motion vectors as input.
2: You say the underlying geometry and textures are unchanged. Is the model actually aware of things like 3D geometry, 3D depth, etc, or is it essentially just looking at a 2D image and then reinterpreting it similar to taking a screenshot of a game and asking an AI model to enhance the screenshot to make it more realistic?
Answer: DLSS 5 is trained end to end to understand complex scene semantics such as characters, hair, fabric and translucent skin, along with environmental lighting conditions like front-lit, back-lit or overcast, all by analyzing a single frame.
4. You say that “With DLSS 5 the underlying geometry and textures are unchanged.” However, in one of the publicly available examples, it looks like a character grew hair in a place where it didn’t exist before. This seems to be a change to the model geometry and/or textures rather than just an enhancement of the lighting. In other words, it seems like an AI image generator reinterpreting the character, which can cause actual details to be changed. It also appears like some characters gain makeup that the original model did not seem to be wearing, which seems like changing the underlying texture.
Answer: The underlying geometry is unchanged. Also worth mentioning this is a very early preview of the tech.
4. You say it enhances PBR properties on materials. Is it aware of the PBR properties by reading them from the game engine in some way, or is it just “looking” at the output image and inferring them. In other words, it is my understanding that game artists traditionally tell a shader what specific parts of a 3D model look like by giving it properties like metallic, roughness, normal maps (to simulate “bumpiness”), etc. Is DLSS5 taking those artist created values as inputs, or is it inferring them from something more like the final output image.
Answer: DLSS 5 only takes the rendered frame and motion vectors as inputs. Materials are inferred from the rendered frame.
5. When you describe the tools the developers have to control the model, it does not sound like there is actually a lot of control over the DLSS5 model’s interpretation of the artistic intent of a scene beyond color grading (or turning DLSS5 “off” or “down” on some or all elements of the scene). For example, in the opening scene of Resident Evil Requiem, Grace is a character with a lot of trauma, who is reluctantly on her way to investigate a murder in the same location where her mother was murdered several years earlier. With DLSS5 Off, I get more of that impression of the character. With DLSS5 on, while she does seem to have more photo realistic lighting and skin, it also looks like she is wearing makeup that was not present in the original version. This gives off a very different impression of the character and emotional impact of the scene. Can developers actually control anything about the output image besides color grading? If they see the DLSS5 output and want to tweak it, like asking it not to apply makeup to the character, are they able to do so?
Answer: Developers will have detailed controls such as intensity and color grading. Artists can use these controls to adjust blending, contrast, saturation and gamma, and determine where and how enhancements are applied to maintain the games’ unique aesthetic. Developers can also mask specific objects or areas to be excluded from enhancement. We continue to talk to developers to understand all the ways they would like to control the technology. Ultimately, we see DLSS 5 as a tool for them to achieve their artistic vision, rather than be limited by the capabilities of traditional real-time rendering.
6. Is this model limited to screen space? Or does it have any awareness of the environment outside of what is visible in an individual game frame?
Answer: DLSS 5 only takes the rendered frame and motion vectors as inputs.
7. With the introduction of DirectML to DirectX allowing for hardware agnostic acceleration of ML tasks in games, does Nvidia plan to continue with vendor locked versions of their machine learning based game technology like DLSS5? Or will Nvidia allow these models to run through DirectML on any hardware?
Answer: We have nothing to announce on this one at this time.
Source:
