ChatRTX, the new demo of local LLM powered by NVIDIA GPUs
NVIDIA has updated the tool now supporting more Large Language Models (LLMs).
If you’re wondering why NVIDIA is launching ChatRTX today when a similar tool has already been released, the confusion might arise from the fact that “Chart with RTX” was just a pre-launch tech demo, whereas ChatRTX is the new app, however still officially a demo.
Unlike online tools such as ChatGPT, Google Bard, or Microsoft Copilot, the AI assistant powered by NVIDIA GeForce GPUs operates locally and doesn’t require external internet calls. This aspect significantly impacts privacy and control over the analyzed data. NVIDIA has emphasized the ability to scan and analyze data in local directories (text files, documents, etc.). This is undoubtedly one of the coolest features of this tool, which enables users to utilize existing open-weight models and local data.
New LLMs in ChatRTX,Source: NVIDIA
New features in ChatRTX include voice recognition for prompts in multiple languages and the ability to output in different languages. Additionally, there’s support for new LLMs like Mistral 7B INT4 or Llama 2 7B INT4, as well as support for Gemma , the latest AI model from Google. A cool new feature is the ability to search images in local directories, powered by CLIP (Contrastive Language-Image Pre-training), an OpenAI model.
Photo searching in Chat RTX, Source: NVIDIA
Unfortunately, one of the most requested features, the ability to ask follow-up questions, is not yet available in ChatRTX. The tool can only understand single queries at the moment. Furthermore, the web-based user interface doesn’t function properly with Firefox yet.
It’s also worth noting that the installation program weighs in at 11GB, while the full application, once installed, requires 35GB of space, so ensure you have enough space before downloading. One more thing to note is that it only works on GeForce RTX 30 and RTX 40 GPUs for now, with at least 8GB of VRAM and 16GB of system memory.
Source: NVIDIA via TechPowerUP