“Chat with Ryzen/Radeon”
AMD guides how to run local AI chats with their hardware.
AMD does not have its own tool alike to NVIDIA Chat with RTX. NVIDIA came up with a simple tool that can be used to run locally smaller LLM (Large Language Models) powered by GeForce GPUs. While AMD has no such tool, the company suggests one should try using a 3rd party software that runs on CPUs and GPUs.
AMD Ryzen 7040 “Phoenix” and 8040 “Hawk Point” are equipped with XDNA AI accelerator, which can offer anything from 10 TOPS to 16 TOPS (Trillion of operations per second) of AI inference. Soon those AI processors will be utilized broadly and thanks to upcoming changes to Windows, the importance of AI cores will only grow.
Meanwhile, rather than develop their own tool, AMD is recommending that users should check the LM Studio software, which is a closed source, cross-platform and cross-vendor solution. It also comes with easier model management, but also many options that Chat with RTX simply does not have.
LM Studio, Source: VideoCardz
LM Studio works with Apple M-Series, CPUs that have AVX2 support and can also take advantage of AMD and NVIDIA GPU acceleration. For GPUs, the tool has a 16GB VRAM recommendation, while CPU based ML algorithms should have at least 6GB of memory available.
The biggest advantage of this solution over any online models is that it runs locally, and no data is collected and used. The company puts a lot of emphasis on the private use of this software and how they care about privacy.
To use LM Studio on AMD hardware, one must choose between LM Studio and LM Studio ROCm Preview . The latter is optimized for Radeon RX 7000 GPUs. Hopefully be integrated into the main branch at some point.
Once you download the tool, you should run it and find proper models you are eager to work with:
- In the search tab copy and paste the following search term depending on what you want to run:
- If you would like to run Mistral 7b, search for: “ TheBloke/OpenHermes-2.5-Mistral-7B-GGUF ” and select it from the results on the left. It will typically be the first result. We are going with Mistral in this example.
- If you would like to run LLAMA v2 7b, search for: “ TheBloke/Llama-2-7B-Chat-GGUF ” and select it from the results on the left. It will typically be the first result.
- On the right-hand panel, scroll down till you see the Q4 K M model file. Click download.
- We recommend Q4 K M for most models on Ryzen AI. Wait for it to finish downloading.
- Go to the chat tab. Select the model from the central, drop-down menu in the top center and wait for it to finish loading up.
If you have an AMD Ryzen AI PC you can start chatting!
If you have an AMD Radeon™ graphics card , please:
Check “GPU Offload” on the right-hand side panel.
Move the slider all the way to “Max”.
Make sure AMD ROCm™ is being shown as the detected GPU type.
ROCm support, Source: AMD
Worth noting is that the tool is not specifically optimized for AMD XDNA or Intel NPU AI accelerators. It does, however, use OpenCL for AMD APU/CPUs which have integrated graphics.