# How to Deploy LLMs on RTX PCs **URL:** \ **Category:** Announcements\ **Tags:** kb, llama\ **Created:** [December 18, 2024, 5:30pm UTC](https://forums.developer.nvidia.com/t/how-to-deploy-llms-on-rtx-pcs/317354 "2024-12-18T17:30:31Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![TomNVIDIA](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tomnvidia/32/14181_2.png) [@TomNVIDIA](https://forums.developer.nvidia.com/u/TomNVIDIA)\ **Post date:** [December 18, 2024, 5:30pm UTC](https://forums.developer.nvidia.com/t/how-to-deploy-llms-on-rtx-pcs/317354/1 "2024-12-18T17:30:31Z")
This document covers our recommendations to deploy large language models on NVIDIA RTX GPUs to get the best performance and features. ## Selecting the Right Models for Your Needs Thousands of models exist in the open-source community, all of which are accelerated on RTX today. Applications have differing needs for use-cases, requirements and performance. These requirements have to be considered in conjunction with the choice of inference backend. Additionally, many application developers choose to customize open-source models to fit their needs. Sample requirements to consider: Context Length, Precision, Use-Case Quality, Languages, Modalities, Memory Usage, base model vs. instruct-tuned model. The table below showcases the most popular foundation models we are seeing in the community today. Additionally, there are 100s of fine-tuned variations, and LoRA adapters available for these models on HF which developers can make use of. Most developers choose a Instruct-tuned variant of the base model for most chat use-cases. Most popular foundation models below: | Base Model | Use-case | | --- | --- | | [gpt-oss:20B](https://huggingface.co/openai/gpt-oss-20b) | Reasoning & Agentic tasks | | [Deepseek-r1 Family](https://huggingface.co/deepseek-ai/DeepSeek-R1) | Reasoning Models | | [Gemma3 Family](https://deepmind.google/models/gemma/gemma-3/) | Text Generation Model | | [Qwen3 Family](https://huggingface.co/collections/Qwen/qwen3-67dd247413f0e2e4f653967f) | Text Generation Model | | [Llama3 Family](https://www.llama.com/models/llama-3/) | Text Generation Model | | [Mistral Family](https://huggingface.co/mistralai) | Text Generation Model | Developers can use this [leaderboard](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) for reference on best performing models (on academic benchmarks). Developers can also filter and explore on HuggingFace. ### Compute Requirements & Considerations One of the key considerations when deploying LLM models on the PC is the VRAM requirement for the model. To get maximum quality within a VRAM budget, developers optimize models with quantization techniques. These techniques shrink the precision format of model weights & activations, to lower the VRAM usage and accelerate the computation during inference. Popular quantization formats include: MXFP4, Q4\_K\_M and Int4. In addition, when considering different LLMs developers often contemplate the following considerations: Context Length, Precision, Use-Case Quality, Languages, Modalities, Memory Usage, base model vs. instruct-tuned model. ## Inference & Deployment For developers who are looking for Python and C++ Windows native solutions. We generally recommend: 1. **Ollama** : For LLM developers who want to experiment quickly with the latest local LLM models. Wide reach cross-vendor and cross-OS support, employs a server-based approach 2. **Llama.cpp** - For LLM developers who want to experiment quickly with the latest local LLM models. Wide reach cross-vendor and cross-OS support, with a path to in-app deployment on the PC. 3. **Windows ML with NVIDIA TensorRT for RTX** - For application developers building AI features for Windows PC applications. 4. **TensorRT for RTX** - Maximum performance and full flexibility on NVIDIA GPUs. | | Ollama / Llama.cpp | Windows ML with NVIDIA TensorRT for RTX | TensorRT for RTX | | --- | --- | --- | --- | | For | LLM developers who want wide reach with cross-vendor and cross-OS support | Application developers building AI features for Windows PC | Windows application developers who want maximum control and flexibility of AI behavior on NVIDIA RTX GPUs | | Performance | Fast | Fastest | Fastest | | OS Support | Windows, Linux, and Mac | Windows | Windows and Linux | | Hardware Support | Any GPU or CPU | Any GPU or CPU | NVIDIA RTX GPUs | | Model Checkpoint Format | GGUF or GGML | ONNX | ONNX | | Installation Process | Installation of Python packages required | Pre-installed on Windows | Install SDK and Python bindings | | LLM Support | | | | | Model Optimizations | Llama.cpp | Microsoft Olive | TensorRT-Model Optimizer | | Python | | | | | C/C++ | | | | | C#/.NET | | | - | | Javascript | | | - | Review these resources for comparisons, information and getting started resources: - [Overview of Inferencing Backends](https://developer.nvidia.com/ai-apps-for-rtx-pcs/inference-backends) - **Getting Started Resources** : - [Ollama Inferencing Guide](https://github.com/ollama/ollama) | [Optimization Guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/README.md) - [Llama.cpp Inferencing Guide](https://github.com/ggml-org/llama.cpp) | [Optimization Guide](https://github.com/ggerganov/llama.cpp/blob/master/examples/quantize/README.md) - Windows ML: [Inferencing Guide](https://learn.microsoft.com/en-us/windows/ai/new-windows-ml/overview) | [Optimization Guide](https://microsoft.github.io/Olive/) | [Samples](https://github.com/microsoft/WindowsAppSDK-Samples/tree/feature/winml-stable/Samples/WindowsML) ## Fine-Tuning Developers can customize and fine-tune LLM models on their RTX AI PCs to tailor model responses to particular use-cases and tasks. For example, a custom personal email writing assistant. To do this, we recommend that you use Unsloth which is a popular community framework for quantization, adapter tuning, fine-tuning, and reinforcement learning on NVIDIA RTX GPUs. Refer to the following [getting started guide for Unsloth](https://docs.unsloth.ai/get-started/beginner-start-here). ## Example Workflows & Application Integration Tools - Function calling chat bot (Electron app using node-llama-cpp in Electron app): [GitHub Link](https://github.com/NVIDIA/RTX-AI-Toolkit/tree/main/examples/node-llama-cpp-app) - Agentic Frameworks: - VectorDBs: ---
**Author:** ![TomNVIDIA](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tomnvidia/32/14181_2.png) [@TomNVIDIA](https://forums.developer.nvidia.com/u/TomNVIDIA)\ **Post date:** [December 22, 2024, 4:02pm UTC](https://forums.developer.nvidia.com/t/how-to-deploy-llms-on-rtx-pcs/317354/2 "2024-12-22T16:02:36Z")