Recommendations on How to Deploy Diffusion Models on NVIDIA RTX PCs

AI & Data Science Announcements
,
1

Introduction

Generative AI has transformed content creation. User interest for these models is growing rapidly, as well as adoption. NVIDIA offers the best hardware solution to run these models, providing the best performance and support.

This document covers our recommendations to deploy these models to get the best performance and features.

Selecting the Right Model for Your Needs

There are many high-quality models that exist in the open-source community, all of which are accelerated on RTX today. When selecting a model for your application, consider both model capabilities (performance and quality), deployment requirements (VRAM and format), and the compatibility with external tools like controlnets, and the ability to fine tune models.

The table below showcases some of the most popular foundation models we are seeing in the community today.

Use Case Models Links
Image Generation FLUX.1 PyTorch
TensorRT
NIM
Image Generation Stable Diffusion 3.5 PyTorch (Large)
TensorRT (Large)
PyTorch (Medium)
TensorRT (Medium)
NIM (Large)
Video Generation WAN 2.2 PyTorch (T2V 14B)
Video Generation LTX Video PyTorch
Video Generation Kling 2.0 Website
3D Generation Microsoft TRELLIS PyTorch
NIM
3D Generation Hunyuan 3D v2 PyTorch

Developers can use this leaderboard for reference on best performing image generation models (on academic benchmarks) and this one for video generation models. Developers can also filter and explore on HuggingFace.

Inferencing & Deployment

Windows Native Devs

For ISV developers looking to deploy diffusion models on production on a large scale applications with Python or C++, we recommend:

For developers looking to prototype a solution or use the latest and greatest models, we recommend:

  • PyTorch -CUDA: Highest flexibility and support with direct compatibility with new models, but harder to deploy.

Compute Requirements & Considerations

One of the key considerations when deploying diffusion models is the VRAM requirement for the model, input data, and output data. Note that different settings may increase VRAM requirements, such as image resolution, number of diffusion steps, batch size, model type and version, and sampler.

Model can be optimized with techniques:

  • Quantization: Reduces the precision format of model weights & activations. This reduces the model’s memory footprint, lowers the VRAM usage and accelerates the computation during inference. Popular quantization formats include: FP8, and NVFP4.

  • Step Distillation: Reduces the number of diffusion steps to generate an image (e.g. SDXL Turbo, FLUX Schnell, etc.)

  • Caching: There are a variety of methods that store intermediate results and re-use them if possible reducing the amount of redundant compute. E.g. TeaCache, CacheDiffusion

These models can be optimized on NVIDIA GPUs with CUDA Graphs which minimize the CPU overhead when running the model. Especially effective on Windows and models with short inference time (e.g. SD 1.5).

Here are some top models that have already been optimized for NVIDIA GPUs. ComfyUI offers a TensorRT node to convert these ONNX models to TensorRT.

Fine-Tuning

Diffusion models can be fine-tuned to customize models to perform better for certain tasks, such as a retailer who wants output images to match their branding style. Popular fine-tuning tools include kohya_ss and DreamBooth.

Example Workflows & Application Integration Tools

  • User Interfaces:

  • Blueprints & Reference Workflows:

    • 3D Guided Generative AI Blueprint - The 3D Guided Generative AI Blueprint unlocks greater control over image generation by laying out the content in Blender to guide the image layout. Users can quickly alter the look of the 3D scene using generative AI, and the image outputs can be iterated on by making simple changes in the 3D viewport - such as changing the image perspective by adjusting the camera angle in Blender. Creators can ideate on scene environments much faster using generative AI, and adjustments are made much faster due to the control offered by using the viewport as a depth map.

    • ComfyUI Workflow Templates - ComfyUI, the most popular PC visual GenAI user interface, contains a variety of ready-to-use, reusable setups of nodes designed to help users quickly get started with complex tasks like AI image generation, video creation, 3D model generation, and advanced post-processing. Rather than building an entire workflow from scratch, users can choose a template that fits their needs, load it into ComfyUI, and immediately begin creating or customizing.