# Accelerate Large-Scale LLM Inference and KV Cache Offload with CPU-GPU Memory Sharing

**URL:** <https://forums.developer.nvidia.com/t/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/344150>\
**Category:** Technical Blog\
**Tags:** llama\
**Created:** [September 5, 2025, 5:24pm UTC](https://forums.developer.nvidia.com/t/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/344150 "2025-09-05T17:24:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![jwitsoe](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jwitsoe/32/16591_2.png) [@jwitsoe](https://forums.developer.nvidia.com/u/jwitsoe)\
**Post date:** [September 5, 2025, 5:24pm UTC](https://forums.developer.nvidia.com/t/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/344150/1 "2025-09-05T17:24:17Z")

</div>

Originally published at: [Accelerate Large-Scale LLM Inference and KV Cache Offload with CPU-GPU Memory Sharing | NVIDIA Technical Blog](https://developer.nvidia.com/blog/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/)  
   
Large Language Models (LLMs) are at the forefront of AI innovation, but their massive size can complicate inference efficiency. Models such as Llama 3 70B and Llama 4 Scout 109B may require more memory than is included in the GPU, especially when including large context windows.&nbsp; For example, loading Llama 3 70B and Llama 4…

---

<div class="post-metadata">

**Author:** ![TomNVIDIA](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tomnvidia/32/14181_2.png) [@TomNVIDIA](https://forums.developer.nvidia.com/u/TomNVIDIA)\
**Post date:** [September 5, 2025, 5:29pm UTC](https://forums.developer.nvidia.com/t/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/344150/2 "2025-09-05T17:29:44Z")

</div>

Listed on September 5, 2025, 5:29pm UTC



---

<div class="post-metadata">

**Author:** ![tom.schelsen](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@tom.schelsen](https://forums.developer.nvidia.com/u/tom.schelsen)\
**Post date:** [September 15, 2025, 12:25pm UTC](https://forums.developer.nvidia.com/t/accelerate-large-scale-llm-inference-and-kv-cache-offload-with-cpu-gpu-memory-sharing/344150/3 "2025-09-15T12:25:57Z")

</div>

Any ready made Docker container that can leverage this to serve LLMs with an OpenAI compatible API, without one having to re-implement the code shown in the blog post + the web server + metrics, etc…. ? Would the **`--cpu-offload-gb`** parameter in vLLM be equivalent, or is some performance left on the table using this instead of RMM ? Thanks