Sorry in advance as this is a little off-topic, but I feel vLLM is hot topic in this forum so I think this is the place to reach the right people.
I’m struggling to figure out why running Gemma 4 31B (specifically, the 4-bit AutoRound version from Intel: Intel/gemma-4-31B-it-int4-AutoRound · Hugging Face) results in excessive VRAM use in vLLM 0.23.0. The model weights are about 18GB, and I am running a dual-GPU system with an RTX 4090 and 5070 Ti. The model’s 60 layers are split across both GPUs unevenly using pipeline parallel mode: 40 on the 4090, and 20 on the 5070 Ti. This all works, with reasonable performance (44 t/s), but after launching vLLM, I only have about 7GB of VRAM left over for KV cache. I think this should be WAY higher. With --gpu-memory-utilization set to 0.94, nvtop reports 14GB used on the 5070 Ti, and 22GB used on the 4090. With 36GB of total VRAM in play, I would expect far more than 7GB left over for KV cache.
vLLM reports an available context length of about 170K tokens. I feel I should be able to max out the model’s 262K context with the amount of VRAM available, but I can’t figure out what the problem is or what I need to change to make it work. Any ideas on this would be greatly appreciated!
this is really off topic, my man. not gb10, and you are splitting unevenly across 2 cards so not tensor parallel. I would try reddit/localllama…
Off topic for this forum but where are you getting your 7GB for cache number?
What is your max concurrency? (Max num seq) You can probably reduce that I believe if you want more cache at the cost of less concurrent requests
160.94 = ~15
240.94 = ~23
128*0.95 = ~121
Math seems to be mathin’.
Thanks for bearing with me. The ~7GB cache number comes from vLLM’s output when it’s loading. I managed to squeeze a bit more out of it after disabling CUDA graphs.
–max-num-seqs 2, but I tried 1 and it didn’t seem to make a difference.
Here’s the relevant snippet from vLLM. Check it out:
(Worker_PP0 pid=4549) INFO 07-04 21:50:56 [default_loader.py:397] Loading weights took 4.46 seconds
(Worker_PP0 pid=4549) INFO 07-04 21:50:57 [gpu_model_runner.py:5187] Model loading took 13.46 GiB memory and 4.970449 seconds
(Worker_PP0 pid=4549) INFO 07-04 21:50:57 [gpu_model_runner.py:6200] Encoder cache will be initialized with a budget of 8192 tokens, and profiled with 3 video items of the maximum feature size.
(Worker_PP0 pid=4549) INFO 07-04 21:51:13 [gpu_worker.py:480] Available KV cache memory: 7.64 GiB
(EngineCore pid=4539) INFO 07-04 21:51:13 [kv_cache_utils.py:1905] Auto-fit max_model_len: reduced from 262144 to 193760 to fit in available GPU memory (7.64 GiB available for KV cache)
(EngineCore pid=4539) INFO 07-04 21:51:13 [kv_cache_utils.py:1744] GPU KV cache size: 193,760 tokens
(EngineCore pid=4539) INFO 07-04 21:51:13 [kv_cache_utils.py:1745] Maximum concurrency for 193,760 tokens per request: 1.00x
(Worker_PP0 pid=4549) INFO 07-04 21:51:13 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(Worker_PP1 pid=4550) INFO 07-04 21:51:13 [jit_monitor.py:54] Kernel JIT monitor activated — Triton JIT compilations during inference will be logged as warnings.
(EngineCore pid=4539) INFO 07-04 21:51:13 [core.py:313] init engine (profile, create kv cache, warmup model) took 16.26 s
(EngineCore pid=4539) INFO 07-04 21:51:15 [vllm.py:999] Asynchronous scheduling is enabled.
(EngineCore pid=4539) WARNING 07-04 21:51:15 [vllm.py:1055] Enforce eager set, disabling torch.compile and CUDAGraphs. This is equivalent to setting -cc.mode=none -cc.cudagraph_mode=none
(EngineCore pid=4539) WARNING 07-04 21:51:15 [vllm.py:1097] Inductor compilation was disabled by user settings, optimizations settings that are only active during inductor compilation will be ignored.
(EngineCore pid=4539) INFO 07-04 21:51:15 [kernel.py:270] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=['vllm_c', 'native'], fused_add_rms_norm=['vllm_c', 'native'])
(EngineCore pid=4539) INFO 07-04 21:51:15 [vllm.py:1273] Cudagraph is disabled under eager mode
Here’s the VRAM use according to nvidia-smi:
Afaik there is overhead that consumes memory on each card beyond just the model weights themselves.
Is MTP on? You can also turn it off to get more memory. Tbf 170k is about in line with what I would expect for 7 GB. The quality of results as context gets very large isn’t great even with frontier models, unfortunately.
I agree with that. The issue is why only 7GB left over from nearly 40GB total, when the model weights themselves are only about 18GB? Where did the other 15GB go? That’s what I’m having trouble figuring out.
