ChatRTX with Gemma 7B and Lamma2 13B

[simonchiu]

My GPU is the RTX4070 12GB. Therefore, I changed the “min_gpu_memory”: 16 in the config.json for “Gemma 7B int4” to “min_gpu_memory”: 12, and it still works normally. Here is the usage situation:

My question is, why was the default set to require 16GB?

[vikklontorza]

I would think only Nvidia knows.