I have been playing around with different clock settings by tweaking the kernel in a Linux environment from an academic interest perspective, with the end goal of making the jump to commercial scaling at datacenters for LLM inference models.
I came across this article, however I don’t really like the python(slow) or arch specific rust(massive) wrappers of native C nvml. So most of the testing is done with lightweight shell scripts with the native Nvidia tools and kernel modifiers. It looks like I could replicate my_oc with nvidia_oc. What I don’t know, is if my_oc works on Tesla GPUs(?)
At the home lab testing levels of my_oc, here are my key results after using a GTP4 *Q8_O model with all 33 layers loaded onto the GPU through and ssh connection. At scale, this should create negligible CPU change.
stock: delta of 170W @ 0.16 tpm/W
my_oc: delta of 147W @ 0.18 tpm/W
[tpm/W = tokens per minute per Watt]
Would these results be worth pursing further on a larger cluster using nvidia_oc?
Ubuntu-Jammy NVIDIA-SMI 535.230.02 Driver Version: 535.230.02 CUDA Version: 12.2
llama.cpp
version: 5863 (a4575513)
built with cc (Ubuntu 11.4.0-2ubuntu1~20.04) 11.4.0 for x86_64-linux-gnu
3070 Compute M. P8
Device 0: NVIDIA GeForce RTX 3070