I have a vm using vmware software, the gpu are running under a vgpu system, the os is ubuntu 22.04 and running cuda version 12.4 with nvidia driver 550.90.07, on this machine i have a gpu Tesla t4
when I starting running a script using pytorch using cuda:0 as a device, it runs normally, I noticed how the gpu works as expected, but after 15-20 minutes suddenly the gpu start working really slowly and when I said slow is like using les than the 1% of the processing power, if i reboot the vm, and run the script again it start working fast and using all gpu capability, but after that period of time (15-20) it slow dramatically.
This happen if i run a complex model like llm llama2-7b or if i run a really simple pytorch code, next the information when it is running at 98%
i need to mention that in all the cases , when running fast or slow, pytorch recognizes cuda available and is able to locale the info in its memory or the gpu
here some data that be helpful
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
llm-lnx1:~$ lshw -c display
WARNING: you should run this program as super-user.
*-display
description: VGA compatible controller
product: SVGA II Adapter
vendor: VMware
physical id: f
bus info: pci@0000:00:0f.0
version: 00
width: 32 bits
clock: 33MHz
capabilities: vga_controller bus_master cap_list rom
configuration: driver=vmwgfx latency=64
resources: irq:16 ioport:1070(size=16) memory:e8000000-efffffff memory:fe000000-fe7fffff memory:c0000-dffff
*-display
description: VGA compatible controller
product: TU104GL [Tesla T4]
vendor: NVIDIA Corporation
physical id: 1
bus info: pci@0000:02:01.0
version: a1
width: 64 bits
clock: 66MHz
capabilities: vga_controller bus_master cap_list
configuration: driver=nvidia latency=0
resources: irq:69 memory:fc000000-fcffffff memory:d0000000-dfffffff memory:fa000000-fbffffff
this is an example code,
import torch
import math
torch.backends.cudnn.benchmark = True
dtype = torch.float
device = torch.device("cuda:0")
torch.backends.cudnn.benchmark = True
# device = torch.device("cuda:0") # Uncomment this to run on GPU
# Create random input and output data
x = torch.linspace(-math.pi, math.pi, 2000, device=device, dtype=dtype)
y = torch.sin(x)
# Randomly initialize weights
a = torch.randn((), device=device, dtype=dtype)
b = torch.randn((), device=device, dtype=dtype)
c = torch.randn((), device=device, dtype=dtype)
d = torch.randn((), device=device, dtype=dtype)
learning_rate = 1e-6
for t in range(2000):
# Forward pass: compute predicted y
y_pred = a + b * x + c * x ** 2 + d * x ** 3
# Compute and print loss
loss = (y_pred - y).pow(2).sum().item()
print(t, loss)
# Backprop to compute gradients of a, b, c, d with respect to loss
grad_y_pred = 2.0 * (y_pred - y)
grad_a = grad_y_pred.sum()
grad_b = (grad_y_pred * x).sum()
grad_c = (grad_y_pred * x ** 2).sum()
grad_d = (grad_y_pred * x ** 3).sum()
# Update weights using gradient descent
a -= learning_rate * grad_a
b -= learning_rate * grad_b
c -= learning_rate * grad_c
d -= learning_rate * grad_d
print(f'Result: y = {a.item()} + {b.item()} x + {c.item()} x^2 + {d.item()} x^3')
for the 98% of the consumption i am using another experiment that is running a llm llama2 model, but in practice the problem is not the amount of processing but the slowness that happen every 15-20 minutos, i need to mention when the gpu start to be so slow it never recovery, only a sudo reboot solves the issue.
here my python libraries:
Package Version
absl-py 2.1.0
accelerate 0.31.0
aiohttp 3.9.5
aiosignal 1.3.1
annotated-types 0.7.0
astunparse 1.6.3
async-timeout 4.0.3
attrs 23.2.0
bitsandbytes 0.43.1
certifi 2024.6.2
charset-normalizer 3.3.2
cmake 3.29.5.1
cuda-python 12.5.0
dataclasses-json 0.6.7
datasets 2.20.0
dill 0.3.8
exceptiongroup 1.2.1
expecttest 0.2.1
filelock 3.15.1
flatbuffers 24.3.25
frozenlist 1.4.1
fsspec 2024.5.0
gast 0.5.4
google-pasta 0.2.0
greenlet 3.0.3
grpcio 1.64.1
h5py 3.11.0
huggingface-hub 0.23.4
hypothesis 6.103.2
idna 3.7
Jinja2 3.1.4
jsonpatch 1.33
jsonpointer 3.0.0
keras 3.3.3
langchain 0.1.11
langchain-community 0.0.38
langchain-core 0.1.52
langchain-text-splitters 0.0.2
langsmith 0.1.81
libclang 18.1.1
lintrunner 0.12.5
Markdown 3.6
markdown-it-py 3.0.0
MarkupSafe 2.1.5
marshmallow 3.21.3
mdurl 0.1.2
ml-dtypes 0.3.2
mpmath 1.3.0
multidict 6.0.5
multiprocess 0.70.16
mypy-extensions 1.0.0
namex 0.0.8
networkx 3.3
ninja 1.11.1.1
numpy 1.26.4
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvcc-cu12 12.3.107
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu12 2.20.5
nvidia-nvjitlink-cu12 12.3.101
nvidia-nvtx-cu12 12.1.105
opt-einsum 3.3.0
optree 0.11.0
orjson 3.10.5
packaging 23.2
pandas 2.2.2
pillow 10.2.0
pip 24.0
protobuf 4.25.3
psutil 5.9.8
psycopg2-binary 2.9.9
pyarrow 16.1.0
pyarrow-hotfix 0.6
pydantic 2.7.4
pydantic_core 2.18.4
Pygments 2.18.0
pyodbc 5.1.0
python-dateutil 2.9.0.post0
pytz 2024.1
PyYAML 6.0.1
quanto 0.2.0
redis 5.0.1
regex 2024.5.15
requests 2.32.3
rich 13.7.1
safetensors 0.4.3
setuptools 65.5.0
six 1.16.0
sortedcontainers 2.4.0
SQLAlchemy 2.0.31
sympy 1.12.1
tenacity 8.4.1
tensorboard 2.16.2
tensorboard-data-server 0.7.2
tensorflow 2.16.1
tensorflow-io-gcs-filesystem 0.37.0
termcolor 2.4.0
tokenizers 0.19.1
torch 2.3.1+cu121
torchaudio 2.3.1+cu121
torchvision 0.18.1+cu121
tqdm 4.66.4
transformers 4.42.0.dev0
triton 2.3.1
types-dataclasses 0.6.6
typing_extensions 4.12.2
typing-inspect 0.9.0
tzdata 2024.1
urllib3 2.2.2
Werkzeug 3.0.3
wheel 0.43.0
wrapt 1.16.0
xxhash 3.4.1
yarl 1.9.4
any help would be really appreciated