Slab memory leak when running cuda apps on 450.66

Graphics / Linux Linux
1

Running our cuda application fills up memory with SUnreclaim kmalloc-32 SLABS over time.
grep SUnreclaim /proc/meminfo increased from 243696 KB to 1668540 KB within 24 hours. That’s about 70MB/h. The same software ran nicely for years, the same exact binaries for months.

One node uses 90 GB on these kmalloc-32 slabs. (uptime: 64 days)
We have 9 heterogeneous nodes, 4 of them show this problem.
All of the affected systems run driver 450.66.
All systems install drivers from rpmfusion.
3 affected systems run centos7 x86_64 on an intel dual socket CPU
1 affected system runs centos8 x86_64 on an AMD Ryzen Threadripper 3960X

any ideas?

2

Hi, did you make any progress with this problem? I’ve been experiencing exactly the same across 10 machines.
Did you try upgrading the Nvidia Drivers?

3

As rpmfusion did not provide newer drivers for centos-7 we upgraded to centos-8.
problem seems solved there with the newer driver.

But now centos8 goes stream (aka. beta) so we do not really know where to go from here :/