nvidia-smi in a side-terminal to loosely track kernels. When I use a 2-GPU system and have heavy workloads with multiple processes and kernels, I see what look like “extra” memory footprints of my kernels on device 0. For example, if I have kernel A on device 0 and kernel B on device 1, nvidia-smi will show the footprint of A on 0, of B on 1, AND something like an extra 256MB for both A and B on device 0. (So in summary, device 0 will be reporting 3 different lines, and device 1 will be reporting just 1.)
Has anyone else experienced this or have an explanation for these “extra” lines?
Thank you in advance.