Child processes show up separately from parent on nvidia-smi

recently, without changing anything knowingly, there is a change in the output of an nvidia-smi output i’m getting.

it used to be that every experiment that was running on a gpu would show up as one line in the output. now it iss broken down to the main process and next to it are many child processes all consisting of 305 MB.

it looks like this:
±----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 54920 C Insufficient Permissions 9009MiB |
| 1 N/A N/A 22440 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 51512 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 59872 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 72412 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 80316 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 84680 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 95284 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 102060 C …3\envs\env_g\python.exe 9023MiB |
| 1 N/A N/A 102312 C …3\envs\env_g\python.exe 9023MiB |
| 1 N/A N/A 108944 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 115004 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 118164 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 129200 C …3\envs\env_g\python.exe 305MiB |
| 1 N/A N/A 158836 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 25628 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 34464 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 39076 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 44792 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 46720 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 55592 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 57384 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 74716 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 83288 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 88856 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 124900 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 136132 C …3\envs\env_g\python.exe 305MiB |
| 2 N/A N/A 137340 C …3\envs\env_g\python.exe 8707MiB |
| 2 N/A N/A 158128 C …3\envs\env_g\python.exe 8799MiB |
| 3 N/A N/A 20152 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 32260 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 49008 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 69284 C …3\envs\env_g\python.exe 8801MiB |
| 3 N/A N/A 76820 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 85804 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 95104 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 97580 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 104040 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 107568 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 111028 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 135936 C …3\envs\env_g\python.exe 8801MiB |
| 3 N/A N/A 136636 C …3\envs\env_g\python.exe 305MiB |
| 3 N/A N/A 158840 C …3\envs\env_g\python.exe 305MiB |
±----------------------------------------------------------------------------+

notice a few things:

  • another user is running a process and it’s not showing any other processes.
  • on every card there are 1 or 2 main processes, and many other small ones.
  • looking up the mini process in the task manager, under “command line” it says: “from multiprocessing.spawn import spawn_main; spawn_main(parent_pid=10234, pipe_handle=34325)” "–multiprocessing-fork
  • i’m using 2 workers per main process running, so i assume it’s not that.

did anybody come across something similar? would appreciate any advice