Horovod using only a gpu no matter what np value?


I am trying to run a horovod program with the tensorflow container https://ngc.nvidia.com/containers/nvidia:tensorflow/. I try to use the 8 GPUs of a DGX1 machine, so I run my code with horovodrun -np 8 (but I also tried mpirun -np 8). By doing so, I actually get 8 python processes created but all of them use the same GPU (i.e., only a GPU out of the 8 available is used). Somehow, give me the impression that all the processes are getting the same rank.