Tao fail when using more than 2 GPUS - NCCL WARN Call to posix_fallocate failed : No space left on device

[Morganh]

@levi3

but we dont have control on docker parameters when docker start container, because tao control docker container.

You can control the docker parameters.

Because we can trigger tao docker with below method as well. For example,
$ docker run --runtime=nvidia -it -rm nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.5-py3 /bin/bash