Error during multi-GPU training of classification_tf1: cma_ep.c process_vm_readv Operation not permitted

Accelerated Computing Intelligent Video Analytics TAO Toolkit
5

Can you run below experiments to check if multi gpus work?

$ docker run --runtime=nvidia -it --rm nvcr.io/nvidia/tensorrt:22.11-py3 /bin/bash

Then inside the docker
$ git clone https://github.com/NVIDIA/nccl-tests.git
$ cd nccl-tests/
$ make
$ ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 4
$ ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3