Just in case someone else is looking at this thread.
I got this issue resolved by adding some parameters on the docker run command
sudo docker run --runtime nvidia -it --rm --network host \
--shm-size=16g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \
-e DISPLAY=$DISPLAY \
Basically i was missing those 3
--shm-size=16g \
--ulimit memlock=-1 \
--ulimit stack=67108864 \