Isaac Sim crashes when running with PathTracing in container

[christianbarcelo]

Hi!
Given the issue with RT that I reported here, I’m trying to use PathTracing from within Isaac Sim’s container. When I enable PathTracing, I start receiving the error:

As can be seen in the image, it has something to do with the Denoiser not being able to load the weights.
The Denoiser seems to instantiate or spawn the same process with the same error over and over again until I eventually get:

Besides the fact that the first dozens of errors are not related to memory, the exact same computer is able to run the same version of Isaac Sim from the host (using the Launcher and so on) with Path Tracing and no issue at all.
Disabling the Denoiser is not an option, the results are simply unusable:

To provide some extra information, my current use case for this is with a custom pipeline using Replicator BUT the same can be achieved by bringing up the container, opening Isaac Sim, and enabling Path Tracing as can be seen in the video:

[christianbarcelo]

Hi! Gentle ping to see if there’re any news here.

[JuanCarosella]

I’ve been struggling with this for a while now too. I’m using the container instructions present in the Nvidia omniverse docs

[Richard3D]

The solution is to download the latest container here:
Release v1.15.0-rc.1 · NVIDIA/nvidia-container-toolkit · GitHub

and then use THIS COMMAND
NVIDIA_DRIVER_CAPABILITIES=all

[christianbarcelo]

Hi! Thanks for your reply.

Do you mean the Nvidia Container Toolkit? If so, I’m using v1.15.0 (which is newer than the rc1 you mentioned)

I’m still seeing the same issue.
My Docker command looks like this:

docker run -it --rm --gpus all --name my_container -e ACCEPT_EULA=Y -e DISPLAY=:1 -e NVIDIA_DRIVER_CAPABILITIES=all -e PRIVACY_CONSENT=Y -v ~/docker/isaac-sim/cache/kit:/isaac-sim/kit/cache:rw -v ~/docker/isaac-sim/cache/ov:/root/.cache/ov:rw -v ~/docker/isaac-sim/cache/pip:/root/.cache/pip:rw -v ~/docker/isaac-sim/cache/glcache:/root/.cache/nvidia/GLCache:rw -v ~/docker/isaac-sim/cache/computecache:/root/.nv/ComputeCache:rw -v ~/docker/isaac-sim/logs:/root/.nvidia-omniverse/logs:rw -v ~/docker/isaac-sim/data:/root/.local/share/ov/data:rw -v ~/docker/isaac-sim/documents:/root/Documents:rw -v /dev/dri:/dev/dri -v /tmp/.X11-unix:/tmp/.X11-unix nvcr.io/nvidia/isaac-sim:2023.1.1

which is the same command found in here with some extras.

[Richard3D]

@Richard3D

NVIDIA_DRIVER_CAPABILITIES=all

And add this

[christianbarcelo]

It’s in the command I’ve shared in the previous message, I tried adding it and still does not work

[nnikfetrat]

Hi. In addition to v1.15.0+ NVIDIA container toolkit, please make sure you are running with NVIDIA runtime.

1. Configure the container runtime by using the nvidia-ctk command:

sudo nvidia-ctk runtime configure --runtime=docker

2. Restart the Docker daemon:

sudo systemctl restart docker

3. Start the container requesting --runtime=nvidia and NVIDIA_DRIVER_CAPABILITIES=all, plus other desired options. For example:

docker run it -rm --runtime=nvidia --gpus all --name=xyz -e NVIDIA_DRIVER_CAPABILITIES=all

[christianbarcelo]

Alright, that did the trick.

I’ve been working with Isaac Sim for quite a while now, and it’s the first time I see this happening. I just changed the computer so I assume the required flags/env vars changed with the latest version of ctk, or something like that.

I’ll take advantage of the ticket to report then that the documentation is out to date. There’s no mention neither to the environment variable or the flag.

Thanks for the help!

[Richard3D]

Great ! I am so glad that worked ! Specifically what in the docs is out of date so we can address ?

[christianbarcelo]

The “recommended” docker run command does not include either the --runtime=nvidia or the -e NVIDIA_DRIVER_CAPABILITIES=all. Which again, is not a problem if you’re not going to use PathTracing, but if you do, there’s nothing explaining you need to add that.

[Richard3D]

Thanks. I requested that those lines be added straight away.