Isaac Sim A100 & apptainer/singularity support?

Omniverse Isaac Sim
,
1

Hi,

I’m trying to run Isaac Sim on a remote HPC cluster with A100 GPUs. Docker isn’t installed, so i’m trying to use apptainer (aka singularity).

The command i’m running and a transcript of the output follow. But first a more basic question: is Isaac Sim supported on the A100? Most of the errors are RTX related, and the A100 is a non-RTX GPU, i believe.

Here’s small sample of the RTX errors:

[Error] [gpu.foundation.plugin] Cannot load shader file 'rtx/database/DirtyPageCopy.compute.hlsl'.
[Error] [gpu.foundation.plugin] Cannot load shader file 'rtx/database/DirtyElementsCopy.compute.hlsl'.
[Error] [gpu.foundation.plugin] buildShaderAsync() failed: rtx/database/DirtyElementsCopy.compute.hlsl
[Error] [gpu.foundation.plugin] buildShaderAsync() failed: rtx/database/DirtyPageCopy.compute.hlsl
[Error] [gpu.foundation.plugin] PsoCompute::createPipeline and shader loading failed. RtxResult: 3
[Error] [gpu.foundation.plugin] PsoCompute::createPipeline and shader loading failed. RtxResult: 3
...

Here’s the apptainer command:

apptainer exec --nv -C \
    --env "ACCEPT_EULA=Y" \
    --env "PRIVACY_CONSENT=Y" \
    -B "$data/cache/kit:/isaac-sim/kit/cache" \
    -B "$data/kit/exts/omni.gpu_foundation/cache:/isaac-sim/kit/exts/omni.gpu_foundation/cache" \
    -B "$data/cache/ov:$HOME/.cache/ov" \
    -B "$data/cache/pip:$HOME/.cache/pip" \
    -B "$data/cache/glcache:$HOME/.cache/nvidia/GLCache" \
    -B "$data/cache/computecache:$HOME/.nv/ComputeCache" \
    -B "$data/logs:$HOME/.nvidia-omniverse/logs" \
    -B "$data/data:$HOME/.local/share/ov/data" \
    -B "$data/documents:$HOME/Documents" \
    ./isaac-sim_2023.1.1.sif \
    /isaac-sim/runheadless.webrtc.sh -v

Here’s the full log output (4.3 MB).

And here’s the output of nvidia-smi:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-PCIE-40GB          On  | 00000000:65:00.0 Off |                    0 |
| N/A   40C    P0              33W / 250W |      0MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
3

Hi. The A100 is not supported. Isaac Sim requires a GPU with RT cores for rendering. NVENC is also required for live-streaming.

4

I have had success running Isaac Sim in Docker with A100s with Isaac Sim 4.0.0. There are still some errors (which may be unrelated) but the simulator works fine. While A100s cant use the built-in streaming functionality because they lack the NVENC chip it is still possible to control remotely with a VNC server. Specifically, here is a sample from my dockerfile:

FROM isaac-lab-base:4.0.0 AS base

RUN apt-get update && apt-get install -y \
    xorg \  
    x11vnc \
    fonts-freefont-ttf \
    websockify \
    xserver-xorg-video-dummy

COPY ./xorg.conf /etc/X11/xorg.conf
RUN git clone https://github.com/novnc/noVNC.git /opt/novnc && \
    cd /opt/novnc && \
    git checkout v1.5.0
RUN cp /opt/novnc/vnc.html /opt/novnc/index.html

COPY ./start.sh ./
CMD ["./start.sh"]

And here the start script:

export DISPLAY=:0
export MESA_GL_VERSION_OVERRIDE=4.6

# Start Xorg
Xorg :0 &

# Start x11vnc to allow VNC connections to the X server
x11vnc -rfbport 5901 -display :0 -forever -nopw -shared &

# Start the noVNC proxy (web app)
/opt/novnc/utils/novnc_proxy --vnc localhost:5901 --listen 6901 &

# Run Isaac Sim
/isaac-sim/isaac-sim.sh

For completion here is the xorg.conf which creates a dummy display for the XServer:

Section "Device"
    Identifier  "Configured Video Device"
    Driver      "dummy"
EndSection

Section "Monitor"
    Identifier  "Configured Monitor"
    HorizSync 31.5-48.5
    VertRefresh 50-70
EndSection

Section "Screen"
    Identifier  "Default Screen"
    Monitor     "Configured Monitor"
    Device      "Configured Video Device"
    DefaultDepth 24
    SubSection "Display"
    Depth 24
    Modes "1920x1080"
    EndSubSection
EndSection

You can access your VNC server with clients such as TigerVNC but with the above solution NoVNC exposes the stream as a web server that you can access at <ip_address>:6901

1 Like