Important: Isaac Sim support
Note: For Isaac Sim support, the community is gradually transitioning from this forum to the Isaac Sim GitHub repository so that questions and issues can be tracked, searched, and resolved more efficiently in one place. Whenever possible, please create a GitHub Discussion or Issue there instead of starting a new forum topic.
Note: For any Isaac Lab topics, please submit your topic to its GitHub repo ( GitHub - isaac-sim/IsaacLab: Unified framework for robot learning built on NVIDIA Isaac Sim · GitHub ) following the instructions provided on Isaac Lab’s Contributing Guidelines ( Contribution Guidelines — Isaac Lab Documentation ).
Please provide all relevant details below before submitting your post. This will help the community provide more accurate and timely assistance. After submitting, you can check the appropriate boxes. Remember, you can always edit your post later to include additional information if needed.
6.0.0
5.1.0
5.0.0
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):
Operating System
Ubuntu 24.04
Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):
GPU Information
- Model: NVIDIA L40S (46GB)
- Driver Version: 595.58.03 (CUDA 13.2)
Segmentation Fault when cameras are enabled
Detailed Description
I have installed IsaacSim 5.1.0 and IsaacLab 2.3.1 on a headless HPC node with L40s. I am able to run non-camera based scripts, however any scripts with cameras fail with the error messages given in the “Error Messages” section.
Steps to Reproduce
- Run any camera enabled script on a headless server with L40S wiith IsaacSim 5.1.0 and IsaacLab 2.3.1
Example:
./isaaclab.sh -p scripts/tutorials/04_sensors/add_sensors_on_robot.py --num_envs 2 --enable_cameras --headless
Error Messages
Segmentation fault logs when enabling cameras
(isaaclab) soham.patil@node14:~/IsaacLab$ ./isaaclab.sh -p scripts/tutorials/04_sensors/add_sensors_on_robot.py --num_envs 2 --enable_cameras --headless
[INFO] Using python from: /home/soham.patil/miniconda3/envs/isaaclab/bin/python
[INFO][AppLauncher]: Using device: cuda:0
[INFO][AppLauncher]: Loading experience file: /home/soham.patil/IsaacLab/apps/isaaclab.python.headless.rendering.kit
Loading user config located at: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/user.config.json'
[Info] [carb] Logging to file: /home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/logs/Kit/Isaac-Sim/5.1/kit_20260417_124436.log
2026-04-17T12:44:36Z [3ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/9af75908-58d5-498d-6656ada4-d2823261.dmp'
2026-04-17T12:44:36Z [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/5d40c2a0-05a1-4ef3-5849ddb1-ff532f07.dmp'
2026-04-17T12:44:36Z [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/b1f6020d-6f62-4d5d-940badb4-f87c6766.dmp'
2026-04-17T12:44:36Z [10ms] [Warning] [carb.crashreporter-breakpad.plugin] [previous crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/7558295e-d673-4b60-8fc79f8d-577e5a55.dmp'
2026-04-17T12:44:37Z [1,011ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:37Z [1,011ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:37Z [1,053ms] [Warning] [omni.platforminfo.plugin] failed to open the default display. Can't verify X Server version.
2026-04-17T12:44:37Z [1,057ms] [Warning] [carb.cudainterop.plugin] CUDA_VISIBLE_DEVICES environment variable is set.
2026-04-17T12:44:37Z [1,057ms] [Warning] [carb.cudainterop.plugin] Note CUDA device enumeration and Omniverse device enumeration are different.
2026-04-17T12:44:37Z [1,057ms] [Warning] [carb.cudainterop.plugin] Setting CUDA_VISIBLE_DEVICES can lead to undesired behavior or crashes.
|---------------------------------------------------------------------------------------------|
| Driver Version: 595.58.03 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA L40S | Yes: 0 | | 46068 MB | 10de | 0 |
| | | | | | 26b9 | bfbe89df.. |
| | | | | | 31 | |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 5.15.0-173-generic
| Processor: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
| Cores: 32 | Logical Cores: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515576 | Free Memory: 510131
| Total Page/Swap (MB): 16383 | Free Page/Swap: 16383
|---------------------------------------------------------------------------------------------|
2026-04-17T12:44:38Z [1,729ms] [Warning] [gpu.foundation.plugin] ECC is enabled on physical device 0
2026-04-17T12:44:39Z [2,709ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,709ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,730ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,730ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,740ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,740ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,767ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,767ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,769ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,769ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [2,772ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2026-04-17T12:44:39Z [2,772ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.5]) (impl: carb.windowing-glfw.plugin)
2026-04-17T12:44:39Z [3,222ms] [Warning] [omni.log] Source: omni.hydra was already registered.
Failed to create secure directory (/run/user/2024701005/pulse): No such file or directory
2026-04-17T12:44:40Z [3,709ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [omni::physx::IPhysxBenchmarks v1.0] (plugin: <default plugin>), by client: omni.physics.physx.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
2026-04-17T12:44:40Z [3,730ms] [Warning] [omni.isaac.dynamic_control] omni.isaac.dynamic_control is deprecated as of Isaac Sim 4.5. No action is needed from end-users.
/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/torch/cuda/__init__.py:61: FutureWarning: The pynvml package is deprecated. Please install nvidia-ml-py instead. If you did not install pynvml directly, please report this to the maintainers of the package that installed pynvml for you.
import pynvml # type: ignore[import]
2026-04-17T12:44:43Z [6,408ms] [Warning] [pxr.Semantics] pxr.Semantics is deprecated - please use Semantics instead
2026-04-17T12:44:43Z [6,558ms] [Warning] [omni.graph.core.plugin] Found duplicate of category 'Replicator' - was 'Annotators', adding 'Fabric Reader'
2026-04-17T12:44:43Z [6,558ms] [Warning] [omni.graph.core.plugin] Category 'Replicator' not accepted on node type 'omni.replicator.core.FabricReader' in extension 'omni.replicator.core'
2026-04-17T12:44:43Z [6,559ms] [Warning] [omni.replicator.core.scripts.extension] No material configuration file, adding configuration to material settings directly.
2026-04-17T12:44:49Z [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred. If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds. This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2026-04-17T12:44:50Z [61ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] Wrote dump file '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/badadab5-b991-457f-fb1a10b0-e9dee10d.dmp'
2026-04-17T12:44:50Z [62ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] preventing upload of minidump due to user opt-out: '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/badadab5-b991-457f-fb1a10b0-e9dee10d.dmp'
2026-04-17T12:44:50Z [63ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] dump file size is 5263312 bytes, file is readable.
2026-04-17T12:44:50Z [64ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Crash detected in pid 246307 thread 246307
2026-04-17T12:44:50Z [64ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata:
2026-04-17T12:44:50Z [64ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] BuildGitlabJobID = '209507876'
2026-04-17T12:44:50Z [64ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] CarbSdkVersion = '206.6+release.9587.07f17b1b.gl'
2026-04-17T12:44:50Z [65ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] CrashTime = 'Fri Apr 17 12:44:50 2026 GMT'
2026-04-17T12:44:50Z [65ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] DriverShaderCacheWrapper = 'disabled'
2026-04-17T12:44:50Z [65ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] DumpId = 'badadab5-b991-457f-fb1a10b0-e9dee10d'
2026-04-17T12:44:50Z [66ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] LastUploadStatus = '0'
2026-04-17T12:44:50Z [66ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] ProductName = 'OmniverseKit'
2026-04-17T12:44:50Z [66ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] PythonTracebackStatus = ''py-spy' successfully wrote info to '/home/soham.patil/miniconda3/envs/isaaclab/lib/python3.11/site-packages/isaacsim/kit/data/Kit/Isaac-Sim/5.1/badadab5-b991-457f-fb1a10b0-e9dee10d.py.txt''
2026-04-17T12:44:50Z [67ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] RetryCount = '0'
2026-04-17T12:44:50Z [67ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] StartupTime = '1776429876'
2026-04-17T12:44:50Z [67ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UploadSuccessful = '0'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UptimeSeconds = '14'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UserStory = ''
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UserStoryStatus = 'Running in headless mode; not gathering user story'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Version = '107.3.3+production.229672.69cbf6ad.gl'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_enabled = '1'
2026-04-17T12:44:50Z [68ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_flags = '3'
2026-04-17T12:44:50Z [69ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_status = 'auto-enabled'
2026-04-17T12:44:50Z [69ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_version = '2024.1'
2026-04-17T12:44:50Z [69ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appName = 'Isaac-Sim'
2026-04-17T12:44:50Z [70ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appState = 'started'
2026-04-17T12:44:50Z [70ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appVersion = '5.1.0'
2026-04-17T12:44:50Z [70ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] autoloadExts = ''
2026-04-17T12:44:50Z [71ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildBranch = 'production'
2026-04-17T12:44:50Z [71ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildCi = 'gl'
2026-04-17T12:44:50Z [71ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildConfig = 'release'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabJobName = 'kit-build-release-linux-x86_64'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabJobStage = 'kit-build'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabPipelineID = '34875094'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildHash = '69cbf6ad'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildMajor = '107'
2026-04-17T12:44:50Z [72ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildMinor = '3'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildNumber = '229672'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildPatch = '3'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildVersion = '107.3.3'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] carboniteFrameworkVersion = '206.6+release.9587.07f17b1b.gl'
2026-04-17T12:44:50Z [73ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] carboniteSdkVersion = '206.6+release.9587.07f17b1b.gl'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] commandLine = '/home/<username>/miniconda3/envs/isaaclab/bin/python scripts/tutorials/04_sensors/add_sensors_on_robot.py --num_envs 2 --enable_cameras --headless'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuCoreLimited = '64'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuId = 'Intel64 Family 6 Model 106 Stepping 6'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuLogicalCoresBareMetal = '64'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuName = 'Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz'
2026-04-17T12:44:50Z [74ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuPhysicalCoresBareMetal = '32'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuUsageQuota = '-1.000000'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuVendor = 'GenuineIntel'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] environmentName = 'Individual'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] experience = 'Isaac Sim'
2026-04-17T12:44:50Z [75ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] externalBuild = '1'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuDriver_0 = '595.58'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuVRAM_0 = '48305799168'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpu_0 = 'NVIDIA L40S'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] kitRendererDriverVersion = '595.58'
2026-04-17T12:44:50Z [76ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildBranch = 'release'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildDate = 'Fri Oct 17 03:55:10 2025 +0000'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildHash = '9c81211'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildRepo = 'https://gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim.git'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildVersion = '5.1.0-rc.19'
2026-04-17T12:44:50Z [77ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildBranch = 'HEAD'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildDate = 'Oct-03-2025'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildHash = '3a61992'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildRepo = 'gitlab-master.nvidia.com/omniverse/physics'
2026-04-17T12:44:50Z [78ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildVersion = '107.3.26'
2026-04-17T12:44:50Z [79ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] memoryStats = '(avail/total) RAM: 496.566/503.493GB, Swap: 16/16GB, VM: 1.71799e+10/1.71799e+10GB'
2026-04-17T12:44:50Z [79ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] nvidia-smi = '
==============NVSMI LOG==============
Timestamp : Fri Apr 17 12:44:37 2026
Driver Version : 595.58.03
CUDA Version : 13.2
Attached GPUs : 1
GPU 00000000:31:00.0
Product Name : NVIDIA L40S
Product Brand : NVIDIA
Product Architecture : Ada Lovelace
Display Mode : Requested functionality has been deprecated
Display Attached : Yes
Display Active : Disabled
Persistence Mode : Enabled
Addressing Mode : None
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : 1792325013960
GPU UUID : GPU-bfbe89df-81e3-6e55-f639-ebc1983b3cad
GPU PDI : 0xf6c17e3388f6ea06
Minor Number : 0
VBIOS Version : 95.02.66.00.15
MultiGPU Board : No
Board ID : 0x3100
Board Part Number : 900-2G133-0080-000
GPU Part Number : 26B9-896-A1
FRU Part Number : N/A
Platform Info
Chassis Serial Number : N/A
Slot Number : N/A
Tray Index : N/A
Host ID : N/A
Peer Type : N/A
Module Id : 1
GPU Fabric GUID : N/A
Inforom Version
Image Version : G133.0242.00.03
OEM Object : 2.1
ECC Object : 6.16
Power Management Object : N/A
Inforom BBX Object Flush
Latest Timestamp : N/A
Latest Duration : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU C2C Mode : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
vGPU Heterogeneous Mode : N/A
GPU Recovery Action : None
GSP Firmware Version : 595.58.03
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x31
Device : 0x00
Domain : 0x0000
Base Classcode : 0x3
Sub Classcode : 0x2
Device Id : 0x26B910DE
Bus Id : 00000000:31:00.0
Sub System Id : 0x185110DE
GPU Link Info
PCIe Generation
Max : 4
Current : 1
Device Current : 1
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 634 KB/s
Rx Throughput : 341 KB/s
Atomic Caps Outbound : N/A
Atomic Caps Inbound : N/A
Fan Speed : N/A
Performance State : P8
Clocks Event Reasons
Idle : Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Clocks Event Reasons Counters
SW Power Capping : 270280264 us
Sync Boost : 0 us
SW Thermal Slowdown : 0 us
HW Thermal Slowdown : 0 us
HW Power Braking : 0 us
Sparse Operation Mode : N/A
FB Memory Usage
Total : 46068 MiB
Reserved : 609 MiB
Used : 1 MiB
Free : 45460 MiB
BAR1 Memory Usage
Total : 65536 MiB
Used : 0 MiB
Free : 65536 MiB
Conf Compute Protected Memory Usage
Total : 0 MiB
Used : 0 MiB
Free : 0 MiB
Compute Mode : Default
Utilization
GPU : 0 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
DRAM Encryption Mode
Current : N/A
Pending : N/A
ECC Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
SRAM Correctable : 0
SRAM Uncorrectable Parity : 0
SRAM Uncorrectable SEC-DED : 0
DRAM Correctable : 0
DRAM Uncorrectable : 0
Aggregate
SRAM Correctable : 0
SRAM Uncorrectable Parity : 0
SRAM Uncorrectable SEC-DED : 0
DRAM Correctable : 0
DRAM Uncorrectable : 0
SRAM Threshold Exceeded : No
Aggregate Uncorrectable SRAM Sources
SRAM L2 : 0
SRAM SM : 0
SRAM Microcontroller : 0
SRAM PCIE : 0
SRAM Other : 0
Channel Repair Pending : No
TPC Repair Pending : No
Unrepairable Memory : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows
Correctable Error : 0
Uncorrectable Error : 0
Pending : No
Remapping Failure Occurred : No
Bank Remap Availability Histogram
Max : 192 bank(s)
High : 0 bank(s)
Partial : 0 bank(s)
Low : 0 bank(s)
None : 0 bank(s)
Temperature
GPU Current Temp : 40 C
GPU T.Limit Temp : 48 C
GPU Shutdown T.Limit Temp : -5 C
GPU Slowdown T.Limit Temp : -2 C
GPU Max Operating T.Limit Temp : 0 C
GPU Target Temperature : N/A
Memory Current Temp : N/A
Memory Max Operating T.Limit Temp : N/A
GPU Power Readings
Average Power Draw : 36.67 W
Instantaneous Power Draw : 36.67 W
Current Power Limit : 350.00 W
Requested Power Limit : 350.00 W
Default Power Limit : 350.00 W
Min Power Limit : 100.00 W
Max Power Limit : 350.00 W
GPU Memory Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Module Power Readings
Average Power Draw : N/A
Instantaneous Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Power Smoothing : N/A
Workload Power Profiles
Requested Profiles : N/A
Enforced Profiles : N/A
EDPp Multiplier : N/A
Clocks
Graphics : 210 MHz
SM : 210 MHz
Memory : 405 MHz
Video : 1185 MHz
Applications Clocks
Graphics : Requested functionality has been deprecated
Memory : Requested functionality has been deprecated
Default Applications Clocks
Graphics : Requested functionality has been deprecated
Memory : Requested functionality has been deprecated
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 2520 MHz
SM : 2520 MHz
Memory : 9001 MHz
Video : 1965 MHz
Max Customer Boost Clocks
Graphics : 2520 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Fabric
State : N/A
Status : N/A
CliqueId : N/A
ClusterUUID : N/A
Health
Summary : N/A
Bandwidth : N/A
Route Recovery in progress : N/A
Route Unhealthy : N/A
Access Timeout Recovery : N/A
Incorrect Configuration : N/A
Partition Assigned : N/A
Processes
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 246307
Type : G
Name : ............................................../python
Used GPU Memory : 0 MiB
Capabilities
EGM : disabled
'
2026-04-17T12:44:50Z [79ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osDistro = 'ubuntu'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osName = '22.04.5 LTS (Jammy Jellyfish)'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osVersion = '22.04.5'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] portableMode = '1'
2026-04-17T12:44:50Z [80ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] runEnvironment = 'Individual'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] runningInContainer = '0'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_debugSymbols = '0'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_dumpIncludeOverrides = '0'
2026-04-17T12:44:50Z [81ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_dumpIntermediates = '0'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_obfuscateCode = '1'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_optimizationLevel = '1'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] systemInfo = '
|---------------------------------------------------------------------------------------------|
| Driver Version: 595.58.03 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA L40S | Yes: 0 | | 46068 MB | 10de | 0 |
| | | | | | 26b9 | bfbe89df.. |
| | | | | | 31 | |
|=============================================================================================|
| OS: 22.04.5 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.5, Kernel: 5.15.0-173-generic
| Processor: Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
| Cores: 32 | Logical Cores: 64
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515576 | Free Memory: 510131
| Total Page/Swap (MB): 16383 | Free Page/Swap: 16383
|---------------------------------------------------------------------------------------------|
'
2026-04-17T12:44:50Z [82ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] telemetrySessionId = '1467242400602034763'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] terminatedByAbort = '0'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalRamBareMetalMB = '515576'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalRamLimitedMB = '515576'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalSwapBareMetalMB = '16383'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] totalSwapLimitedMB = '16383'
2026-04-17T12:44:50Z [83ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] userId = 'default'
2026-04-17T12:44:50Z [84ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] workingDirectory = '/home/<username>/IsaacLab'
2026-04-17T12:44:50Z [84ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 246307 backtrace follows:
2026-04-17T12:44:50Z [91ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: libc.so.6!__sigaction+0x50 (??:?)
2026-04-17T12:44:50Z [95ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: librtx.scenedb.plugin.so!void std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>*, std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > > >, std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>&&)+0x123ef (??:?)
2026-04-17T12:44:50Z [99ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: librtx.scenedb.plugin.so!void std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>*, std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > > >, std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>&&)+0x12766 (??:?)
2026-04-17T12:44:50Z [102ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: librtx.scenedb.plugin.so!void std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > >::_M_realloc_insert<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> >(__gnu_cxx::__normal_iterator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>*, std::vector<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>, std::allocator<std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int> > > >, std::tuple<char const*, float, float, unsigned int, unsigned int, unsigned int>&&)+0x14a14 (??:?)
2026-04-17T12:44:50Z [107ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: librtx.scenedb.plugin.so!carbOnPluginStartup+0x3b4de (??:?)
2026-04-17T12:44:50Z [111ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: librtx.scenedb.plugin.so!void std::vector<int, std::allocator<int> >::_M_realloc_insert<int>(__gnu_cxx::__normal_iterator<int*, std::vector<int, std::allocator<int> > >, int&&)+0x7ce1 (??:?)
2026-04-17T12:44:50Z [115ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x4655f (??:?)
2026-04-17T12:44:50Z [119ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libcarb.scenerenderer-rtx.plugin.so!std::unordered_map<unsigned int, unsigned int, std::hash<unsigned int>, std::equal_to<unsigned int>, std::allocator<std::pair<unsigned int const, unsigned int> > >::~unordered_map()+0x64486 (??:?)
2026-04-17T12:44:50Z [124ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: libcarb.scenerenderer-rtx.plugin.so!carbOnPluginShutdown+0xe4b (??:?)
2026-04-17T12:44:50Z [127ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: libomni.hydra.rtx.plugin.so!+0x4459
2026-04-17T12:44:50Z [131ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libomni.usd.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)+0x10da (??:?)
2026-04-17T12:44:50Z [135ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libomni.usd.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)+0x58ce (??:?)
2026-04-17T12:44:50Z [139ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libomni.usd.so!omni::usd::UsdManager::destroyHydraEngine(unsigned int)+0x3b7 (??:?)
2026-04-17T12:44:50Z [143ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libomni.usd.so!void std::vector<unsigned long, std::allocator<unsigned long> >::_M_realloc_insert<unsigned long>(__gnu_cxx::__normal_iterator<unsigned long*, std::vector<unsigned long, std::allocator<unsigned long> > >, unsigned long&&)+0x8807 (??:?)
2026-04-17T12:44:50Z [147ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libomni.usd.so!omni::usd::UsdManager::createHydraEngine(char const*, omni::usd::UsdContext*)+0x85 (??:?)
2026-04-17T12:44:50Z [151ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: _usd.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)+0x15ff4 (??:?)
2026-04-17T12:44:50Z [155ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: _usd.cpython-311-x86_64-linux-gnu.so!std::__detail::_Map_base<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, void*> >, std::__detail::_Select1st, std::equal_to<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::hash<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true>, true>::operator[](std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&)+0x9cf2 (??:?)
Additional Information
What I’ve Tried
- The hpc node doesn’t seem to have libvulkan.so.1 so tried to set LD_LIBRARY_PATH to use the bundled Vulkan loader
- Set VK_ICD_FILENAMES and __EGL_VENDOR_LIBRARY_FILENAMES to NVIDIA paths
- Ran in headless mode with display checks disabled
- Cleared shader/cache directories
- We followed the same setup procedure on a local 5090 Workstation with GUI and that works
Additional Context
- Running this on a shared HPC cluster so don’t have sudo access. The node has 4xL40s, but I am only requesting 1.
- As mentioned, thought it was a missing libvulkan.so.1 issues, I attempted to work around this by using bundled libraries and setting environment variables, but the issue persists when enabling cameras, maybe I am doing this wrong?