Isaac Sim Version
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):
Operating System
Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify): WSL2 Ubuntu-22.04 with Ros2 Humble
GPU Information
- Model: NVIDIA RTX A4000
- Driver Version: 552.86 (572.61)
- Cuda: 12.4 (12.8)
Topic Description
GPU crash when rendering
Detailed Description
Im trying to simulate a wheeled robot and aim to implement an autonomous control. I want to extract a target coordinate out of image processing with a stereo camera.
Originally planned with a ZEDX, (I’ve managed to get the ZED SDK installed on wsl, also sourcing successfully the zed-ros2-wrapper & zed-ros2-interfaces in my workspace after the step-by-step tutorial of stereolabs -->simulation leaded always to GPU crashes), I decided to produce any stereo image that I can proceed my task with image processing.
Yesterday, after a lot of try and error I finally managed to render and publish two rgb images and a depth-image of my stereo camera dummy with the “Ros2 camera helper” node into my ros2 workspace. I was able to see it in rviz and didn’t have any crash whilest simulating many seperate times. The last status of the day was saved. Today I couldn’t simulate once without a GPU crash.
Steps to Reproduce
- open a stage with the setup
- include any create render product node
- press play and wait for the crash
Error Messages
==============NVSMI LOG==============
Timestamp : Tue Jun 3 15:56:26 2025
Driver Version : 552.86
CUDA Version : 12.4
Attached GPUs : 1
GPU 00000000:51:00.0
Product Name : NVIDIA RTX A4000
Product Brand : NVIDIA RTX
Product Architecture : Ampere
Display Mode : Disabled
Display Active : Disabled
Persistence Mode : N/A
Addressing Mode : N/A
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : WDDM
Pending : WDDM
Serial Number : 1321023012608
GPU UUID : GPU-c9c8709f-9af0-4585-c725-c1fc8b28987e
Minor Number : N/A
VBIOS Version : 94.04.57.00.09
MultiGPU Board : No
Board ID : 0x5100
Board Part Number : 900-5G190-0100-001
GPU Part Number : 24B0-875-A1
FRU Part Number : N/A
Module ID : 1
Inforom Version
Image Version : G190.0510.00.02
OEM Object : 2.0
ECC Object : 6.16
Power Management Object : N/A
Inforom BBX Object Flush
Latest Timestamp : 2025/06/03 15:45:09.644
Latest Duration : 38408 us
GPU Operation Mode
Current : N/A
Pending : N/A
GPU C2C Mode : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
vGPU Heterogeneous Mode : N/A
GPU Reset Status
Reset Required : No
Drain and Reset Recommended : No
GSP Firmware Version : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x51
Device : 0x00
Domain : 0x0000
Device Id : 0x24B010DE
Bus Id : 00000000:51:00.0
Sub System Id : 0x14AD1028
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 139400 KB/s
Rx Throughput : 5150 KB/s
Atomic Caps Inbound : N/A
Atomic Caps Outbound : N/A
Fan Speed : 41 %
Performance State : P2
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
Sparse Operation Mode : N/A
FB Memory Usage
Total : 16376 MiB
Reserved : 207 MiB
Used : 766 MiB
Free : 15404 MiB
BAR1 Memory Usage
Total : 32768 MiB
Used : 1 MiB
Free : 32767 MiB
Conf Compute Protected Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : 13 %
Memory : 0 %
Encoder : 0 %
Decoder : 0 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
ECC Mode
Current : Disabled
Pending : Disabled
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable Parity : N/A
SRAM Uncorrectable SEC-DED : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
SRAM Threshold Exceeded : N/A
Aggregate Uncorrectable SRAM Sources
SRAM L2 : N/A
SRAM SM : N/A
SRAM Microcontroller : N/A
SRAM PCIE : N/A
SRAM Other : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows
Correctable Error : 0
Uncorrectable Error : 0
Pending : No
Remapping Failure Occurred : No
Bank Remap Availability Histogram
Max : 128 bank(s)
High : 0 bank(s)
Partial : 0 bank(s)
Low : 0 bank(s)
None : 0 bank(s)
Temperature
GPU Current Temp : 36 C
GPU T.Limit Temp : N/A
GPU Shutdown Temp : 103 C
GPU Slowdown Temp : 100 C
GPU Max Operating Temp : 98 C
GPU Target Temperature : 90 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
GPU Power Readings
Power Draw : 18.69 W
Current Power Limit : 140.00 W
Requested Power Limit : 140.00 W
Default Power Limit : 140.00 W
Min Power Limit : 100.00 W
Max Power Limit : 140.00 W
GPU Memory Power Readings
Power Draw : N/A
Module Power Readings
Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : 1560 MHz
SM : 1560 MHz
Memory : 6500 MHz
Video : 1365 MHz
Applications Clocks
Graphics : 1560 MHz
Memory : 7001 MHz
Default Applications Clocks
Graphics : 1560 MHz
Memory : 7001 MHz
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 2100 MHz
SM : 2100 MHz
Memory : 7001 MHz
Video : 1950 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Voltage
Graphics : 831.250 mV
Fabric
State : N/A
Status : N/A
CliqueId : N/A
ClusterUUID : N/A
Health
Bandwidth : N/A
Processes
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 4
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 1944
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 1976
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 5676
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 11516
Type : C+G
Name : …\WebViewHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 12492
Type : C+G
Name : …\kit.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 14052
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 14148
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 15940
Type : C+G
Name : …\OneDrive.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 16096
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 16484
Type : C+G
Name : …\ms-teams.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 16796
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 16984
Type : C+G
Name : …\StartMenuExperienceHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 18332
Type : C+G
Name : …\PhoneExperienceHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 19012
Type : C+G
Name : …\ms-teams.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 19656
Type : C+G
Name : …\explorer.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 21828
Type : C+G
Name : …\SearchHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 24720
Type : C+G
Name : …\ShellExperienceHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 25264
Type : C+G
Name : …\WindowsTerminal.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 26180
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 26684
Type : C+G
Name : …\logioptionsplus_agent.exe
Used GPU Memory : Not available in WDDM driver model
’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] cpuId: ‘AMD64 Family 25 Model 8 Stepping 2’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] cpuVendor: ‘AuthenticAMD’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] cpuPhysicalCoresBareMetal: ‘16’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] cpuLogicalCoresBareMetal: ‘32’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] cpuCoreLimited: ‘-1’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] cpuUsageQuota: ‘-1.000000’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] osName: ‘Windows 11 Enterprise’
2025-06-03 14:48:16 [3,111,761ms] [Verbose] [carb.crashreporter-breakpad.plugin] osDistro: ‘23H2’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] osVersion: ‘10.00.22631’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] primaryDisplayRes: ‘1920x1200x32bit@32Hz’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] desktopSize: ‘1920x1200’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] desktopOrigin: ‘(0, 0)’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] displayCount: ‘1’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] displayRes_0: ‘1920x1200x32bit@32Hz’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] systemInfo: ’
|---------------------------------------------------------------------------------------------|
| Driver Version: 552.86 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA RTX A4000 | Yes: 0 | | 19403 MB | 10de | 25070100.. |
| | | | | | 24b0 | c9c8709f.. |
| | | | | | 51 | |
|=============================================================================================|
| OS: Windows 11 Enterprise, Version: 10.0 (23H2), Build: 22631, Kernel: 10.0.22621.5185
| Processor: AMD Ryzen Threadripper PRO 5955WX 16-Cores
| Cores: 16 | Logical Cores: 32
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 130957 | Free Memory: 117293
| Total Page/Swap (MB): 141197 | Free Page/Swap: 126416
|---------------------------------------------------------------------------------------------|
’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] gpu_0: ‘NVIDIA RTX A4000’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] gpuVRAM_0: ‘20345520128’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] gpuDriver_0: ‘552.86’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] shaderdb_obfuscateCode: ‘1’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] shaderdb_debugSymbols: ‘0’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] shaderdb_dumpIntermediates: ‘0’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] shaderdb_dumpIncludeOverrides: ‘0’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] shaderdb_optimizationLevel: ‘1’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] kitRendererDriverVersion: ‘552.86’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_physx_buildVersion: ‘106.5.7’
2025-06-03 14:48:16 [3,111,762ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_physx_buildRepo: ‘gitlab-master.nvidia.com/omniverse/physics’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_physx_buildHash: ‘c80d589’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_physx_buildBranch: ‘HEAD’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_physx_buildDate: ‘Jan-16-2025’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_isaacSim_buildVersion: ‘4.5.0-beta.62’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_isaacSim_buildRepo: ‘https://gitlab-ci-token:glcbt-64_GypoPwsGy-_r5mdGSQpL@gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim.git’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_isaacSim_buildHash: ‘e6aa30b’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_isaacSim_buildBranch: ‘HEAD’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lib_isaacSim_buildDate: ‘Tue Jan 28 16:05:58 2025 -0800’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lastCommands: ‘Group,CreateGraphAsNodeCommand(graph_path=/Orchestrator,node_name=Orchestrator,evaluator_name=push,is_global_graph=True,backed_by_usd=True,fc_backing_type=?,pipeline_stage=?,evaluation_mode=?,graph=?),CreateNodeCommand(graph=?,node_path=/Orchestrator/OgnReadFabricTime,node_type=omni.replicator.core.ReadFabricTime,create_usd=True),CreateAttrCommand(node=?,attr_name=state:sdp_intergraph_downstream_node_handles,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateGraphAsNodeCommand(graph_path=/WriterOrchestrator,node_name=WriterOrchestrator,evaluator_name=execution,is_global_graph=True,backed_by_usd=True,fc_backing_type=?,pipeline_stage=?,evaluation_mode=?,graph=?),SetAttrCommand(attr=?,value=1,on_gpu=False,update_usd=False),CreateNodeCommand(graph=?,node_path=/Render/PostProcess/SDGPipeline/Replicator_NodeWriterWriter_01,node_type=isaacsim.ros2.bridge.ROS2PublishCameraInfo,create_usd=True),CreateNodeCommand(graph=?,node_path=/Render/PostProcess/SDGPipeline/WriterSyncGate_01,node_type=omni.graph.action.RationalTimeSyncGate,create_usd=True),SetAttrCommand(attr=?,value=1,on_gpu=False,update_usd=False),SetAttrCommand(attr=?,value=True,on_gpu=False,update_usd=False)’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] lastCommand: ‘SetAttrCommand(attr=?,value=True,on_gpu=False,update_usd=False)’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] carboniteSdkVersion: ‘180.6+release.7436.c5558fdf.gl’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] carboniteFrameworkVersion: ‘180.6+release.7436.c5558fdf.gl’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] gpuCrash: ‘1’
2025-06-03 14:48:16 [3,111,763ms] [Verbose] [carb.crashreporter-breakpad.plugin] aftermath_active_shader_count: ‘0’
2025-06-03 14:48:16 [3,111,827ms] [Warning] [gpu.foundation.plugin] successfully waited and produced aftermath dump
2025-06-03 14:48:17 [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred. If a debugger should be attached, please set the ‘/crashreporter/debuggerAttachTimeoutMs’ setting to a timeout in milliseconds. This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2025-06-03 14:48:17 [7ms] [Warning] [carb.crashreporter-breakpad.plugin] Launched process ‘py-spy.exe’ as PID 4772
2025-06-03 14:48:18 [247ms] [Error] [carb.crashreporter-breakpad.plugin] ‘py-spy.exe’ exited with code 1
2025-06-03 14:48:18 [256ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] compressing 4 files into ‘C:/Users/u29e44/AppData/Local/ov/data/Kit/Isaac-Sim Full/4.5\7bc19af1-c560-4e4a-84dc-f951306fd358.dmp.zip’ for upload.
2025-06-03 14:48:19 [1,337ms] [Info] [carb.crashreporter-breakpad.plugin] attempting to delete the file ‘C:/Users/u29e44/AppData/Local/ov/data/Kit/Isaac-Sim Full/4.5\7bc19af1-c560-4e4a-84dc-f951306fd358.dmp’ with 10 retries remaining.
2025-06-03 14:48:19 [1,338ms] [Info] [carb.crashreporter-breakpad.plugin] attempting to delete the file ‘C:/Users/u29e44/AppData/Local/ov/data/Kit/Isaac-Sim Full/4.5\7bc19af1-c560-4e4a-84dc-f951306fd358.dmp.toml’ with 10 retries remaining.
2025-06-03 14:48:19 [1,348ms] [Warning] [carb.crashreporter-breakpad.plugin] Launched process ‘crashreport.gui.exe’ as PID 28224
2025-06-03 14:48:49 [31,349ms] [Warning] [carb.crashreporter-breakpad.plugin] … waiting for PID 28224 to exit …
2025-06-03 14:49:19 [61,359ms] [Warning] [carb.crashreporter-breakpad.plugin] … waiting for PID 28224 to exit …
2025-06-03 14:49:49 [91,362ms] [Warning] [carb.crashreporter-breakpad.plugin] … waiting for PID 28224 to exit …
2025-06-03 14:50:19 [121,369ms] [Warning] [carb.crashreporter-breakpad.plugin] … waiting for PID 28224 to exit …
Screenshots or Videos
What I’ve Tried
I tried debugging with adapting the configuration of my preferences, physicsScene and lowering render settings. I searched for wrong connections in my action graph, I deleted the action graph pipeline from yesterday and included it again in the same way like I did yesterday, step by step with publishing just one image at first. As the last step I uninstalled my driver 572.61 (setup of yesterday working simulation) and downgraded for the version 552.86, wich lead to stable simulations including image publishing to my ros2 workspace for around 30seconds until it crashes again. The downgade had by far the best impact for now but still not satisfying.
I have no clue anymore. I cannot reproduce my yesterday simulation experience. Please help me.