Dear NVIDIA team,
With every next release after 2023.1.0-hotfix.1 the rendering capabilities using the rep.orchestrator degrade.
On version 2023.1.0-hotfix.1 we could render scene on resolution of 5000x4000 with 12Gb GPU (NVIDIA GeForce RTX 3080 Ti) on Windows10. We saw that it used CPU RAM somehow when out of GPU memory. It was slow but it was working.
On 4.1.0 same scene, same PC, same GPU just crashes on 5000x4000. But it can be rendered with 4000x3000 resolution.
After we tried 4.2.0 relese it crashes even with 3000x3000.
All 3 isaac-sim versions are installed and tested simultaneously on the same PC.
We have notised that you update pytorch with isaac versions.
Also notised strange behaviour of ‘/rtx/pathtracin/ris/meshLights/’. When enabled, the scene we are trying to render (with light materials like https://omniverse-content-production.s3.us-west-2.amazonaws.com/Materials/Base/Emissives/Light_2900K.mdl) becomes completely black.
The last crash log from 4.2.0 is below.
2024-09-20 13:58:19 [3,067,135ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,136ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,136ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,137ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,137ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 70
2024-09-20 13:58:19 [3,067,138ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp Path Tracing RT. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2024-09-20 13:58:19 [3,067,138ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 7
2024-09-20 13:58:19 [3,067,139ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,140ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,178ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,179ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,179ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,179ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,180ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,181ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,181ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,181ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,182ms] [Error] [rtx.raytracing.plugin] Failed to create buffers for picking results!
2024-09-20 13:58:19 [3,067,185ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,185ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,185ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,186ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,187ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,187ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,187ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,188ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,188ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,189ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,196ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,196ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,197ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,197ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,198ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,198ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,199ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,199ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,199ms] [Error] [rtx.raytracing.plugin] Failed to create buffers for picking results!
2024-09-20 13:58:19 [3,067,204ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,204ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,205ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,206ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,227ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,228ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,235ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,235ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,236ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,237ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,237ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,238ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,238ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,239ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,239ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,240ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,252ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,253ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,253ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,254ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,254ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,255ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,281ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,282ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,283ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,283ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,284ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,284ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,284ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,285ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,285ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,286ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,298ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,299ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,299ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,300ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,300ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,301ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,314ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,315ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,319ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,319ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,320ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,320ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,320ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,321ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,321ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,322ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,332ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,332ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,332ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,333ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,333ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,334ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,348ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,348ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,357ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,358ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,358ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,358ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,359ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,359ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,360ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,360ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,367ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,367ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,368ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,368ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,368ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,369ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,383ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,383ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,384ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,384ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,384ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,385ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,385ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,385ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,386ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,386ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,400ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,401ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,401ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,401ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,401ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,402ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,416ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,416ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,419ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,419ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,420ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,420ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,421ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,421ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,421ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,422ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred. If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds. This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2024-09-20 13:58:19 [4ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] failed to write .dmp to: 'C:/Users/user/Documents/ov/data/Kit/Isaac-Sim/4.2\906a9963-065e-4c04-970a-f54649cdb7d1.dmp'
2024-09-20 13:58:20 [310ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Crash detected in pid 18224 thread 3456
2024-09-20 13:58:20 [311ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata:
2024-09-20 13:58:20 [311ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] BuildGitlabJobID = '110521995'
2024-09-20 13:58:20 [312ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] CarbSdkVersion = '173.5+release.6562.3a7379eb.gl'
2024-09-20 13:58:20 [312ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] CrashTime = ''
2024-09-20 13:58:20 [313ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] DriverShaderCacheWrapper = 'disabled'
2024-09-20 13:58:20 [313ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] DumpId = '906a9963-065e-4c04-970a-f54649cdb7d1'
2024-09-20 13:58:20 [314ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] LastUploadStatus = ''
2024-09-20 13:58:20 [314ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] ProductName = 'OmniverseKit'
2024-09-20 13:58:20 [315ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] PythonTracebackStatus = ''
2024-09-20 13:58:20 [316ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] RetryCount = '0'
2024-09-20 13:58:20 [316ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] StartupTime = '1726837632'
2024-09-20 13:58:20 [317ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UploadSuccessful = '0'
2024-09-20 13:58:20 [317ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UptimeSeconds = ''
2024-09-20 13:58:20 [318ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UserStory = ''
2024-09-20 13:58:20 [318ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] UserStoryStatus = ''
2024-09-20 13:58:20 [318ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Version = '106.1.0+release.140981.10a4b5c0.gl'
2024-09-20 13:58:20 [319ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_enabled = '1'
2024-09-20 13:58:20 [320ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_flags = '3'
2024-09-20 13:58:20 [320ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_status = 'auto-enabled'
2024-09-20 13:58:20 [320ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] aftermath_version = '2024.1'
2024-09-20 13:58:20 [321ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appName = 'Isaac-Sim'
2024-09-20 13:58:20 [321ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appState = 'started'
2024-09-20 13:58:20 [322ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] appVersion = '4.2.0'
2024-09-20 13:58:20 [322ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] autoloadExts = ''
2024-09-20 13:58:20 [323ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildBranch = 'release'
2024-09-20 13:58:20 [323ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildCi = 'gl'
2024-09-20 13:58:20 [324ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildConfig = 'release'
2024-09-20 13:58:20 [324ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabJobName = 'kit-build-release-windows-x86_64'
2024-09-20 13:58:20 [325ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabJobStage = 'kit-build'
2024-09-20 13:58:20 [325ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildGitlabPipelineID = '18272252'
2024-09-20 13:58:20 [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildHash = '10a4b5c0'
2024-09-20 13:58:20 [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildMajor = '106'
2024-09-20 13:58:20 [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildMinor = '1'
2024-09-20 13:58:20 [327ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildNumber = '140981'
2024-09-20 13:58:20 [327ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildPatch = '0'
2024-09-20 13:58:20 [329ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] buildVersion = '106.1.0'
2024-09-20 13:58:20 [329ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] carboniteFrameworkVersion = '173.5+release.6562.3a7379eb.gl'
2024-09-20 13:58:20 [330ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] carboniteSdkVersion = '173.5+release.6562.3a7379eb.gl'
2024-09-20 13:58:20 [331ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] commandLine = '"C:\Users\<username>\Documents\ov\pkg\isaac-sim-4.2.0\kit\kit.exe" "C:\Users\<username>\Documents\ov\pkg\isaac-sim-4.2.0\apps/omni.isaac.sim.kit" --ext-folder "C:\Users\<username>\Documents\ov\pkg\isaac-sim-4.2.0\/apps" '
2024-09-20 13:58:20 [332ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuId = 'AMD64 Family 25 Model 33 Stepping 0'
2024-09-20 13:58:20 [332ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuName = 'AMD Ryzen 5 5600X 6-Core Processor '
2024-09-20 13:58:20 [333ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] cpuVendor = 'AuthenticAMD'
2024-09-20 13:58:20 [333ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] desktopOrigin = '(0, 0)'
2024-09-20 13:58:20 [334ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] desktopSize = '3968x1152'
2024-09-20 13:58:20 [334ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] displayCount = '2'
2024-09-20 13:58:20 [335ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] displayRes_0 = '1920x1080x32bit@144Hz'
2024-09-20 13:58:20 [335ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] displayRes_1 = '2048x1152x32bit@60Hz'
2024-09-20 13:58:20 [336ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] email = ''
2024-09-20 13:58:20 [336ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] environmentName = 'Individual'
2024-09-20 13:58:20 [337ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] experience = 'Isaac Sim'
2024-09-20 13:58:20 [337ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] externalBuild = '1'
2024-09-20 13:58:20 [338ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] extraExts = 'optisim-1.0.0'
2024-09-20 13:58:20 [338ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuDriver_0 = '537.58'
2024-09-20 13:58:20 [339ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpuVRAM_0 = '12687769600'
2024-09-20 13:58:20 [339ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] gpu_0 = 'NVIDIA GeForce RTX 3080 Ti'
2024-09-20 13:58:20 [340ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] kitRendererDriverVersion = '537.58'
2024-09-20 13:58:20 [340ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lastCommand = 'CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?)'
2024-09-20 13:58:20 [341ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lastCommands = 'ChangeSetting(path=/rtx/pathtracing/maxSpecularAndTransmissionBounces,value=12,prev=6),ChangeSetting(path=/rtx/pathtracing/ris/meshLights,value=True),ChangeSetting(path=/rtx/pathtracing/ris/meshLights,value=False),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?)'
2024-09-20 13:58:20 [342ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildBranch = 'HEAD'
2024-09-20 13:58:20 [342ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildDate = 'Wed Sep 18 06:06:37 2024 +0000'
2024-09-20 13:58:20 [343ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildHash = 'c99988b'
2024-09-20 13:58:20 [344ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildRepo = 'gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim'
2024-09-20 13:58:20 [345ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_isaacSim_buildVersion = '4.2.0-rc.17'
2024-09-20 13:58:20 [345ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildBranch = 'HEAD'
2024-09-20 13:58:20 [346ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildDate = 'Sep-17-2024'
2024-09-20 13:58:20 [346ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildHash = '48c0995'
2024-09-20 13:58:20 [347ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildRepo = 'gitlab-master.nvidia.com/omniverse/physics'
2024-09-20 13:58:20 [347ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] lib_physx_buildVersion = '106.1.9'
2024-09-20 13:58:20 [348ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] memoryStats = ''
2024-09-20 13:58:20 [348ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] nvidia-smi = '
==============NVSMI LOG==============
Timestamp : Fri Sep 20 16:07:13 2024
Driver Version : 537.58
CUDA Version : 12.2
Attached GPUs : 1
GPU 00000000:0A:00.0
Product Name : NVIDIA GeForce RTX 3080 Ti
Product Brand : GeForce
Product Architecture : Ampere
Display Mode : Enabled
Display Active : Enabled
Persistence Mode : N/A
Addressing Mode : N/A
MIG Mode
Current : N/A
Pending : N/A
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : WDDM
Pending : WDDM
Serial Number : N/A
GPU UUID : GPU-da664789-a3cb-7a3f-3e93-3defcee6c459
Minor Number : N/A
VBIOS Version : 94.02.71.40.4c
MultiGPU Board : No
Board ID : 0xa00
Board Part Number : N/A
GPU Part Number : 2208-225-A1
FRU Part Number : N/A
Module ID : 1
Inforom Version
Image Version : G001.0000.03.03
OEM Object : 2.0
ECC Object : N/A
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GSP Firmware Version : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
GPU Reset Status
Reset Required : No
Drain and Reset Recommended : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x0A
Device : 0x00
Domain : 0x0000
Device Id : 0x220810DE
Bus Id : 00000000:0A:00.0
Sub System Id : 0x40881458
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 205000 KB/s
Rx Throughput : 2000 KB/s
Atomic Caps Inbound : N/A
Atomic Caps Outbound : N/A
Fan Speed : 0 %
Performance State : P2
Clocks Event Reasons
Idle : Not Active
Applications Clocks Setting : Not Active
SW Power Cap : Not Active
HW Slowdown : Not Active
HW Thermal Slowdown : Not Active
HW Power Brake Slowdown : Not Active
Sync Boost : Not Active
SW Thermal Slowdown : Not Active
Display Clock Setting : Not Active
FB Memory Usage
Total : 12288 MiB
Reserved : 188 MiB
Used : 981 MiB
Free : 11118 MiB
BAR1 Memory Usage
Total : 256 MiB
Used : 1 MiB
Free : 255 MiB
Conf Compute Protected Memory Usage
Total : N/A
Used : N/A
Free : N/A
Compute Mode : Default
Utilization
Gpu : 5 %
Memory : 0 %
Encoder : 0 %
Decoder : 2 %
JPEG : 0 %
OFA : 0 %
Encoder Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
FBC Stats
Active Sessions : 0
Average FPS : 0
Average Latency : 0
ECC Mode
Current : N/A
Pending : N/A
ECC Errors
Volatile
SRAM Correctable : N/A
SRAM Uncorrectable : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Aggregate
SRAM Correctable : N/A
SRAM Uncorrectable : N/A
DRAM Correctable : N/A
DRAM Uncorrectable : N/A
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows : N/A
Temperature
GPU Current Temp : 48 C
GPU T.Limit Temp : N/A
GPU Shutdown Temp : 98 C
GPU Slowdown Temp : 95 C
GPU Max Operating Temp : 93 C
GPU Target Temperature : 83 C
Memory Current Temp : N/A
Memory Max Operating Temp : N/A
GPU Power Readings
Power Draw : 117.75 W
Current Power Limit : 370.00 W
Requested Power Limit : 370.00 W
Default Power Limit : 370.00 W
Min Power Limit : 100.00 W
Max Power Limit : 370.00 W
Module Power Readings
Power Draw : N/A
Current Power Limit : N/A
Requested Power Limit : N/A
Default Power Limit : N/A
Min Power Limit : N/A
Max Power Limit : N/A
Clocks
Graphics : 1710 MHz
SM : 1710 MHz
Memory : 9251 MHz
Video : 1500 MHz
Applications Clocks
Graphics : N/A
Memory : N/A
Default Applications Clocks
Graphics : N/A
Memory : N/A
Deferred Clocks
Memory : N/A
Max Clocks
Graphics : 2160 MHz
SM : 2160 MHz
Memory : 9501 MHz
Video : 1950 MHz
Max Customer Boost Clocks
Graphics : N/A
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Voltage
Graphics : 875.000 mV
Fabric
State : N/A
Status : N/A
Processes
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 5984
Type : C+G
Name : .........................................................\Code.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 8744
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 10312
Type : C+G
Name : .............................................................\NVIDIA
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 10444
Type : C+G
Name : ..........\explorer.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 10928
Type : C+G
Name : ........................................\PowerToys.AlwaysOnTop.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 11012
Type : C+G
Name : ........................................\PowerToys.FancyZones.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 11816
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 12412
Type : C+G
Name : .......................................................\ShellExperienceHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 12676
Type : C+G
Name : ............................................................\SearchApp.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 12708
Type : C+G
Name : ..........................................\chrome.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 13364
Type : C+G
Name : ........................................\PowerToys.PowerLauncher.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 13696
Type : C+G
Name : .............................\X52_Profiler.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 14328
Type : C+G
Name : ...................................................\PowerToys.Peek.UI.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 14852
Type : C+G
Name : ...............................................................\TextInputHost.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 15980
Type : C+G
Name :
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 16756
Type : C+G
Name : ..........................................\chrome.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 18224
Type : C+G
Name : .....................................................\kit.exe
Used GPU Memory : Not available in WDDM driver model
GPU instance ID : N/A
Compute instance ID : N/A
Process ID : 19216
Type : C+G
Name : ..\Program
Used GPU Memory : Not available in WDDM driver model
'
2024-09-20 13:58:20 [353ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osDistro = '22H2'
2024-09-20 13:58:20 [354ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osName = 'Windows 10 Pro'
2024-09-20 13:58:20 [354ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] osVersion = '10.00.19045'
2024-09-20 13:58:20 [354ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] portableMode = '0'
2024-09-20 13:58:20 [355ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] primaryDisplayRes = '1920x1080x32bit@144Hz'
2024-09-20 13:58:20 [355ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] runEnvironment = 'Individual'
2024-09-20 13:58:20 [356ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] runningInContainer = '0'
2024-09-20 13:58:20 [356ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_debugSymbols = '0'
2024-09-20 13:58:20 [357ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_dumpIncludeOverrides = '0'
2024-09-20 13:58:20 [357ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_dumpIntermediates = '0'
2024-09-20 13:58:20 [357ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_obfuscateCode = '1'
2024-09-20 13:58:20 [358ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] shaderdb_optimizationLevel = '1'
2024-09-20 13:58:20 [358ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] systemInfo = '
|---------------------------------------------------------------------------------------------|
| Driver Version: 537.58 | Graphics API: D3D12
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
| | | | | | Bus-ID | |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA GeForce RTX 3080 Ti | Yes: 0 | | 12100 MB | 10de | d1df0000.. |
| | | | | | 2208 | 0 |
| | | | | | a | |
|=============================================================================================|
| OS: Windows 10 Pro, Version: 10.0 (22H2), Build: 19045, Kernel: 10.0.19041.4170
| Processor: AMD Ryzen 5 5600X 6-Core Processor | Cores: 6 | Logical: 12
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 32712 | Free Memory: 19259
| Total Page/Swap (MB): 40904 | Free Page/Swap: 18048
|---------------------------------------------------------------------------------------------|
'
2024-09-20 13:58:20 [360ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] telemetrySessionId = ''
2024-09-20 13:58:20 [361ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] terminatedByAbort = '0'
2024-09-20 13:58:20 [361ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] userId = 'default'
2024-09-20 13:58:20 [362ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] workingDirectory = ''
2024-09-20 13:58:20 [362ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 3456 backtrace follows:
2024-09-20 13:58:20 [386ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: nvrtum64.dll!+0x1127018
2024-09-20 13:58:20 [386ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: nvrtum64.dll!+0x11270f1
2024-09-20 13:58:20 [387ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: nvrtum64.dll!+0x107ac0a
2024-09-20 13:58:20 [387ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: nvrtum64.dll!+0x108a301
2024-09-20 13:58:20 [388ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: nvrtum64.dll!+0x108dd29
2024-09-20 13:58:20 [388ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: nvrtum64.dll!+0x1090129
2024-09-20 13:58:20 [389ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: nvrtum64.dll!+0x110214c
2024-09-20 13:58:20 [389ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: nvrtum64.dll!+0x1100aab
2024-09-20 13:58:20 [390ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: nvrtum64.dll!+0xed1347
2024-09-20 13:58:20 [390ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: nvrtum64.dll!+0xb4591c
2024-09-20 13:58:20 [391ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: nvrtum64.dll!+0xb44494
2024-09-20 13:58:20 [391ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: nvrtum64.dll!+0x7e280b
2024-09-20 13:58:20 [392ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: nvrtum64.dll!+0x7e2136
2024-09-20 13:58:20 [392ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: nvrtum64.dll!+0x7e7dda
2024-09-20 13:58:20 [392ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: nvrtum64.dll!+0x12079a
2024-09-20 13:58:20 [393ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: nvrtum64.dll!+0xbf933
2024-09-20 13:58:20 [393ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: nvrtum64.dll!+0xb3e1b
2024-09-20 13:58:20 [394ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: nvrtum64.dll!+0xb4532
2024-09-20 13:58:20 [413ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: nvwgf2umx.dll!+0x16828e4
2024-09-20 13:58:20 [414ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: nvwgf2umx.dll!+0x1683919
2024-09-20 13:58:20 [414ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: nvwgf2umx.dll!+0x1683e2f
2024-09-20 13:58:20 [415ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: nvwgf2umx.dll!+0x16bfa8a
2024-09-20 13:58:20 [415ms] [Fatal] [carb.crashreporter-breakpad.plugin] 022: nvwgf2umx.dll!+0x16c42cd
2024-09-20 13:58:20 [416ms] [Fatal] [carb.crashreporter-breakpad.plugin] 023: nvwgf2umx.dll!+0x16bd803
2024-09-20 13:58:20 [417ms] [Fatal] [carb.crashreporter-breakpad.plugin] 024: nvwgf2umx.dll!+0x16b764f
2024-09-20 13:58:20 [417ms] [Fatal] [carb.crashreporter-breakpad.plugin] 025: nvwgf2umx.dll!+0x1da980e
2024-09-20 13:58:20 [418ms] [Fatal] [carb.crashreporter-breakpad.plugin] 026: KERNEL32.DLL!+0x17344
2024-09-20 13:58:20 [418ms] [Fatal] [carb.crashreporter-breakpad.plugin] 027: ntdll.dll!RtlUserThreadStart+0x21
2024-09-20 13:58:20 [419ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Warning: Breakpad failed to save crash minidump. This can be caused by a failure in the child process cloned to debug this process, or a failure to attach to this process (i.e. already being debugged).