Degradation of rendering capabilities of Isaac-sim on 4.2.0 and 4.1.0

Dear NVIDIA team,

With every next release after 2023.1.0-hotfix.1 the rendering capabilities using the rep.orchestrator degrade.

On version 2023.1.0-hotfix.1 we could render scene on resolution of 5000x4000 with 12Gb GPU (NVIDIA GeForce RTX 3080 Ti) on Windows10. We saw that it used CPU RAM somehow when out of GPU memory. It was slow but it was working.

On 4.1.0 same scene, same PC, same GPU just crashes on 5000x4000. But it can be rendered with 4000x3000 resolution.

After we tried 4.2.0 relese it crashes even with 3000x3000.

All 3 isaac-sim versions are installed and tested simultaneously on the same PC.

We have notised that you update pytorch with isaac versions.

Also notised strange behaviour of ‘/rtx/pathtracin/ris/meshLights/’. When enabled, the scene we are trying to render (with light materials like https://omniverse-content-production.s3.us-west-2.amazonaws.com/Materials/Base/Emissives/Light_2900K.mdl) becomes completely black.

The last crash log from 4.2.0 is below.

2024-09-20 13:58:19 [3,067,135ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,136ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,136ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,137ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,137ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp 70
2024-09-20 13:58:19 [3,067,138ms] [Error] [gpu.foundation.plugin] Failed to update params for RenderOp Path Tracing RT. Will not execute this or subsequent RenderGraph operations. Aborting RenderGraph execution
2024-09-20 13:58:19 [3,067,138ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 7
2024-09-20 13:58:19 [3,067,139ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,140ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,178ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,179ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,179ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,179ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,180ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,181ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,181ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,181ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,182ms] [Error] [rtx.raytracing.plugin] Failed to create buffers for picking results!
2024-09-20 13:58:19 [3,067,185ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,185ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,185ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,186ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,187ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,187ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,187ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,188ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,188ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,189ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,196ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,196ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,197ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,197ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,198ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,198ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,199ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,199ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,199ms] [Error] [rtx.raytracing.plugin] Failed to create buffers for picking results!
2024-09-20 13:58:19 [3,067,204ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,204ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,205ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,206ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,227ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,228ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,235ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,235ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,236ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,237ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,237ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,238ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,238ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,239ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,239ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,240ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,252ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,253ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,253ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,254ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,254ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,255ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,281ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,282ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,283ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,283ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,284ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,284ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,284ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,285ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,285ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,286ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,298ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,299ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,299ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,300ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,300ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,301ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,314ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,315ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,319ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,319ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,320ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,320ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,320ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,321ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,321ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,322ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,332ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,332ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,332ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,333ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,333ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,334ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,348ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,348ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,357ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,358ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,358ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,358ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,359ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,359ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,360ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,360ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,367ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,367ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,368ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,368ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,368ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,369ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,383ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,383ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,384ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,384ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,384ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,385ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,385ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,385ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,386ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,386ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,400ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,401ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,401ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,401ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,401ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,402ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [3,067,416ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,416ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,419ms] [Warning] [gpu.foundation.plugin] Reaping thread purging all memory, this should only occur when the system cannot allocate more memory.
2024-09-20 13:58:19 [3,067,419ms] [Info] [gpu.foundation.plugin] There was nothing to purge from the ReapingThread for device: 0
2024-09-20 13:58:19 [3,067,420ms] [Error] [carb.graphics-direct3d.plugin] HRESULT: 0xffffffff
2024-09-20 13:58:19 [3,067,420ms] [Error] [carb.graphics-direct3d.plugin] CreateHeap failed.
2024-09-20 13:58:19 [3,067,421ms] [Error] [gpu.foundation.plugin] Unable to allocate buffer
2024-09-20 13:58:19 [3,067,421ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-20 13:58:19 [3,067,421ms] [Warning] [omni.hydra] Rendering failed.
2024-09-20 13:58:19 [3,067,422ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-20 13:58:19 [0ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred.  If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds.  This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2024-09-20 13:58:19 [4ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] failed to write .dmp to: 'C:/Users/user/Documents/ov/data/Kit/Isaac-Sim/4.2\906a9963-065e-4c04-970a-f54649cdb7d1.dmp'
2024-09-20 13:58:20 [310ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Crash detected in pid 18224 thread 3456
2024-09-20 13:58:20 [311ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata:
2024-09-20 13:58:20 [311ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  BuildGitlabJobID = '110521995'
2024-09-20 13:58:20 [312ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  CarbSdkVersion = '173.5+release.6562.3a7379eb.gl'
2024-09-20 13:58:20 [312ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  CrashTime = ''
2024-09-20 13:58:20 [313ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  DriverShaderCacheWrapper = 'disabled'
2024-09-20 13:58:20 [313ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  DumpId = '906a9963-065e-4c04-970a-f54649cdb7d1'
2024-09-20 13:58:20 [314ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  LastUploadStatus = ''
2024-09-20 13:58:20 [314ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  ProductName = 'OmniverseKit'
2024-09-20 13:58:20 [315ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  PythonTracebackStatus = ''
2024-09-20 13:58:20 [316ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  RetryCount = '0'
2024-09-20 13:58:20 [316ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  StartupTime = '1726837632'
2024-09-20 13:58:20 [317ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UploadSuccessful = '0'
2024-09-20 13:58:20 [317ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UptimeSeconds = ''
2024-09-20 13:58:20 [318ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UserStory = ''
2024-09-20 13:58:20 [318ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  UserStoryStatus = ''
2024-09-20 13:58:20 [318ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  Version = '106.1.0+release.140981.10a4b5c0.gl'
2024-09-20 13:58:20 [319ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_enabled = '1'
2024-09-20 13:58:20 [320ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_flags = '3'
2024-09-20 13:58:20 [320ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_status = 'auto-enabled'
2024-09-20 13:58:20 [320ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_version = '2024.1'
2024-09-20 13:58:20 [321ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  appName = 'Isaac-Sim'
2024-09-20 13:58:20 [321ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  appState = 'started'
2024-09-20 13:58:20 [322ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  appVersion = '4.2.0'
2024-09-20 13:58:20 [322ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  autoloadExts = ''
2024-09-20 13:58:20 [323ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildBranch = 'release'
2024-09-20 13:58:20 [323ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildCi = 'gl'
2024-09-20 13:58:20 [324ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildConfig = 'release'
2024-09-20 13:58:20 [324ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildGitlabJobName = 'kit-build-release-windows-x86_64'
2024-09-20 13:58:20 [325ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildGitlabJobStage = 'kit-build'
2024-09-20 13:58:20 [325ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildGitlabPipelineID = '18272252'
2024-09-20 13:58:20 [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildHash = '10a4b5c0'
2024-09-20 13:58:20 [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildMajor = '106'
2024-09-20 13:58:20 [326ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildMinor = '1'
2024-09-20 13:58:20 [327ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildNumber = '140981'
2024-09-20 13:58:20 [327ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildPatch = '0'
2024-09-20 13:58:20 [329ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  buildVersion = '106.1.0'
2024-09-20 13:58:20 [329ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  carboniteFrameworkVersion = '173.5+release.6562.3a7379eb.gl'
2024-09-20 13:58:20 [330ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  carboniteSdkVersion = '173.5+release.6562.3a7379eb.gl'
2024-09-20 13:58:20 [331ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  commandLine = '"C:\Users\<username>\Documents\ov\pkg\isaac-sim-4.2.0\kit\kit.exe"  "C:\Users\<username>\Documents\ov\pkg\isaac-sim-4.2.0\apps/omni.isaac.sim.kit" --ext-folder "C:\Users\<username>\Documents\ov\pkg\isaac-sim-4.2.0\/apps"  '
2024-09-20 13:58:20 [332ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuId = 'AMD64 Family 25 Model 33 Stepping 0'
2024-09-20 13:58:20 [332ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuName = 'AMD Ryzen 5 5600X 6-Core Processor             '
2024-09-20 13:58:20 [333ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  cpuVendor = 'AuthenticAMD'
2024-09-20 13:58:20 [333ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  desktopOrigin = '(0, 0)'
2024-09-20 13:58:20 [334ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  desktopSize = '3968x1152'
2024-09-20 13:58:20 [334ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  displayCount = '2'
2024-09-20 13:58:20 [335ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  displayRes_0 = '1920x1080x32bit@144Hz'
2024-09-20 13:58:20 [335ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  displayRes_1 = '2048x1152x32bit@60Hz'
2024-09-20 13:58:20 [336ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  email = ''
2024-09-20 13:58:20 [336ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  environmentName = 'Individual'
2024-09-20 13:58:20 [337ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  experience = 'Isaac Sim'
2024-09-20 13:58:20 [337ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  externalBuild = '1'
2024-09-20 13:58:20 [338ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  extraExts = 'optisim-1.0.0'
2024-09-20 13:58:20 [338ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  gpuDriver_0 = '537.58'
2024-09-20 13:58:20 [339ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  gpuVRAM_0 = '12687769600'
2024-09-20 13:58:20 [339ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  gpu_0 = 'NVIDIA GeForce RTX 3080 Ti'
2024-09-20 13:58:20 [340ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  kitRendererDriverVersion = '537.58'
2024-09-20 13:58:20 [340ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lastCommand = 'CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?)'
2024-09-20 13:58:20 [341ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lastCommands = 'ChangeSetting(path=/rtx/pathtracing/maxSpecularAndTransmissionBounces,value=12,prev=6),ChangeSetting(path=/rtx/pathtracing/ris/meshLights,value=True),ChangeSetting(path=/rtx/pathtracing/ris/meshLights,value=False),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?),CreateAttrCommand(node=?,attr_name=state:_sdp_intergraph_downstream_node_handles_,attr_type=?,attr_port=?,attr_default=?,attr_extended_type=?)'
2024-09-20 13:58:20 [342ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildBranch = 'HEAD'
2024-09-20 13:58:20 [342ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildDate = 'Wed Sep 18 06:06:37 2024 +0000'
2024-09-20 13:58:20 [343ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildHash = 'c99988b'
2024-09-20 13:58:20 [344ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildRepo = 'gitlab-master.nvidia.com/omniverse/isaac/omni_isaac_sim'
2024-09-20 13:58:20 [345ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_isaacSim_buildVersion = '4.2.0-rc.17'
2024-09-20 13:58:20 [345ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildBranch = 'HEAD'
2024-09-20 13:58:20 [346ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildDate = 'Sep-17-2024'
2024-09-20 13:58:20 [346ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildHash = '48c0995'
2024-09-20 13:58:20 [347ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildRepo = 'gitlab-master.nvidia.com/omniverse/physics'
2024-09-20 13:58:20 [347ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  lib_physx_buildVersion = '106.1.9'
2024-09-20 13:58:20 [348ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  memoryStats = ''
2024-09-20 13:58:20 [348ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  nvidia-smi = '
==============NVSMI LOG==============

Timestamp                                 : Fri Sep 20 16:07:13 2024
Driver Version                            : 537.58
CUDA Version                              : 12.2

Attached GPUs                             : 1
GPU 00000000:0A:00.0
    Product Name                          : NVIDIA GeForce RTX 3080 Ti
    Product Brand                         : GeForce
    Product Architecture                  : Ampere
    Display Mode                          : Enabled
    Display Active                        : Enabled
    Persistence Mode                      : N/A
    Addressing Mode                       : N/A
    MIG Mode
        Current                           : N/A
        Pending                           : N/A
    Accounting Mode                       : Disabled
    Accounting Mode Buffer Size           : 4000
    Driver Model
        Current                           : WDDM
        Pending                           : WDDM
    Serial Number                         : N/A
    GPU UUID                              : GPU-da664789-a3cb-7a3f-3e93-3defcee6c459
    Minor Number                          : N/A
    VBIOS Version                         : 94.02.71.40.4c
    MultiGPU Board                        : No
    Board ID                              : 0xa00
    Board Part Number                     : N/A
    GPU Part Number                       : 2208-225-A1
    FRU Part Number                       : N/A
    Module ID                             : 1
    Inforom Version
        Image Version                     : G001.0000.03.03
        OEM Object                        : 2.0
        ECC Object                        : N/A
        Power Management Object           : N/A
    GPU Operation Mode
        Current                           : N/A
        Pending                           : N/A
    GSP Firmware Version                  : N/A
    GPU Virtualization Mode
        Virtualization Mode               : None
        Host VGPU Mode                    : N/A
    GPU Reset Status
        Reset Required                    : No
        Drain and Reset Recommended       : N/A
    IBMNPU
        Relaxed Ordering Mode             : N/A
    PCI
        Bus                               : 0x0A
        Device                            : 0x00
        Domain                            : 0x0000
        Device Id                         : 0x220810DE
        Bus Id                            : 00000000:0A:00.0
        Sub System Id                     : 0x40881458
        GPU Link Info
            PCIe Generation
                Max                       : 4
                Current                   : 4
                Device Current            : 4
                Device Max                : 4
                Host Max                  : 4
            Link Width
                Max                       : 16x
                Current                   : 16x
        Bridge Chip
            Type                          : N/A
            Firmware                      : N/A
        Replays Since Reset               : 0
        Replay Number Rollovers           : 0
        Tx Throughput                     : 205000 KB/s
        Rx Throughput                     : 2000 KB/s
        Atomic Caps Inbound               : N/A
        Atomic Caps Outbound              : N/A
    Fan Speed                             : 0 %
    Performance State                     : P2
    Clocks Event Reasons
        Idle                              : Not Active
        Applications Clocks Setting       : Not Active
        SW Power Cap                      : Not Active
        HW Slowdown                       : Not Active
            HW Thermal Slowdown           : Not Active
            HW Power Brake Slowdown       : Not Active
        Sync Boost                        : Not Active
        SW Thermal Slowdown               : Not Active
        Display Clock Setting             : Not Active
    FB Memory Usage
        Total                             : 12288 MiB
        Reserved                          : 188 MiB
        Used                              : 981 MiB
        Free                              : 11118 MiB
    BAR1 Memory Usage
        Total                             : 256 MiB
        Used                              : 1 MiB
        Free                              : 255 MiB
    Conf Compute Protected Memory Usage
        Total                             : N/A
        Used                              : N/A
        Free                              : N/A
    Compute Mode                          : Default
    Utilization
        Gpu                               : 5 %
        Memory                            : 0 %
        Encoder                           : 0 %
        Decoder                           : 2 %
        JPEG                              : 0 %
        OFA                               : 0 %
    Encoder Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    FBC Stats
        Active Sessions                   : 0
        Average FPS                       : 0
        Average Latency                   : 0
    ECC Mode
        Current                           : N/A
        Pending                           : N/A
    ECC Errors
        Volatile
            SRAM Correctable              : N/A
            SRAM Uncorrectable            : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
        Aggregate
            SRAM Correctable              : N/A
            SRAM Uncorrectable            : N/A
            DRAM Correctable              : N/A
            DRAM Uncorrectable            : N/A
    Retired Pages
        Single Bit ECC                    : N/A
        Double Bit ECC                    : N/A
        Pending Page Blacklist            : N/A
    Remapped Rows                         : N/A
    Temperature
        GPU Current Temp                  : 48 C
        GPU T.Limit Temp                  : N/A
        GPU Shutdown Temp                 : 98 C
        GPU Slowdown Temp                 : 95 C
        GPU Max Operating Temp            : 93 C
        GPU Target Temperature            : 83 C
        Memory Current Temp               : N/A
        Memory Max Operating Temp         : N/A
    GPU Power Readings
        Power Draw                        : 117.75 W
        Current Power Limit               : 370.00 W
        Requested Power Limit             : 370.00 W
        Default Power Limit               : 370.00 W
        Min Power Limit                   : 100.00 W
        Max Power Limit                   : 370.00 W
    Module Power Readings
        Power Draw                        : N/A
        Current Power Limit               : N/A
        Requested Power Limit             : N/A
        Default Power Limit               : N/A
        Min Power Limit                   : N/A
        Max Power Limit                   : N/A
    Clocks
        Graphics                          : 1710 MHz
        SM                                : 1710 MHz
        Memory                            : 9251 MHz
        Video                             : 1500 MHz
    Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Default Applications Clocks
        Graphics                          : N/A
        Memory                            : N/A
    Deferred Clocks
        Memory                            : N/A
    Max Clocks
        Graphics                          : 2160 MHz
        SM                                : 2160 MHz
        Memory                            : 9501 MHz
        Video                             : 1950 MHz
    Max Customer Boost Clocks
        Graphics                          : N/A
    Clock Policy
        Auto Boost                        : N/A
        Auto Boost Default                : N/A
    Voltage
        Graphics                          : 875.000 mV
    Fabric
        State                             : N/A
        Status                            : N/A
    Processes
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 5984
            Type                          : C+G
            Name                          : .........................................................\Code.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 8744
            Type                          : C+G
            Name                          : ..\Program
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 10312
            Type                          : C+G
            Name                          : .............................................................\NVIDIA
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 10444
            Type                          : C+G
            Name                          : ..........\explorer.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 10928
            Type                          : C+G
            Name                          : ........................................\PowerToys.AlwaysOnTop.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 11012
            Type                          : C+G
            Name                          : ........................................\PowerToys.FancyZones.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 11816
            Type                          : C+G
            Name                          : ..\Program
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 12412
            Type                          : C+G
            Name                          : .......................................................\ShellExperienceHost.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 12676
            Type                          : C+G
            Name                          : ............................................................\SearchApp.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 12708
            Type                          : C+G
            Name                          : ..........................................\chrome.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 13364
            Type                          : C+G
            Name                          : ........................................\PowerToys.PowerLauncher.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 13696
            Type                          : C+G
            Name                          : .............................\X52_Profiler.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 14328
            Type                          : C+G
            Name                          : ...................................................\PowerToys.Peek.UI.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 14852
            Type                          : C+G
            Name                          : ...............................................................\TextInputHost.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 15980
            Type                          : C+G
            Name                          : 
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 16756
            Type                          : C+G
            Name                          : ..........................................\chrome.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 18224
            Type                          : C+G
            Name                          : .....................................................\kit.exe
            Used GPU Memory               : Not available in WDDM driver model
        GPU instance ID                   : N/A
        Compute instance ID               : N/A
        Process ID                        : 19216
            Type                          : C+G
            Name                          : ..\Program
            Used GPU Memory               : Not available in WDDM driver model

'
2024-09-20 13:58:20 [353ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osDistro = '22H2'
2024-09-20 13:58:20 [354ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osName = 'Windows 10 Pro'
2024-09-20 13:58:20 [354ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  osVersion = '10.00.19045'
2024-09-20 13:58:20 [354ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  portableMode = '0'
2024-09-20 13:58:20 [355ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  primaryDisplayRes = '1920x1080x32bit@144Hz'
2024-09-20 13:58:20 [355ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  runEnvironment = 'Individual'
2024-09-20 13:58:20 [356ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  runningInContainer = '0'
2024-09-20 13:58:20 [356ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_debugSymbols = '0'
2024-09-20 13:58:20 [357ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_dumpIncludeOverrides = '0'
2024-09-20 13:58:20 [357ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_dumpIntermediates = '0'
2024-09-20 13:58:20 [357ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_obfuscateCode = '1'
2024-09-20 13:58:20 [358ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_optimizationLevel = '1'
2024-09-20 13:58:20 [358ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  systemInfo = '
|---------------------------------------------------------------------------------------------|
| Driver Version: 537.58        | Graphics API: D3D12
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 3080 Ti       | Yes: 0 |     | 12100   MB | 10de      | d1df0000.. |
|     |                                  |        |     |            | 2208      | 0          |
|     |                                  |        |     |            | a         |            |
|=============================================================================================|
| OS: Windows 10 Pro, Version: 10.0 (22H2), Build: 19045, Kernel: 10.0.19041.4170
| Processor: AMD Ryzen 5 5600X 6-Core Processor              | Cores: 6 | Logical: 12
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 32712 | Free Memory: 19259
| Total Page/Swap (MB): 40904 | Free Page/Swap: 18048
|---------------------------------------------------------------------------------------------|
'
2024-09-20 13:58:20 [360ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  telemetrySessionId = ''
2024-09-20 13:58:20 [361ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  terminatedByAbort = '0'
2024-09-20 13:58:20 [361ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  userId = 'default'
2024-09-20 13:58:20 [362ms] [Warning] [carb.crashreporter-breakpad.plugin] [crash]  workingDirectory = ''
2024-09-20 13:58:20 [362ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 3456 backtrace follows:
2024-09-20 13:58:20 [386ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: nvrtum64.dll!+0x1127018
2024-09-20 13:58:20 [386ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: nvrtum64.dll!+0x11270f1
2024-09-20 13:58:20 [387ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: nvrtum64.dll!+0x107ac0a
2024-09-20 13:58:20 [387ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: nvrtum64.dll!+0x108a301
2024-09-20 13:58:20 [388ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: nvrtum64.dll!+0x108dd29
2024-09-20 13:58:20 [388ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: nvrtum64.dll!+0x1090129
2024-09-20 13:58:20 [389ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: nvrtum64.dll!+0x110214c
2024-09-20 13:58:20 [389ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: nvrtum64.dll!+0x1100aab
2024-09-20 13:58:20 [390ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: nvrtum64.dll!+0xed1347
2024-09-20 13:58:20 [390ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: nvrtum64.dll!+0xb4591c
2024-09-20 13:58:20 [391ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: nvrtum64.dll!+0xb44494
2024-09-20 13:58:20 [391ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: nvrtum64.dll!+0x7e280b
2024-09-20 13:58:20 [392ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: nvrtum64.dll!+0x7e2136
2024-09-20 13:58:20 [392ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: nvrtum64.dll!+0x7e7dda
2024-09-20 13:58:20 [392ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: nvrtum64.dll!+0x12079a
2024-09-20 13:58:20 [393ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: nvrtum64.dll!+0xbf933
2024-09-20 13:58:20 [393ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: nvrtum64.dll!+0xb3e1b
2024-09-20 13:58:20 [394ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: nvrtum64.dll!+0xb4532
2024-09-20 13:58:20 [413ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: nvwgf2umx.dll!+0x16828e4
2024-09-20 13:58:20 [414ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: nvwgf2umx.dll!+0x1683919
2024-09-20 13:58:20 [414ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: nvwgf2umx.dll!+0x1683e2f
2024-09-20 13:58:20 [415ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: nvwgf2umx.dll!+0x16bfa8a
2024-09-20 13:58:20 [415ms] [Fatal] [carb.crashreporter-breakpad.plugin] 022: nvwgf2umx.dll!+0x16c42cd
2024-09-20 13:58:20 [416ms] [Fatal] [carb.crashreporter-breakpad.plugin] 023: nvwgf2umx.dll!+0x16bd803
2024-09-20 13:58:20 [417ms] [Fatal] [carb.crashreporter-breakpad.plugin] 024: nvwgf2umx.dll!+0x16b764f
2024-09-20 13:58:20 [417ms] [Fatal] [carb.crashreporter-breakpad.plugin] 025: nvwgf2umx.dll!+0x1da980e
2024-09-20 13:58:20 [418ms] [Fatal] [carb.crashreporter-breakpad.plugin] 026: KERNEL32.DLL!+0x17344
2024-09-20 13:58:20 [418ms] [Fatal] [carb.crashreporter-breakpad.plugin] 027: ntdll.dll!RtlUserThreadStart+0x21
2024-09-20 13:58:20 [419ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Warning: Breakpad failed to save crash minidump. This can be caused by a failure in the child process cloned to debug this process, or a failure to attach to this process (i.e. already being debugged).

I am seeing the same. My scripts that work fine on 4.1 and crashes on 4.2 with memory errors. The crash happens in the window app, not in the headless docker container. I will be running in headless docker extensively to confirm if it is fine. I am running replicator workflows.

[10.754s] app ready
[11.415s] Isaac Sim App is loaded.
2024-09-24 13:30:15 [43,228ms] [Warning] [omni.usd] Unexpected reference count of 2 for UsdStage 'anon:0x1ab9c2c0:World0.usd' while being closed in UsdContext (this may indicate it is still resident in memory).
2024-09-24 13:31:17 [105,944ms] [Warning] [omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.
2024-09-24 13:31:22 [110,479ms] [Error] [carb.cudainterop.plugin] dup failed for resourceType: 3 (file descriptor: 1023)
  Too many open files (error: 24)
2024-09-24 13:31:22 [110,479ms] [Error] [gpu.foundation.plugin] Cannot create cuda external memory for resource!
2024-09-24 13:31:22 [110,479ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,479ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,479ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,479ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,480ms] [Error] [gpu.foundation.plugin] Texture creation failed for the device: 0.
2024-09-24 13:31:22 [110,480ms] [Warning] [omni.syntheticdata.plugin] SdPostRenderVarToHost : invalid input resource for renderVar SemanticOcclusionSD
2024-09-24 13:31:22 [110,480ms] [Error] [omni.syntheticdata.plugin] SdPostRenderVarTextureToBuffer : corrupted input renderVar InstanceSegmentationReductionSD
2024-09-24 13:31:22 [110,482ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,482ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,482ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,482ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,483ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,483ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,483ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,483ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,483ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,483ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,483ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,483ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,484ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,484ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,484ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,484ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,484ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,484ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,484ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,484ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,487ms] [Warning] [omni.replicator.core.plugin] InstanceSegmentation : missing RenderVar InstanceMapSD
2024-09-24 13:31:22 [110,487ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,487ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,487ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,487ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,487ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,487ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,487ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,487ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,487ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,487ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,487ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,488ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,488ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,488ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,488ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,488ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,488ms] [Warning] [omni.replicator.core.plugin] SemanticOcclusionReduction : missing RenderVar InstanceMapSD
2024-09-24 13:31:22 [110,488ms] [Warning] [omni.syntheticdata.plugin] SdPostRenderVarTextureToBuffer missing valid input renderVar InstanceSegmentationReductionSD
2024-09-24 13:31:22 [110,488ms] [Warning] [omni.syntheticdata.plugin] SdPostRenderVarToHost missing valid input renderVar SemanticOcclusionSD
2024-09-24 13:31:22 [110,488ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,488ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,488ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,488ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,489ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,489ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,489ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,489ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,489ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,489ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,489ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,489ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,489ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,489ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,489ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,489ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,490ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,490ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,490ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,490ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,494ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,494ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,494ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,494ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,494ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,494ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,494ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,494ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,494ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,494ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,494ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,495ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,495ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,495ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,495ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,495ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,497ms] [Error] [carb.cudainterop.plugin] dup failed for resourceType: 3 (file descriptor: 954)
  Too many open files (error: 24)
2024-09-24 13:31:22 [110,497ms] [Error] [gpu.foundation.plugin] Cannot create cuda external memory for resource!
2024-09-24 13:31:22 [110,497ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,498ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,498ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,500ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,500ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,501ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,501ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,501ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,501ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,501ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,501ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,501ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,502ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,502ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,502ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,502ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,502ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,502ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,502ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,502ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,502ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,502ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,502ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,502ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,503ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,503ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,503ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,503ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,503ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,503ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,503ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,503ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,503ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,503ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,503ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,503ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,504ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,504ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,504ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,504ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,504ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,504ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,504ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,504ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,504ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,504ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,504ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,504ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,505ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,505ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,505ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,505ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,505ms] [Error] [carb.graphics-vulkan.plugin] VkResult: ERROR_INITIALIZATION_FAILED
2024-09-24 13:31:22 [110,505ms] [Error] [carb.graphics-vulkan.plugin] vkGetMemoryFdKHR failed.
2024-09-24 13:31:22 [110,505ms] [Error] [gpu.foundation.plugin] Cannot create shared handle for resource!
2024-09-24 13:31:22 [110,505ms] [Error] [gpu.foundation.plugin] Buffer creation failed for the device: 0.
2024-09-24 13:31:22 [110,539ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2024-09-24 13:31:22 [110,539ms] [Error] [carb.cudainterop.plugin] Failed to signal external semaphore in CUDA.
2024-09-24 13:31:22 [110,539ms] [Error] [gpu.foundation.plugin] Signal external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 2)"
2024-09-24 13:31:22 [110,564ms] [Error] [carb.cudainterop.plugin] CUDA error 700: cudaErrorIllegalAddress - an illegal memory access was encountered)
2024-09-24 13:31:22 [110,564ms] [Error] [carb.cudainterop.plugin] Failed to wait on external semaphore in CUDA.
2024-09-24 13:31:22 [110,565ms] [Error] [gpu.foundation.plugin] Wait for external semaphore failed in CUDA submission in command list "Render graph command list (Render queue 0, device 0, frame submission index 0)"
2024-09-24 13:31:22 [110,566ms] [Error] [carb.scenerenderer-rtx.plugin] Failed to execute RenderGraph on device 0. Error Code: 1
2024-09-24 13:31:22 [110,566ms] [Warning] [omni.hydra] Rendering failed.
2024-09-24 13:31:22 [110,566ms] [Error] [omni.usd] HydraEngine::render failed to end the compute graph: error code 6
2024-09-24 13:31:22 [111,010ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,010ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,010ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,010ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,010ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,010ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002d742ecd6927d29ebfc974a2ab760b8130c35e5a468ada4ed92feddd9269b
2024-09-24 13:31:22 [111,011ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002b541c1cd4b60c9307ba28aa7d3204c9eb5235e0fab0d7c996e414bde1da9
2024-09-24 13:31:22 [111,011ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,011ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002903348b4d1029eec446c6590a864a3a008ef22a68e0be50e4e59de312a61
2024-09-24 13:31:22 [111,011ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00021243dc82abc1b3e7c2aae400953dc7b37893e934e528e1f4d148e516d449
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00025e4152f525d16ddcd913970ba7476c55a4afa5a8c4d5ff5223efe9635d40
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002d54822295a901dfcdc7a7fc2d378c0f7da8ac054a20ad9b7a48c0fad0ff8
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002733978048b4d2500ad9fe1902a8a20497f27879687b8ad78d7575e7e87e1
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00022d381ab9f2d028798c5114bd24fa6c28a9f59436b299cfd5966e569047ae
2024-09-24 13:31:22 [111,011ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00024639ea220b98852f00f74cc5271bd7c2aadf895edaca647d587697ef6a58
2024-09-24 13:31:22 [111,042ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,042ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,042ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00021c3d3ccd3d610f2a508e31db3f7cba664d4c2c3eb0468f935c0bfe0f27b3
2024-09-24 13:31:22 [111,042ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00026a3f5696a3cdc183d63d6e49e4b2d5dc1ab8170f117356b007b2f46a13f7
2024-09-24 13:31:22 [111,042ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,042ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002a6399a2ff26ecb660dca3b4e3394efb1a9d3793167f791bda3047d6c17b7
2024-09-24 13:31:22 [111,068ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,068ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,068ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:22 [111,068ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00028d3fec46e63e93808db556e67d6a921f9620460bb7f8b7c982ba56f4bb2c
2024-09-24 13:31:22 [111,068ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002a140572bbb67ce8b3167ec027c5fcf5ece2d9ac0e4f9e6a56d25d4360f3f
2024-09-24 13:31:22 [111,068ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00028a3564c3331225a09bda63b27b60be94d2a9e524bdcc2a1a73d813863c27
2024-09-24 13:31:23 [111,511ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,511ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,511ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00020c3dadc06337908898a38e1c77b88b0df8b8dfcc01da80fbbfe992fd740b
2024-09-24 13:31:23 [111,511ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,511ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,511ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002043cf7bd7e78aa24a96256c43dc504d56cfe6be9e54dd983bf0d81e5ef8d
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00029b2df608d549d105db9e4b5b049bf78d839d427234657632789bf6eb3a65
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002593d1e720db58b44116464aa0f1b03e5350f6f00f1a958d5aeecb328fb5f
2024-09-24 13:31:23 [111,512ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,512ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,512ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002be32812eaa28de7789931ed2be683af3ed218ea265e81a9561b79b00acbf
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00028042a50349ce3735e06fe180d350362d7325767864b29ec36353ae8adb97
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002ee34da2b695736b86440e488363a1f0927841cfb8a01c8e3266cf8304e33
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00029d3b269e0e0cddf144333941f99e732b3c8dab9208b713e8e44e367a729c
2024-09-24 13:31:23 [111,512ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002e93443c9af914d18b4c29f857e5b415d5b589a69eeb35460fabdc2e3ef0a
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,512ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,542ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,542ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,542ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00021734235441148f2c537ff0f0cf270c4c5f819b3c84eb28aa522e52ed033b
2024-09-24 13:31:23 [111,542ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,542ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002f33926c89447e10cec3c746adb6386470c556ecdd870609c5baab544bd3d
2024-09-24 13:31:23 [111,542ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002af367028af0f051c53845ff09b2f0ce885f7c26933c5caa0272c14057a9b
2024-09-24 13:31:23 [111,542ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,543ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,543ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,568ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,568ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0
2024-09-24 13:31:23 [111,568ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 0002183c6f5a55e640f675fdd8a2cc7efb589bee57afcd1321fcf2db9f7b6edf
2024-09-24 13:31:23 [111,569ms] [Error] [omni.physx.cooking.plugin] MeshCookingContext::postbuild: triangulation data request failed.
2024-09-24 13:31:23 [111,569ms] [Error] [omni.datastore] LocalDataStore : Unable to load block 00022339025541a9b8159a9225cbf93c41e076aaf44ed05186aca588f754a7f9
2024-09-24 13:31:23 [111,569ms] [Warning] [omni.datastore] LocalDataStore : Failed to open data bucket file for read: /home/wojtek/.cache/ov/Kit/106.1/10a4b5c0/PhysXLocalMeshCache/LocalDS0/largecachedata_0

cc @VickNV

Thank you for bringing this issue to our attention. We’ll provide an update as soon as we have more information or a solution.

@asek.s, @Turowicz could you post the full log of a run that lead to the memory error? Any additional info on the scene and configuration (render mode, SPP (if applicable)) would also be helpful in trying to narrow down the cause. We definitely don’t expect capability to degrade with new releases so thank you so much for bringing this to our attention.

1 Like

@asek.s @Turowicz Are you using one of the usd scenes provided with Isaac Sim?
If so can you provide its path?

Yes we are based on the following scenes:

http://omniverse-content-production.s3-us-west-2.amazonaws.com/Samples/Showcases/2023_2_1/IsaacWarehouse/IsaacWarehouse.usd
http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/4.2/Isaac/Environments/Simple_Warehouse/full_warehouse.usd
http://omniverse-content-production.s3-us-west-2.amazonaws.com/Demos/AEC/TowerDemo/CityTowerDemopack/World_CityTowerDemopack.usd

On my work laptop with RTX3060 with Isaac 4.1 (Ubuntu 24.04) I was able to use the Isaac Window Application with 6 cameras on scene (+ 1 viewport) linked to the BasicWriter. On 4.2 that is not possible it gets stuck forever in loading part. I was forced to introduce debug logic for loading the scene up with only the viewport. I run SDG in containers, so those 6 cameras are not needed anyway when using the window app because I use it for debugging the scripts only. But it does show a serious degradation.

Edit: with an exception for World_CityTowerDemopack.usd where I was only able to run 2 extra cameras.

Unfortunately I am unable to provide a script at the moment.

I am also having the similar issue. Also, How can I clear this warning? Basically, how can I copy to the device buffer first?

[omni.syntheticdata.plugin] OgnSdPostRenderVarToHost : rendervar copy from texture directly to host buffer is counter-performant. Please use copy from texture to device buffer first.