Crash

Turowicz · February 22, 2024, 9:24am

I am launching the generation with:

hardware: Ryzen 9 5900X, RTX3090, 64GB RAM, 1TB NVMe
container: nvcr.io/nvidia/isaac-sim:2023.1.1
command: /isaac-sim/runheadless.native.sh --no-window --/app/viewport/grid/enabled=false --/app/viewport/defaults/guide/grid/visible=false --/omni/replicator/replicatorYaml/yamlPath=$FILE
file: synthetic-data-examples/omni.replicator_yaml/core2.yaml at feature/custom · Surveily/synthetic-data-examples · GitHub

Mid-generation (after like 10-15 frames) I get the following crash

[23.372s] Isaac Sim Headless Native App is loaded.
Module omni.replicator.core.ogn.python._impl.nodes.OgnSemanticSegmentation load on device 'cuda:0' took 2.06 ms
malloc(): unaligned tcache chunk detected
2024-02-22 09:19:04 [0ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] A crash has occurred.  If a debugger should be attached, please set the '/crashreporter/debuggerAttachTimeoutMs' setting to a timeout in milliseconds.  This can be used to allow the crash reporter to wait for up to that long for a debugger to attach before processing or sending the crash report.
2024-02-22 09:19:04 [23ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] compressing 3 files into '/root/.local/share/ov/data/Kit/Isaac-Sim/2023.1/fbc3f9d0-9fe3-4257-33db99b8-58b80cbd.dmp.zip' for upload.
'crashreport.gui' exited with code 1272024-02-22 09:19:04 [318ms] [Error] [carb.crashreporter-breakpad.plugin] [crash] Wrote dump file '/root/.local/share/ov/data/Kit/Isaac-Sim/2023.1/fbc3f9d0-9fe3-4257-33db99b8-58b80cbd.dmp.zip'
2024-02-22 09:19:04 [319ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] preventing upload of minidump due to user opt-out: '/root/.local/share/ov/data/Kit/Isaac-Sim/2023.1/fbc3f9d0-9fe3-4257-33db99b8-58b80cbd.dmp.zip'
2024-02-22 09:19:04 [320ms] [Error] [carb.crashreporter-breakpad.plugin] [crash]     dump file size is 442884 bytes, file is readable.
2024-02-22 09:19:04 [322ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Crash detected in pid 15547 thread 15557
2024-02-22 09:19:04 [323ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash] Crash metadata:
2024-02-22 09:19:04 [324ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  CarbSdkVersion = '158.3+release158.tc9506.b2fcff1c'
2024-02-22 09:19:04 [325ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  CrashTime = 'Thu Feb 22 09:19:04 2024 GMT'
2024-02-22 09:19:04 [326ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  DriverShaderCacheWrapper = 'disabled'
2024-02-22 09:19:04 [327ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  DumpId = 'fbc3f9d0-9fe3-4257-33db99b8-58b80cbd'
2024-02-22 09:19:04 [328ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  ProductName = 'OmniverseKit'
2024-02-22 09:19:04 [329ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  PythonTracebackStatus = ''py-spy' successfully wrote info to '/root/.local/share/ov/data/Kit/Isaac-Sim/2023.1/fbc3f9d0-9fe3-4257-33db99b8-58b80cbd.py.txt''
2024-02-22 09:19:04 [329ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  RetryCount = '0'
2024-02-22 09:19:04 [330ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  StartupTime = '1708593462'
2024-02-22 09:19:04 [331ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  UploadSuccessful = '0'
2024-02-22 09:19:04 [332ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  UptimeSeconds = '82'
2024-02-22 09:19:04 [333ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  UserStory = ''
2024-02-22 09:19:04 [334ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  UserStoryStatus = ''crashreport.gui' exited with code 127'
2024-02-22 09:19:04 [335ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  Version = '105.1.2+release.133510.b82c1e1e.tc'
2024-02-22 09:19:04 [336ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_enabled = '1'
2024-02-22 09:19:04 [337ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_flags = '3'
2024-02-22 09:19:04 [338ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_status = 'auto-enabled'
2024-02-22 09:19:04 [339ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  aftermath_version = '2022.2'
2024-02-22 09:19:04 [339ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  appName = 'Isaac-Sim'
2024-02-22 09:19:04 [340ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  appState = 'started'
2024-02-22 09:19:04 [341ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  appVersion = '2023.1.1'
2024-02-22 09:19:04 [342ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  autoloadExts = ''
2024-02-22 09:19:04 [343ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildBranch = 'release'
2024-02-22 09:19:04 [344ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildCi = 'tc'
2024-02-22 09:19:04 [345ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildHash = 'b82c1e1e'
2024-02-22 09:19:04 [346ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildId = '18571592'
2024-02-22 09:19:04 [347ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildMajor = '105'
2024-02-22 09:19:04 [348ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildMinor = '1'
2024-02-22 09:19:04 [349ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildMr = '0'
2024-02-22 09:19:04 [350ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildNumber = '133510'
2024-02-22 09:19:04 [350ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildPatch = '2'
2024-02-22 09:19:04 [351ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  buildVersion = '105.1.2'
2024-02-22 09:19:04 [352ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  carboniteFrameworkVersion = '158.3+release158.tc9506.b2fcff1c'
2024-02-22 09:19:04 [353ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  carboniteSdkVersion = '158.3+release158.tc9506.b2fcff1c'
2024-02-22 09:19:04 [354ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  commandLine = '/isaac-sim/kit/kit /isaac-sim/apps/omni.isaac.sim.headless.native.kit --ext-folder /isaac-sim/apps --no-window --/persistent/isaac/asset_root/default=http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Isaac/2023.1.1 --merge-config=/isaac-sim/config/open_endpoint.toml --allow-root --no-window --/app/viewport/grid/enabled=false --/app/viewport/defaults/guide/grid/visible=false --/omni/replicator/replicatorYaml/yamlPath=/workspaces/synthetic-data-examples/omni.replicator_yaml/core2.yaml'
2024-02-22 09:19:04 [355ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  cpuId = 'Intel64 Family 25 Model 33 Stepping 2'
2024-02-22 09:19:04 [356ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  cpuName = 'AMD Ryzen 9 5900X 12-Core Processor'
2024-02-22 09:19:04 [357ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  cpuVendor = 'AuthenticAMD'
2024-02-22 09:19:04 [358ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  desktopOrigin = '(0, 0)'
2024-02-22 09:19:04 [359ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  desktopSize = '3440x1440'
2024-02-22 09:19:04 [360ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  displayCount = '1'
2024-02-22 09:19:04 [360ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  displayRes_0 = '3440x1440x32bit@100Hz'
2024-02-22 09:19:04 [361ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  environmentName = 'Individual'
2024-02-22 09:19:04 [362ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  experience = 'Isaac Sim'
2024-02-22 09:19:04 [363ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  externalBuild = '1'
2024-02-22 09:19:04 [364ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  gpuDriver_0 = '525.147'
2024-02-22 09:19:04 [365ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  gpuVRAM_0 = '26027753472'
2024-02-22 09:19:04 [366ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  gpu_0 = 'NVIDIA GeForce RTX 3090'
2024-02-22 09:19:04 [367ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  kitRendererDriverVersion = '525.147'
2024-02-22 09:19:04 [368ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  lastCommand = 'CreateNodeCommand(graph=?,node_path=/Orchestrator/OgnReadFabricTime,node_type=omni.replicator.core.ReadFabricTime,create_usd=True)'
2024-02-22 09:19:04 [369ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  lastCommands = 'CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnWriteSemantics,node_type=omni.replicator.core.OgnWriteSemantics,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/SyncGate_04,node_type=omni.graph.action.RationalTimeSyncGate,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnSampleSequencePrim,node_type=omni.replicator.core.OgnSampleSequencePrim,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnModifyAnimationTarget,node_type=omni.replicator.core.OgnModifyAnimationTarget,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/SyncGate_05,node_type=omni.graph.action.RationalTimeSyncGate,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/OgnScatter2D,node_type=omni.replicator.core.OgnScatter2D,create_usd=True),CreateNodeCommand(graph=?,node_path=/Replicator/SDGPipeline/SyncGate_06,node_type=omni.graph.action.RationalTimeSyncGate,create_usd=True),Group,CreateGraphAsNodeCommand(graph_path=/Orchestrator,node_name=Orchestrator,evaluator_name=push,is_global_graph=True,backed_by_usd=True,fc_backing_type=?,pipeline_stage=?,evaluation_mode=?,graph=?),CreateNodeCommand(graph=?,node_path=/Orchestrator/OgnReadFabricTime,node_type=omni.replicator.core.ReadFabricTime,create_usd=True)'
2024-02-22 09:19:04 [370ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  memoryStats = '(avail/total) RAM: 51.9798/62.7571GB, Swap: 8/8GB, VM: 1.71799e+10/1.71799e+10GB'
2024-02-22 09:19:04 [371ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  nvidia-smi = 'Thu Feb 22 09:17:43 2024       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.147.05   Driver Version: 525.147.05   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:09:00.0  On |                  N/A |
| 72%   69C    P2   141W / 350W |   1397MiB / 24576MiB |     16%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
'
2024-02-22 09:19:04 [372ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  osDistro = 'ubuntu'
2024-02-22 09:19:04 [373ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  osName = '22.04.3 LTS (Jammy Jellyfish)'
2024-02-22 09:19:04 [374ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  osVersion = '22.04.3'
2024-02-22 09:19:04 [375ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  portableMode = '0'
2024-02-22 09:19:04 [376ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  primaryDisplayRes = '3440x1440x32bit@100Hz'
2024-02-22 09:19:04 [377ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  runEnvironment = 'Individual'
2024-02-22 09:19:04 [378ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_debugSymbols = '0'
2024-02-22 09:19:04 [379ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_dumpIncludeOverrides = '0'
2024-02-22 09:19:04 [380ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_dumpIntermediates = '0'
2024-02-22 09:19:04 [380ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_obfuscateCode = '1'
2024-02-22 09:19:04 [381ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  shaderdb_optimizationLevel = '1'
2024-02-22 09:19:04 [382ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  systemInfo = '
|---------------------------------------------------------------------------------------------|
| Driver Version: 525.147.05    | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA GeForce RTX 3090          | Yes: 0 |     | 24822   MB | 10de      | 0          |
|     |                                  |        |     |            | 2204      | a77afd12.. |
|     |                                  |        |     |            | 9         |            |
|=============================================================================================|
| OS: 22.04.3 LTS (Jammy Jellyfish) ubuntu, Version: 22.04.3, Kernel: 6.5.0-17-generic
| XServer Vendor: The X.Org Foundation, XServer Version: 12101007 (1.21.1.7)
| Processor: AMD Ryzen 9 5900X 12-Core Processor | Cores: 12 | Logical: 24
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 64263 | Free Memory: 58952
| Total Page/Swap (MB): 8191 | Free Page/Swap: 8191
|---------------------------------------------------------------------------------------------|
'
2024-02-22 09:19:04 [383ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  telemetrySessionId = '14439070724791762453'
2024-02-22 09:19:04 [384ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  terminatedByAbort = '0'
2024-02-22 09:19:04 [385ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  userId = 'default'
2024-02-22 09:19:04 [386ms] [Warn] [carb.crashreporter-breakpad.plugin] [crash]  workingDirectory = '/workspaces/synthetic-data-examples'
2024-02-22 09:19:04 [387ms] [Fatal] [carb.crashreporter-breakpad.plugin] [crash] Thread 15557 backtrace follows:
2024-02-22 09:19:04 [396ms] [Fatal] [carb.crashreporter-breakpad.plugin] 000: libc.so.6!__sigaction+0x50
2024-02-22 09:19:04 [404ms] [Fatal] [carb.crashreporter-breakpad.plugin] 001: libc.so.6!pthread_kill+0x12c
2024-02-22 09:19:04 [412ms] [Fatal] [carb.crashreporter-breakpad.plugin] 002: libc.so.6!raise+0x16
2024-02-22 09:19:04 [420ms] [Fatal] [carb.crashreporter-breakpad.plugin] 003: libc.so.6!abort+0xd3
2024-02-22 09:19:04 [428ms] [Fatal] [carb.crashreporter-breakpad.plugin] 004: libc.so.6!__fsetlocking+0x426
2024-02-22 09:19:04 [436ms] [Fatal] [carb.crashreporter-breakpad.plugin] 005: libc.so.6!timer_settime+0x2cc
2024-02-22 09:19:04 [444ms] [Fatal] [carb.crashreporter-breakpad.plugin] 006: libc.so.6!malloc+0x33c
2024-02-22 09:19:04 [452ms] [Fatal] [carb.crashreporter-breakpad.plugin] 007: libstdc++.so.6!operator new(unsigned long)+0x1c
2024-02-22 09:19:04 [453ms] [Fatal] [carb.crashreporter-breakpad.plugin] 008: kit!_start+0x173
2024-02-22 09:19:04 [455ms] [Fatal] [carb.crashreporter-breakpad.plugin] 009: kit!std::string::_M_mutate(unsigned long, unsigned long, unsigned long)+0x53
2024-02-22 09:19:04 [459ms] [Fatal] [carb.crashreporter-breakpad.plugin] 010: libcarb.usdresolver.plugin.so!carbOnPluginStartup+0x150e
2024-02-22 09:19:04 [466ms] [Fatal] [carb.crashreporter-breakpad.plugin] 011: libcarb.datasource-omniclient.plugin.so!std::_Hashtable<unsigned long, std::pair<unsigned long const, std::thread::id>, std::allocator<std::pair<unsigned long const, std::thread::id> >, std::__detail::_Select1st, std::equal_to<unsigned long>, std::hash<unsigned long>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<false, false, true> >::_M_insert_unique_node(unsigned long, unsigned long, std::__detail::_Hash_node<std::pair<unsigned long const, std::thread::id>, false>*)+0x1c55
2024-02-22 09:19:04 [472ms] [Fatal] [carb.crashreporter-breakpad.plugin] 012: libcarb.datasource-omniclient.plugin.so!carbOnPluginPreStartup+0x26bf
2024-02-22 09:19:04 [478ms] [Fatal] [carb.crashreporter-breakpad.plugin] 013: libcarb.assets.plugin.so!std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long)+0x6b77
2024-02-22 09:19:04 [485ms] [Fatal] [carb.crashreporter-breakpad.plugin] 014: libcarb.assets.plugin.so!std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long)+0x7203
2024-02-22 09:19:04 [491ms] [Fatal] [carb.crashreporter-breakpad.plugin] 015: libcarb.assets.plugin.so!std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long)+0x7913
2024-02-22 09:19:04 [497ms] [Fatal] [carb.crashreporter-breakpad.plugin] 016: libcarb.assets.plugin.so!std::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::string const&, unsigned long, unsigned long)+0xae8d
2024-02-22 09:19:04 [504ms] [Fatal] [carb.crashreporter-breakpad.plugin] 017: libcarb.assets.plugin.so!std::__shared_count<(__gnu_cxx::_Lock_policy)2>::operator=(std::__shared_count<(__gnu_cxx::_Lock_policy)2> const&)+0x471
2024-02-22 09:19:04 [511ms] [Fatal] [carb.crashreporter-breakpad.plugin] 018: libcarb.tasking.plugin.so!char* std::string::_S_construct<char*>(char*, char*, std::allocator<char> const&, std::forward_iterator_tag)+0x2553
2024-02-22 09:19:04 [518ms] [Fatal] [carb.crashreporter-breakpad.plugin] 019: libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::*)(), carb::tasking::ThreadPool*> > >::~_State_impl()+0xb9a5
2024-02-22 09:19:04 [525ms] [Fatal] [carb.crashreporter-breakpad.plugin] 020: libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::ThreadPool::*)(), carb::tasking::ThreadPool*> > >::~_State_impl()+0xb9f9
2024-02-22 09:19:04 [532ms] [Fatal] [carb.crashreporter-breakpad.plugin] 021: libcarb.tasking.plugin.so!make_fcontext+0x2f
Aborted (core dumped)

4170e8d8-698e-43d0-9befe1b7-235a4d68.dmp.zip (427.2 KB)

Turowicz · February 22, 2024, 9:30am

cc @dennis.lynch @pcallender @wilfried.rakow

Turowicz · February 22, 2024, 10:22am

Looks like its the distant light randomization that is crashing the generation. Both angle and rotation and either one of them end up with a crash eventually.

pcallender · February 28, 2024, 4:42pm

Thanks for bringing this up. I’ll try and repro and will circle back when I have more info. Thanks for posting the full dmp.

Turowicz · February 28, 2024, 6:45pm

This currently is the biggest problem for me as I am only able to generate ~200 frames in one go. The generation crashes between frame 5 and 200.

pcallender · February 28, 2024, 6:50pm

@Turowicz thanks for the extra info. It might be a day or two before I have anything to report. If you don’t hear from me, feel free to @ me like you’ve been doing.

Turowicz · February 28, 2024, 7:59pm

Its possible this crash is also caused by backplateTexture randomization.

Turowicz · February 28, 2024, 8:15pm

OK I am 99.99% certain the cause of the crash is in fact the background randomization:

# Set the trigger as on_frame
trigger:
  trigger.on_frame:
    interval: 2
    max_execs: 600

# When the trigger executes, run the following logic
with_trigger:
  with.trigger:
    # Set background
    background:
      settings.carb_settings:
        setting: /rtx/post/backgroundZeroAlpha/backplateTexture
        value: 
          distribution.sequence:
            items: 
              - omniverse://localhost/Library/CRH/cam-06/cam6_10.48.190.166_554_20240205_132913-00.01.56.594.jpeg
              - omniverse://localhost/Library/CRH/cam-06/cam6_10.48.190.166_554_20240206_074317-00.04.00.405.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240108064113_20240108072806-00.01.44.687.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240108064113_20240108072806-00.26.35.407.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240108064113_20240108072806-00.44.06.425.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240117124752_20240117134752-00.00.00.000.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240117124752_20240117134752-00.43.49.228.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240201142421_20240201151119-00.08.55.831.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240201142421_20240201151119-00.38.38.710.jpeg
              - omniverse://localhost/Library/CRH/cam-06/CRH NVR11_ch29_20240201142421_20240201151119-00.42.59.721.jpeg

If I only have one item on that list then it works flawless.

Any ideas @pcallender ?

Turowicz · February 28, 2024, 10:08pm

OK I can now confirm 100% its the background distribution. I split my tasks into separate yamls with specific background images and they went thorugh with all other complex randomizations.

Turowicz · February 29, 2024, 8:49am

I just generated 6000 frames with this approach, zero crashes. backplateTexture distribution is definitely the cause of the problem. @pcallender

pcallender · February 29, 2024, 4:57pm

@Turowicz Nice sleuthing! I’ll file this as a ticket and get some eyes on it. Thanks for digging and finding the issue.

Turowicz · March 13, 2024, 9:28am

Any sort of workaround to fix this issue would be very much appreciated. @pcallender

dennis.lynch · May 11, 2024, 12:03am

I don’t think this is the correct way to do background randomization (using setting: /rtx/post/backgroundZeroAlpha/backplateTexture ), which is likely causing the crash

We have better “backround randomization” using dome light textures
Randomizer Examples — Omniverse Extensions latest documentation (nvidia.com)

Or using the background augmentation
Data Augmentations — Omniverse Extensions latest documentation (nvidia.com)

system · June 10, 2024, 12:05pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
I started isaac-sim and got segmentation fault Isaac Sim ubuntu , isaac-sim-v4-2-0	2	53	November 3, 2024
App is crashing after starting the simulation Isaac Sim	2	50	August 14, 2024
Isaac sim 2023.1.1 with omniverse 1.9.11 on ubuntu 22.04 failed to launch Isaac Sim	1	125	July 16, 2024
Isaac_sim crashed when I clicke "load world" in my extension Isaac Sim	2	688	May 30, 2023
Isaac Sim crashes in headless with multi-robot ROS2 navigation examples Isaac Sim	1	26	August 23, 2024
Omniverse Create is crashing while exporting to gltf/glb Samples & Examples create	2	637	October 8, 2022
Omniverse Code Computer Freezeup General Discussion	10	1415	February 23, 2023
Crash in carter_warehouse_apriltags_worker scene with deactivation of default Nova_Carter_ROS robot Isaac Sim isaac-sim-v4-2-0	0	17	December 8, 2024
Failed to launch and attach the Issac sim in Nsight Graphics Nsight Graphics	1	492	April 30, 2024
Isaac Sim crashes when importing URDF with many links Isaac Sim	5	500	August 5, 2023

Crash

Related topics