Isaac Sim Version
4.5.0
4.2.0
4.1.0
4.0.0
4.5.0
2023.1.1
2023.1.0-hotfix.1
Other (please specify):
Operating System
Ubuntu 22.04
Ubuntu 20.04
Windows 11
Windows 10
Other (please specify):
GPU Information
- Model: NVIDIA RTX 3500 Ada
- Driver Version: 570.133.07
Topic Description
Detailed Description
I am experiencing an unexpected crash of the Isaac Sim simulation app when changing the orientation or position of an RTX lidar sensor when attempting to generate synthetic data using the replicator. I have attached an example standalone script that first instantiates an RTX lidar sensor and a cube prim and then sets up the synthetic data generation workflow and runs the replicator with the rep.orchestrator.step() function. After a few iterations of the replicator, the simulation app will encounter a seg fault and cause the simulation app to crash.
The crash seems to be happening randomly, but I have a feeling it is related to the way I am updating the sensor’s state while the replicator is running. Perhaps there is a preferred way to issue state updates while the replicator is running. Any guidance on fixes/workarounds would be much appreciated.
Steps to Reproduce
Run this simple standalone workflow script I created to reproduce the crash easily.
from isaacsim.simulation_app import SimulationApp
launch_config = {
"headless": False,
"renderer": "RayTracedLighting",
}
simulation_app = SimulationApp(launch_config)
import omni.usd
import omni.kit.commands
import omni.replicator.core as rep
from pxr.Gf import Quatd
from pxr import UsdGeom, UsdPhysics
import numpy as np
if __name__ == "__main__":
np.random.seed(8)
stage = omni.usd.get_context().get_stage()
cubeGeom = UsdGeom.Cube.Define(stage, "/World/Cube")
cubePrim = stage.GetPrimAtPath("/World/Cube")
cubeGeom.AddTranslateOp().Set((0, 0, -10))
UsdPhysics.CollisionAPI.Apply(cubePrim)
result, lidar = omni.kit.commands.execute(
"IsaacSensorCreateRtxLidar",
path = "/Lidar",
parent = "/World",
config = "Example_Rotary",
translation = (0, 0, -1.0),
orientation = Quatd(1, 0, 0, 0),
)
render_product = rep.create.render_product(lidar.GetPath(), [1, 1])
annotator = rep.AnnotatorRegistry.get_annotator("RtxSensorCpuIsaacCreateRTXLidarScanBuffer")
annotator.initialize()
annotator.attach([render_product])
debug_writer = rep.writers.get("RtxLidarDebugDrawPointCloudBuffer")
debug_writer.attach([render_product])
while simulation_app.is_running():
rep.orchestrator.step()
y = np.random.uniform(0.0, 1.0)
print(y)
stage.GetPrimAtPath("/World/Lidar").GetAttribute("xformOp:orient").Set(Quatd(1, 0, y, 0))
simulation_app.close()
Note: my use case does NOT require random updates to the orientation of the lidar, this was only used to mimic small changes to the sensor’s orientation for reproducibility. In my actual use case, the timing of the crash is unpredictable; the random state here is used to make the results deterministic.
Error Messages
Here is the stack trace for the error that is causing the crash:
Fatal Python error: Segmentation fault
Thread 0x00007f2e2e6ec640 (most recent call first):
File "/isaac-sim/kit/python/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 953 in run
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 973 in _bootstrap
Thread 0x00007f2e96ffe640 (most recent call first):
File "/isaac-sim/kit/python/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 953 in run
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 973 in _bootstrap
Thread 0x00007f303bfff640 (most recent call first):
File "/isaac-sim/kit/python/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 953 in run
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 973 in _bootstrap
Thread 0x00007f30419ff640 (most recent call first):
File "/isaac-sim/kit/python/lib/python3.10/concurrent/futures/thread.py", line 81 in _worker
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 953 in run
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
File "/isaac-sim/kit/python/lib/python3.10/threading.py", line 973 in _bootstrap
Current thread 0x00007f335c6f1b80 (most recent call first):
File "/isaac-sim/extscache/omni.syntheticdata-0.6.10+d02c707b.lx64.r.cp310/omni/syntheticdata/scripts/SyntheticData.py", line 1501 in _post_process_graph_tick
File "/isaac-sim/extscache/omni.replicator.core-1.11.35+106.5.0.lx64.r.cp310/omni/replicator/core/scripts/orchestrator.py", line 1459 in step
File "/app/python/debug.py", line 48 in <module>
Extension modules: omni.mdl.pymdlsdk._pymdlsdk, psutil._psutil_linux, psutil._psutil_posix, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, scipy._lib._ccallback_c, scipy.sparse._sparsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._isolve._iterative, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg._cythonized_array_utils, scipy.linalg._flinalg, scipy.linalg._solve_toeplitz, scipy.linalg._matfuncs_sqrtm_triu, scipy.linalg.cython_lapack, scipy.linalg.cython_blas, scipy.linalg._matfuncs_expm, scipy.linalg._decomp_update, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpack, scipy.sparse.csgraph._tools, scipy.sparse.csgraph._shortest_path, scipy.sparse.csgraph._traversal, scipy.sparse.csgraph._min_spanning_tree, scipy.sparse.csgraph._flow, scipy.sparse.csgraph._matching, scipy.sparse.csgraph._reordering, scipy.spatial._ckdtree, scipy._lib.messagestream, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial._distance_wrap, scipy.spatial._hausdorff, scipy.special._ufuncs_cxx, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.special._ellip_harm_2, scipy.spatial.transform._rotation, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, PIL._imaging, PIL._imagingft, charset_normalizer.md, requests.packages.charset_normalizer.md, requests.packages.chardet.md, osqp._osqp, scipy.io.matlab._mio_utils, scipy.io.matlab._streams, scipy.io.matlab._mio5_utils, yaml._yaml (total: 73)
/isaac-sim/python.sh: line 41: 41 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
The crash can be traced back to /isaac-sim/extscache/omni.syntheticdata-0.6.10+d02c707b.lx64.r.cp310/omni/syntheticdata/scripts/SyntheticData.py line 1501: graph.evaluate(), but I have not been able to find the implementation for this function to see exactly where things are going wrong.
Additional Information
What I’ve Tried
In my testing, these are some things I have observed:
- Same results with both headless and non-headless workflow.
- The crash can be reproduced with both the
RayTracedLightingandPathTracingrenderers. - I got the same results updating lidar state using the
BaseSensor.set_world_pose()API and the API used in the script above. - From my testing, modifying the
rt_subframesanddelta_timeparameters inside ofrep.orchestrator.step()can change when the crash occurs, but does not prevent a crash from occurring. - I have tried using
with rep.new_layer(), but this has not changed my results.