Isaac Sim loaded fine for the first time, but if I exit and relaunch, it will freeze at “[ext: omni.kit.window.viewport-0.0.0] startup”. I have tried the headless and non-headless.
I am on Ubuntu 20.04, Driver Version: 525.78.01, Cuda 11.3 & tested with Cuda 11.8, GPU 4x NVIDIA RTX A550, CPU: AMD
Hi. Please share your log files.
Are you running the container or from the Launcher?
Hi,
No, it is not running in the container. It is loaded using the launcher.
[3.624s] [ext: omni.kit.window.viewport-0.0.0] startup
2023-02-02 03:46:11 [3,616ms] [Error] [carb] Failed to create change watch for `/home/abc/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extsPhysics/omni.localcache`: errno=28/No space left on device
2023-02-02 03:46:11 [3,616ms] [Error] [carb] Failed to create change watch for `/home/abc/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extsPhysics/omni.localcache/config`: errno=28/No space left on device
2023-02-02 03:46:11 [3,616ms] [Error] [carb] Failed to create change watch for `/home/abc/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extsPhysics/omni.localcache/omni`: errno=28/No space left on device
2023-02-02 03:46:11 [3,616ms] [Error] [carb] Failed to create change watch for `/home/abc/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extsPhysics/omni.localcache/omni/localcache`: errno=28/No space left on device
2023-02-02 03:46:11 [3,616ms] [Error] [carb] Failed to create change watch for `/home/abc/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extsPhysics/omni.localcache/omni/localcache/scripts`: errno=28/No space left on device
2023-02-02 03:46:11 [3,617ms] [Error] [carb] Failed to create change watch for `/home/abc/.local/share/ov/pkg/isaac_sim-2022.2.0/kit/extsPhysics/omni.localcache/omni/localcache/bindings`: errno=28/No space left on device
Try rebooting the machine and re-launcher Isaac Sim.
See also:
https://docs.omniverse.nvidia.com/app_isaacsim/prod_kit/linux-troubleshooting.html#q11-getting-spam-of-failures-failed-to-create-change-watch-for-xxx-errno-28-no-space-left-on-device
Thanks, following Q11 solved the problem.
Hi,
The solutiuon you mentioned solved that problem but it causes another problem in running the Replicator composer. I keep getting this error./python.sh: line 46: 128695 Segmentation fault (core dumped) $python_exe "$@" $args There was an error running python
@Sheikh_Dawood
Can you please address this issue? It messed up most scripts that I executed using ./python. Undoing the settings does not help at all, and the core dump error has become persistent.
Can you run any default standalone python example?
Please share a log. See Python Environment — Omniverse Robotics documentation
I am trying to use Replicator Composer offline and got the core dump error, Also I tried this:
~/.local/share/ov/pkg/isaac_sim-2022.1.0$ ./python.sh standalone_examples/api/omni.kit.app/app_framework.py
[Warning] [omni.kit.app.plugin] No crash reporter present, dumps uploading isn't available.
Loading user config located at: '/home/abc/.local/share/ov/data/Kit/omni.app.mini-hydra/0.1/user.config.json'
[Info] [carb] Logging to file: /home/abc/.nvidia-omniverse/logs/Kit/omni.app.mini-hydra/0.1/kit_20230206_093629.log
[0.047s] [ext: omni.usd.config-1.0.0] startup
[0.068s] [ext: omni.usd.libs-1.0.0] startup
[0.174s] [ext: omni.kit.async_engine-0.0.0] startup
[0.177s] [ext: omni.assets.plugins-0.0.0] startup
[0.177s] [ext: omni.client-0.1.0] startup
[0.184s] [ext: omni.kit.registry.nucleus-0.0.0] startup
2023-02-06 17:36:30 [567ms] [Warning] [omni.client.plugin] Tick: authentication: Could not resolve host name "kit-extensions.ov.nvidia.com"
2023-02-06 17:36:30 [885ms] [Warning] [omni.client.plugin] Tick: authentication: Could not resolve host name "kit-extensions.ov.nvidia.com"
2023-02-06 17:36:30 [1,169ms] [Warning] [omni.client.plugin] Tick: authentication: Could not resolve host name "kit-extensions.ov.nvidia.com"
2023-02-06 17:36:31 [1,426ms] [Error] [omni.ext.plugin] Failed to resolve extension dependencies. Failure hints:
[omni.app.mini-hydra-0.1.0] dependency: (name: 'omni.example.hydra_texture', tag: '', version: '', exact: 0, optional: 0) can't be satisfied. Available versions:
(none found)
2023-02-06 17:36:31 [1,426ms] [Error] [omni.ext.plugin] Exiting app because of dependency solver failure...
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]: Traceback (most recent call last):
Traceback (most recent call last):
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]: File "standalone_examples/api/omni.kit.app/app_framework.py", line 45, in <module>
File "standalone_examples/api/omni.kit.app/app_framework.py", line 45, in <module>
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]:
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]: stage_task = asyncio.ensure_future(omni.usd.get_context().new_stage_async())
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]:
stage_task = asyncio.ensure_future(omni.usd.get_context().new_stage_async())
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]: AttributeError
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]: :
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]: 'NoneType' object has no attribute 'new_stage_async'
2023-02-06 17:36:31 [1,469ms] [Error] [omni.kit.app.impl] [py stderr]:
AttributeError: 'NoneType' object has no attribute 'new_stage_async'
./python.sh: line 46: 160322 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
Also, here you are the core dump error on Replicator composer. I have tried with/without this fix
[2.554s] [ext: omni.kit.pipapi-0.0.0] startup
[2.567s] [ext: omni.kit.pip_archive-0.0.0] startup
[2.578s] [ext: omni.isaac.core_archive-0.3.0] startup
[2.715s] [ext: omni.usd.config-1.0.0] startup
[2.717s] [ext: omni.usd.libs-1.0.0] startup
[2.832s] [ext: omni.isaac.ml_archive-0.1.0] startup
[2.885s] [ext: omni.kit.loop-isaac-0.1.0] startup
[2.886s] [ext: omni.kit.async_engine-0.0.0] startup
[2.887s] [ext: omni.appwindow-1.0.0] startup
[2.889s] [ext: omni.client-0.1.0] startup
[2.896s] [ext: omni.kit.test-0.0.0] startup
[2.929s] [ext: omni.kit.renderer.core-0.0.0] startup
[2.980s] [ext: omni.ui-2.10.3] startup
[2.993s] [ext: carb.audio-0.1.0] startup
[3.013s] [ext: omni.kit.mainwindow-0.0.0] startup
[3.016s] [ext: omni.uiaudio-1.0.0] startup
[3.017s] [ext: omni.kit.uiapp-0.0.0] startup
[3.017s] [ext: omni.usd.schema.physics-1.0.0] startup
[3.050s] [ext: omni.usd.schema.audio-0.0.0] startup
[3.057s] [ext: omni.usd.schema.semantics-0.0.0] startup
[3.067s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[3.074s] [ext: omni.usd.schema.anim-0.0.0] startup
[3.105s] [ext: omni.kit.commands-1.2.2] startup
[3.109s] [ext: omni.timeline-1.0.2] startup
[3.111s] [ext: omni.hydra.scene_delegate-0.2.0] startup
[3.116s] [ext: omni.kit.audiodeviceenum-1.0.0] startup
[3.117s] [ext: omni.usd-1.5.3] startup
[3.364s] [ext: omni.kit.asset_converter-1.2.30] startup
[3.378s] [ext: omni.usd.schema.isaac-0.2.0] startup
[3.410s] [ext: omni.usd.schema.physx-0.0.0] startup
[3.431s] [ext: omni.kit.search_core-1.0.2] startup
[3.433s] [ext: omni.renderer-rtx-0.0.0] startup
[3.433s] [ext: omni.kit.widget.graph-1.4.2] startup
[3.440s] [ext: omni.kit.widget.filebrowser-2.2.26] startup
[3.444s] [ext: omni.kit.window.popup_dialog-2.0.7] startup
[3.448s] [ext: omni.mdl.neuraylib-0.1.0] startup
[3.451s] [ext: omni.kit.widget.path_field-2.0.3] startup
[3.452s] [ext: omni.kit.widget.versioning-1.3.8] startup
[3.453s] [ext: omni.kit.notification_manager-1.0.5] startup
[3.458s] [ext: omni.kit.widget.browser_bar-2.0.3] startup
[3.460s] [ext: omni.kit.menu.utils-1.2.11] startup
[3.468s] [ext: omni.kit.window.filepicker-2.4.29] startup
OmniAssetFileFormat
[3.535s] [ext: omni.mdl-0.1.0] startup
[3.550s] [ext: omni.kit.menu.create-1.0.2] startup
[3.551s] [ext: omni.kit.window.file_exporter-1.0.4] startup
[3.552s] [ext: omni.kit.window.drop_support-1.0.0] startup
[3.552s] [ext: omni.kit.material.library-1.3.10] startup
[3.554s] [ext: omni.kit.window.property-1.6.3] startup
[3.555s] [ext: omni.kit.context_menu-1.3.9] startup
[3.558s] [ext: omni.kit.window.file_importer-1.0.4] startup
[3.559s] [ext: omni.kit.stage_templates-1.1.2] startup
[3.561s] [ext: omni.kit.widget.stage-2.6.15] startup
[3.563s] [ext: omni.kit.window.file-1.3.16] startup
[3.565s] [ext: omni.debugdraw-0.1.0] startup
[3.569s] [ext: omni.kit.window.content_browser-2.4.28] startup
[3.584s] [ext: omni.kit.widget.prompt-1.0.1] startup
[3.584s] [ext: omni.kit.property.usd-3.14.8] startup
[3.608s] [ext: omni.hydra.engine.stats-1.0.0] startup
[3.612s] [ext: omni.kit.widget.settings-1.0.0] startup
[3.613s] [ext: omni.graph.tools-1.3.5] startup
[3.633s] [ext: omni.graph.core-2.27.0] startup
[3.635s] [ext: omni.hydra.rtx-0.1.0] startup
[3.642s] [ext: omni.kit.viewport.legacy_gizmos-1.0.0] startup
[3.644s] [ext: omni.ui_query-1.1.1] startup
[3.646s] [ext: omni.graph-1.22.1] startup
[3.687s] [ext: omni.kit.window.viewport-0.0.0] startup
2023-02-06 18:02:18 [3,925ms] [Error] [rtx.neuraylib.plugin] [DYNLIB:IO] 0.1 DYNLIB io error: /home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/bin/libs/iray/libnvindex.so: cannot open shared object file: No such file or directory
2023-02-06 18:02:18 [3,925ms] [Error] [rtx.neuraylib.plugin] [INDEX:MAIN] 0.1 INDEX main error: Failed to load /home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/python/bin/libs/iray/libnvindex.so
[4.722s] [ext: omni.kit.window.preferences-1.2.1] startup
[4.748s] [ext: omni.kit.ui_test-1.2.0] startup
[4.750s] [ext: omni.graph.ui-1.6.1] startup
[4.767s] [ext: omni.kvdb-0.0.0] startup
[4.769s] [ext: omni.kit.widget.searchfield-1.0.6] startup
[4.770s] [ext: omni.convexdecomposition-1.4.12] startup
[4.773s] [ext: omni.graph.action-1.17.0] startup
[4.782s] [ext: omni.localcache-0.0.0] startup
[4.784s] [ext: omni.usdphysics-1.4.12] startup
[4.786s] [ext: omni.graph.scriptnode-0.5.0] startup
[4.788s] [ext: omni.physx-1.4.12-5.1] startup
[4.799s] [ext: omni.kit.usd_undo-0.1.0] startup
[4.800s] [ext: omni.graph.nodes-1.25.0] startup
[4.853s] [ext: omni.physx.commands-1.4.12-5.1] startup
[4.857s] [ext: omni.syntheticdata-0.2.1] startup
[4.888s] [ext: omni.physx.ui-1.4.12-5.1] startup
[4.939s] [ext: omni.warp-0.2.1] startup
Warp initialized:
Version: 0.2.1
Using CUDA device: NVIDIA RTX A5500
Using CPU compiler: /usr/bin/g++
[6.202s] [ext: omni.kit.renderer.capture-0.0.0] startup
[6.208s] [ext: omni.kit.property.material-1.8.5] startup
[6.211s] [ext: omni.physx.demos-1.4.12-5.1] startup
[6.213s] [ext: omni.physics.tensors-0.1.0] startup
[6.219s] [ext: omni.kit.property.physx-0.1.0] startup
2023-02-06 18:02:20 [6,265ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[6.273s] [ext: omni.kit.window.toolbar-1.2.4] startup
[6.277s] [ext: omni.physx.tensors-0.1.0] startup
[6.282s] [ext: omni.physx.vehicle-1.4.12-5.1] startup
[6.291s] [ext: omni.physx.tests-1.4.12-5.1] startup
[6.322s] [ext: omni.kit.numpy.common-0.1.0] startup
[6.324s] [ext: omni.physx.camera-1.4.12-5.1] startup
[6.329s] [ext: omni.physx.cct-1.4.12-5.1] startup
[6.366s] [ext: omni.isaac.version-1.0.0] startup
[6.367s] [ext: omni.isaac.dynamic_control-1.0.0] startup
[6.373s] [ext: omni.physx.bundle-1.4.12-5.1] startup
[6.373s] [ext: omni.kit.primitive.mesh-1.0.0] startup
[6.376s] [ext: omni.command.usd-1.0.1] startup
[6.379s] [ext: omni.isaac.core-1.15.1] startup
[6.522s] [ext: omni.replicator.core-1.2.0] startup
[6.680s] [ext: omni.kit.window.extensions-1.1.0] startup
[6.684s] [ext: omni.isaac.core_nodes-0.9.0] startup
[6.692s] [ext: omni.isaac.ui-0.2.1] startup
[6.694s] [ext: omni.kit.window.script_editor-1.6.2] startup
[6.698s] [ext: omni.isaac.wheeled_robots-0.5.4] startup
[6.706s] [ext: omni.kit.menu.common-1.0.0] startup
[6.707s] [ext: omni.kit.graph.delegate.default-1.0.15] startup
[6.709s] [ext: omni.kit.graph.delegate.modern-1.6.0] startup
[6.711s] [ext: omni.kit.widget.zoombar-1.0.3] startup
[6.711s] [ext: omni.kit.graph.editor.core-1.3.3] startup
[6.714s] [ext: omni.kit.widget.stage_icons-1.0.2] startup
[6.715s] [ext: omni.kit.browser.core-2.0.12] startup
[6.718s] [ext: omni.kit.graph.widget.variables-2.0.2] startup
[6.719s] [ext: omni.kit.window.stage-2.3.7] startup
[6.722s] [ext: omni.kit.browser.folder.core-1.1.13] startup
[6.725s] [ext: omni.graph.window.core-1.22.1] startup
[6.732s] [ext: omni.isaac.lula-1.1.0] startup
[6.741s] [ext: omni.graph.instancing-1.1.4] startup
[6.745s] [ext: omni.graph.window.action-1.3.8] startup
[6.746s] [ext: omni.graph.tutorials-1.1.2] startup
[6.757s] [ext: omni.rtx.window.settings-0.6.1] startup
[6.761s] [ext: omni.isaac.motion_planning-0.2.0] startup
[6.767s] [ext: omni.graph.bundle.action-1.0.0] startup
[6.767s] [ext: omni.rtx.settings.core-0.5.5] startup
[6.771s] [ext: omni.isaac.motion_generation-3.1.2] startup
[6.774s] [ext: omni.isaac.kit-0.1.9] startup
[6.775s] [ext: omni.isaac.debug_draw-0.1.2] startup
[6.779s] [ext: omni.kit.selection-0.1.0] startup
[6.780s] [ext: omni.isaac.franka-0.0.0] startup
[6.780s] [ext: omni.kit.widget.layers-1.5.17] startup
[6.791s] [ext: omni.kit.menu.edit-1.0.6] startup
[6.792s] [ext: omni.isaac.isaac_sensor-0.3.4] startup
2023-02-06 18:02:21 [6,789ms] [Warning] [omni.physx.plugin] Deprecated: getSimulationEventStream is deprecated, please use getSimulationEventStreamV2
[6.797s] [ext: omni.kit.widget.live-0.1.0] startup
[6.800s] [ext: omni.isaac.surface_gripper-0.1.2] startup
[6.803s] [ext: omni.kit.property.layer-1.1.2] startup
[6.805s] [ext: omni.isaac.range_sensor-0.4.2] startup
[6.826s] [ext: omni.graph.window.generic-1.3.8] startup
[6.828s] [ext: omni.isaac.utils-0.1.11] startup
[6.830s] [ext: omni.isaac.universal_robots-0.2.1] startup
[6.831s] [ext: omni.kit.property.audio-1.0.5] startup
[6.833s] [ext: omni.kit.property.skel-1.0.1] startup
[6.834s] [ext: omni.kit.property.render-1.1.0] startup
[6.835s] [ext: omni.kit.property.camera-1.0.3] startup
[6.836s] [ext: omni.kit.property.geometry-1.2.0] startup
[6.837s] [ext: omni.kit.property.light-1.0.5] startup
[6.838s] [ext: omni.kit.property.transform-1.0.2] startup
[6.841s] [ext: omni.isaac.occupancy_map-0.2.4] startup
[6.851s] [ext: omni.ui.scene-1.4.6] startup
[6.856s] [ext: omni.kit.window.console-0.2.0] startup
[6.860s] [ext: omni.kit.window.status_bar-0.1.1] startup
[6.863s] [ext: omni.kit.property.bundle-1.2.4] startup
[6.864s] [ext: omni.kit.menu.file-1.0.8] startup
[6.865s] [ext: omni.kit.manipulator.viewport-1.0.6] startup
[6.867s] [ext: omni.isaac.urdf-0.2.2] startup
[6.891s] [ext: omni.isaac.dofbot-0.2.0] startup
[6.892s] [ext: omni.kit.window.title-1.1.1] startup
[6.892s] [ext: omni.kit.profiler.window-1.4.4] startup
[6.898s] [ext: omni.graph.visualization.nodes-1.1.1] startup
[6.901s] [ext: omni.isaac.synthetic_utils-0.2.1] startup
[6.907s] [ext: semantics.schema.editor-0.2.2] startup
[6.909s] [ext: omni.isaac.sim.python-2022.1.0] startup
[6.910s] Simulation App Starting
2023-02-06 18:02:21 [6,907ms] [Warning] [gpu.foundation.plugin] Different GPUs have different available VRAM. The texture streaming system will be limited to the lower VRAM number
[7.545s] app ready
[11.446s] Simulation App Startup Complete
Parsing and checking input parameterization.
2023-02-06 18:02:26 [11,504ms] [Warning] [omni.client.python] Waiting for a request that has taken longer than 20ms to complete:
File "tools/composer/src/main.py", line 243, in <module>
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 39, in __init__
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 310, in parse_input
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 296, in parse_params
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 153, in verify_nucleus_paths
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/extscore/omni.client/omni/client/__init__.py", line 360, in read_file
2023-02-06 18:02:26 [11,911ms] [Warning] [omni.client.python] Waiting for a request that has taken longer than 20ms to complete:
File "tools/composer/src/main.py", line 243, in <module>
File "/home/sbc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 39, in __init__
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 310, in parse_input
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 296, in parse_params
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 131, in verify_nucleus_paths
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 131, in verify_nucleus_paths
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/tools/composer/src/input/parse.py", line 153, in verify_nucleus_paths
File "/home/abc/.local/share/ov/pkg/isaac_sim-2022.1.0/kit/extscore/omni.client/omni/client/__init__.py", line 360, in read_file
Starting at index 0
Isaac Sim starting up...
2023-02-06 18:02:27 [12,514ms] [Warning] [carb.flatcache.plugin] Type tag does not have a corresponding USD type
2023-02-06 18:02:50 [35,779ms] [Warning] [omni.isaac.core.simulation_context.simulation_context] A new stage was opened, World or Simulation Object are invalidated and you would need to initialize them again before using them.
2023-02-06 18:02:52 [37,633ms] [Warning] [gpu.foundation.plugin] Requesting texture to use texture streaming, but the service isn't available.The texture will be created as a regular resource.
Camera intrinsics
- width, height: 1920, 1080
- focal_length: 18.149999618530273
- horiz_aperture, vert_aperture: 20.95, 15.29
- horiz_fov, vert_fov: 59.99, 45.68
- focal_x, focal_y: 1662.99, 1281.95
- proj_mat:
[[-1.73 0. 0. 0. ]
[ 0. 3.08 0. 0. ]
[ 0. 0. 1. 1. ]
[ 0. 0. 1. 0. ]]
./python.sh: line 46: 161609 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
I’m not very familiar with Replicator Composer but can you running the script once with an Internet connection to try pull the dependencies online?
The workstation is continuously connected to the internet when I execute any of the mentioned scripts.
@mahdi65 Can you share your yaml file. The error might be due to connection error to nucleus server.
Hi @mahdi65 - Please confirm if you still see this issue after updating to the latest Isaac Sim 2022.2.1 release.