[5.375s] [ext: omni.kit.context_menu-0.1.0] startup
[5.502s] [ext: omni.kit.default-0.1] startup
[5.507s] [ext: omni.physx] startup
2021-03-24 02:00:22 [5,613ms] [Error] [omni.physx.plugin] PhysX error: Could not find registered CUDA function ‘updateBodyExternalVelocitiesLaunch’., FILE /buildAgent/work/ce945f8f79bc260f/source/cudamanager/src/CudaKernelWrangler.cpp, LINE 101
./_build/linux-x86_64/release/isaac-sim.sh: line 4: 5115 Segmentation fault (core dumped) “$SCRIPT_DIR/…/…/target-deps/kit_sdk_release/_build/linux-x86_64/release/omniverse-kit” --merge-config=“$SCRIPT_DIR//experiences/isaac-sim.json” $@
It runs fine before, but after I resized my root partition suddenly Isaac gives me this error everytime I start the app. I have tried to reinstall nvidia drivers, cuda, Isaac sim itself. none works. What should I do? Thanks.
Hi, please try uninstalling CUDA and the NVIDIA drivers. Then install the NVIDIA 455.45 drivers only.
Hi, thanks for the reply. I have tried to uninstall and install 455.45, still no luck. The following is my nvidia-smi output:
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 Off | 00000000:01:00.0 On | N/A |
| N/A 55C P8 7W / N/A | 292MiB / 7982MiB | 2% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 2060 G /usr/lib/xorg/Xorg 24MiB |
| 0 N/A N/A 2314 G /usr/bin/gnome-shell 50MiB |
| 0 N/A N/A 2749 G /usr/lib/xorg/Xorg 128MiB |
| 0 N/A N/A 2877 G /usr/bin/gnome-shell 83MiB |
| 0 N/A N/A 2907 G ...mviewer/tv_bin/TeamViewer 2MiB |
+-----------------------------------------------------------------------------+
And below is all the output from Isaac sim:
[Info] [carb] Logging to file: /home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/data/Kit/isaac-sim/2020.2.28460/omniverse-kit.log
2021-03-24 15:33:40 [2ms] [Warning] [omni.kit.app.plugin] [Graphics API] Vulkan
[0.160s] [ext: omni.timeline-0.1.0] startup
[0.198s] [ext: omni.kit.async_engine-0.1.0] startup
[0.263s] [ext: omni.assets-0.1.0] startup
[0.271s] [ext: omni.kit.test-0.1.0] startup
[0.285s] [ext: omni.usd-0.1.0] startup
[2.253s] [ext: omni.client-0.1.0] startup
2021-03-24 15:33:42 [2,245ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [carb::tokens::ITokens v0.1] (plugin: (null)), by client: omni.client.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
[2.281s] [ext: omni.kit.registry.nucleus-0.1] startup
[2.287s] [ext: omni.kit.pipapi-0.1.0] startup
[2.528s] [ext: omni.stats-0.1.0] startup
[2.533s] [ext: omni.kit.pip_archive-0.1] startup
2021-03-24 15:33:42 [2,510ms] [Warning] [carb.python] There is already a module 'toml' in '/home/henry/.local/lib/python3.6/site-packages/toml/__init__.py'. Can't import another module with the same name in public space from '/home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/exts/omni.kit.pip_archive-0.1/pip_prebundle'.
[2.559s] [ext: omni.appwindow-0.1.0] startup
fakexrandr/mmap(): Invalid argument
fakexrandr/mmap(): Invalid argument
[2.609s] [ext: omni.ansel-0.1.0] startup
[2.612s] [ext: omni.kit.commands-0.1] startup
[2.620s] [ext: omni.ui-1.0.0] startup
[2.641s] [ext: omni.kit.selection-0.1] startup
[2.867s] [ext: omni.renderer-0.1.0-rtx] startup
[2.869s] [ext: omni.kit.editor-0.1.0-full] startup
|---------------------------------------------------------------------------------------|
| Driver Version: 20.0
|=======================================================================================|
| GPU | Name | Active | GPU Memory | Vendor-ID | LUID |
| | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------|
| 0 | Intel(R) UHD Graphics (CML GT2) | | 11848 MB | 8086 | 0 |
| | | | | 9bc4 | bfae6895.. |
|---------------------------------------------------------------------------------------|
| 1 | GeForce RTX 2070 | Yes | 8438 MB | 10de | 0 |
| | | | | 1f14 | cf365b30.. |
|=======================================================================================|
| Processor: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz | Cores: Unknown | Logical: 12
|---------------------------------------------------------------------------------------|
| Total Memory (MB): 15797 | Free Memory: 12083
| Total Page/Swap (MB): 10420 | Free Page/Swap: 10420
|---------------------------------------------------------------------------------------|
2021-03-24 15:33:45 [5,613ms] [Warning] [carb.tasking.plugin] Counter 0x7f4574001080 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:45 [5,616ms] [Warning] [carb.tasking.plugin] Counter 0x7f45740051a0 released with 1 waiters; spinning until waiters finish
2021-03-24 15:33:45 [5,662ms] [Warning] [carb.tasking.plugin] Counter 0x7f4550001080 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:47 [7,570ms] [Warning] [carb.tasking.plugin] Counter 0x7f45440c9580 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:47 [7,585ms] [Warning] [carb.tasking.plugin] Counter 0x7f457cab3890 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:47 [7,669ms] [Warning] [carb.tasking.plugin] Counter 0x7f456037b610 released with 3 waiters; spinning until waiters finish
[7.842s] [ext: omni.kit.widget.live-0.1.0] startup
2021-03-24 15:33:48 [7,816ms] [Warning] [omni.ext-live.plugin] Unabled to detect Omniverse Cache Server. File /home/henry/.config/omniverse/Cache/Omniverse.toml is not found. Consider installing it (version >= 280) for better IO performance.
[7.845s] [ext: omni.convexdecomposition] startup
[7.852s] [ext: omni.debugdraw] startup
[7.859s] [ext: omni.isaac.about] startup
[7.864s] [ext: omni.isaac.manip] startup
[7.875s] [ext: omni.kit.primitive.mesh-0.1.0] startup
[7.883s] [ext: omni.kit.stage_templates-0.1] startup
[7.900s] [ext: omni.kit.window.console-0.1.0] startup
[7.902s] [ext: omni.kit.window.script_editor-0.1.0] startup
[7.903s] [ext: omni.kit.window.stats-0.1.0] startup
[7.909s] [ext: omni.kit.window.toolbar-0.1.0] startup
[7.921s] [ext: omni.kit.window.usd_paths-0.1.0] startup
[7.927s] [ext: omni.kit.window.viewport-0.1.0] startup
[7.953s] [ext: omni.syntheticdata] startup
2021-03-24 15:33:48 [8,022ms] [Warning] [carb.tasking.plugin] Counter 0x7f4558095b40 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:48 [8,036ms] [Warning] [carb.tasking.plugin] Counter 0x7f454c005550 released with 2 waiters; spinning until waiters finish
[8.087s] [ext: omni.tagging-0.1.0] startup
[8.101s] [ext: omni.kit.window.content-0.1.0] startup
[8.110s] [ext: omni.kit.window.tagging-0.1.0] startup
[8.115s] [ext: omni.isaac.synthetic_utils] startup
[8.533s] [ext: omni.kit.context_menu-0.1.0] startup
[8.659s] [ext: omni.kit.default-0.1] startup
[8.663s] [ext: omni.physx] startup
2021-03-24 15:33:48 [8,728ms] [Error] [omni.physx.plugin] PhysX error: Could not find registered CUDA function 'updateBodyExternalVelocitiesLaunch'., FILE /buildAgent/work/ce945f8f79bc260f/source/cudamanager/src/CudaKernelWrangler.cpp, LINE 101
/home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/linux-x86_64/release/isaac-sim.sh: line 4: 3367 Segmentation fault (core dumped) "$SCRIPT_DIR/../../target-deps/kit_sdk_release/_build/linux-x86_64/release/omniverse-kit" --merge-config="$SCRIPT_DIR//experiences/isaac-sim.json" $@
Would really appreciate your help since I’ll be needing Isaac sim urgently, thanks!
Just checking, have you uninstalled the CUDA drivers too? See Installation Guide Linux :: CUDA Toolkit Documentation.
Yes I did. I have followed the instruction to uninstall and install the nvidia driver again, sadly the CUDA error is still there. Do I need to install cuda or the toolkit again after installing the 455.45 driver?
No, you do not need to install the CUDA toolkit to run Isaac Sim. CUDA runtime is included in the NVIDIA drivers.
Please try the 450.80 drivers instead. Linux x64 (AMD64/EM64T) Display Driver | 450.80.02 | Linux 64-bit | NVIDIA
Thanks. However it still didn’t work. I have removed everything and installed the 450.80 driver, it still produces the same error. Could there be something misconfigured other than the driver?
[5.454s] [ext: omni.physx] startup
2021-03-24 20:48:36 [5,512ms] [Error] [omni.physx.plugin] PhysX error: Could not find registered CUDA function 'updateBodyExternalVelocitiesLaunch'., FILE /buildAgent/work/ce945f8f79bc260f/source/cudamanager/src/CudaKernelWrangler.cpp, LINE 101
/home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/linux-x86_64/release/isaac-sim.sh: line 4: 3233 Segmentation fault (core dumped) "$SCRIPT_DIR/../../target-deps/kit_sdk_release/_build/linux-x86_64/release/omniverse-kit" --merge-config="$SCRIPT_DIR//experiences/isaac-sim.json" $@
➜ Downloads nvidia-smi
Wed Mar 24 20:48:49 2021
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 2070 Off | 00000000:01:00.0 Off | N/A |
| N/A 53C P8 6W / N/A | 262MiB / 7982MiB | 19% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1959 G /usr/lib/xorg/Xorg 27MiB |
| 0 N/A N/A 2150 G /usr/bin/gnome-shell 50MiB |
| 0 N/A N/A 2603 G /usr/lib/xorg/Xorg 103MiB |
| 0 N/A N/A 2731 G /usr/bin/gnome-shell 74MiB |
| 0 N/A N/A 2739 G ...mviewer/tv_bin/TeamViewer 2MiB |
+-----------------------------------------------------------------------------+
I have tried to install the container version which launches ok, however I can’t switch to it since I need the ROS bridge to be able to work. So since it works in docker, could there be some config that affects the physx plugin and cuda?
That looks like you still have a conflicting CUDA version installed. I think in a container, the CUDA version is set in the image itself.
Try running nvcc –version. It should say Command 'nvcc' not found.
Also check that CUDA is not in your LD_LIBRARY_PATH.
It still does not work. I have check CUDA is not in LD_LIBRARY_PATH, and nvcc --version will return Command ‘nvcc’ not found.
Here is my nvidia-smi output:
±----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 208… Off | 00000000:1A:00.0 Off | N/A |
| 18% 40C P8 14W / 250W | 10MiB / 11019MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 208… Off | 00000000:68:00.0 On | N/A |
| 18% 48C P8 20W / 250W | 366MiB / 11016MiB | 7% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
by the way, isaac sim works in the container.
@henry_z @frank_hu Do you have PyTorch installed? Please try running Isaac Sim again after removing PyTorch. We may have a workaround if PyTorch is the issue.
Hi @frank_hu @Sheikh_Dawood, sorry I didn’t reply earlier and thanks for the help, in my case it was indeed because of PyTorch. After days of struggle, reinstalling drivers and cuda, even reinstalling the ubuntu system in the root partition, the problem still exists. Then after I remove the newly installed python dependencies from .local folder in the home partition, Isaac finally launches. The error message is quite misleading!
1 Like
Thanks for confirming it. Yes, sorry it took long to track it down. I’ve marked this issue as [OM-29498] to keep track before the next release.
2 Likes
Another workaround is searching for import torch in your code base and add import omni.physx in front.
import omni.physx
import torch
1 Like
@HaiLocLu Thank you. Actually, it works. I think this should be the solution. Thanks