Isaac sim startup segmentation fault - Could not find registered CUDA function

[5.375s] [ext: omni.kit.context_menu-0.1.0] startup
[5.502s] [ext: omni.kit.default-0.1] startup
[5.507s] [ext: omni.physx] startup
2021-03-24 02:00:22 [5,613ms] [Error] [omni.physx.plugin] PhysX error: Could not find registered CUDA function ‘updateBodyExternalVelocitiesLaunch’., FILE /buildAgent/work/ce945f8f79bc260f/source/cudamanager/src/CudaKernelWrangler.cpp, LINE 101
./_build/linux-x86_64/release/isaac-sim.sh: line 4: 5115 Segmentation fault (core dumped) “$SCRIPT_DIR/…/…/target-deps/kit_sdk_release/_build/linux-x86_64/release/omniverse-kit” --merge-config="SCRIPT_DIR//experiences/isaac-sim.json" @

It runs fine before, but after I resized my root partition suddenly Isaac gives me this error everytime I start the app. I have tried to reinstall nvidia drivers, cuda, Isaac sim itself. none works. What should I do? Thanks.

Hi, please try uninstalling CUDA and the NVIDIA drivers. Then install the NVIDIA 455.45 drivers only.

Hi, thanks for the reply. I have tried to uninstall and install 455.45, still no luck. The following is my nvidia-smi output:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01    Driver Version: 455.45.01    CUDA Version: 11.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0  On |                  N/A |
| N/A   55C    P8     7W /  N/A |    292MiB /  7982MiB |      2%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2060      G   /usr/lib/xorg/Xorg                 24MiB |
|    0   N/A  N/A      2314      G   /usr/bin/gnome-shell               50MiB |
|    0   N/A  N/A      2749      G   /usr/lib/xorg/Xorg                128MiB |
|    0   N/A  N/A      2877      G   /usr/bin/gnome-shell               83MiB |
|    0   N/A  N/A      2907      G   ...mviewer/tv_bin/TeamViewer        2MiB |
+-----------------------------------------------------------------------------+

And below is all the output from Isaac sim:

[Info] [carb] Logging to file: /home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/data/Kit/isaac-sim/2020.2.28460/omniverse-kit.log
2021-03-24 15:33:40 [2ms] [Warning] [omni.kit.app.plugin] [Graphics API] Vulkan

[0.160s] [ext: omni.timeline-0.1.0] startup
[0.198s] [ext: omni.kit.async_engine-0.1.0] startup
[0.263s] [ext: omni.assets-0.1.0] startup
[0.271s] [ext: omni.kit.test-0.1.0] startup
[0.285s] [ext: omni.usd-0.1.0] startup
[2.253s] [ext: omni.client-0.1.0] startup
2021-03-24 15:33:42 [2,245ms] [Warning] [carb] Acquiring non optional plugin interface which is not listed as dependency: [carb::tokens::ITokens v0.1] (plugin: (null)), by client: omni.client.plugin. Add it to CARB_PLUGIN_IMPL_DEPS() macro of a client.
[2.281s] [ext: omni.kit.registry.nucleus-0.1] startup
[2.287s] [ext: omni.kit.pipapi-0.1.0] startup
[2.528s] [ext: omni.stats-0.1.0] startup
[2.533s] [ext: omni.kit.pip_archive-0.1] startup
2021-03-24 15:33:42 [2,510ms] [Warning] [carb.python] There is already a module 'toml' in '/home/henry/.local/lib/python3.6/site-packages/toml/__init__.py'. Can't import another module with the same name in public space from '/home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/target-deps/kit_sdk_release/_build/linux-x86_64/release/exts/omni.kit.pip_archive-0.1/pip_prebundle'.
[2.559s] [ext: omni.appwindow-0.1.0] startup
fakexrandr/mmap(): Invalid argument
fakexrandr/mmap(): Invalid argument
[2.609s] [ext: omni.ansel-0.1.0] startup
[2.612s] [ext: omni.kit.commands-0.1] startup
[2.620s] [ext: omni.ui-1.0.0] startup
[2.641s] [ext: omni.kit.selection-0.1] startup
[2.867s] [ext: omni.renderer-0.1.0-rtx] startup
[2.869s] [ext: omni.kit.editor-0.1.0-full] startup

|---------------------------------------------------------------------------------------|
| Driver Version: 20.0
|=======================================================================================|
| GPU | Name                             | Active | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |            | Device-ID | UUID       |
|---------------------------------------------------------------------------------------|
| 0   | Intel(R) UHD Graphics (CML GT2)  |        | 11848   MB | 8086      | 0          |
|     |                                  |        |            | 9bc4      | bfae6895.. |
|---------------------------------------------------------------------------------------|
| 1   | GeForce RTX 2070                 | Yes    | 8438    MB | 10de      | 0          |
|     |                                  |        |            | 1f14      | cf365b30.. |
|=======================================================================================|
| Processor: Intel(R) Core(TM) i7-10750H CPU @ 2.60GHz | Cores: Unknown | Logical: 12
|---------------------------------------------------------------------------------------|
| Total Memory (MB): 15797 | Free Memory: 12083
| Total Page/Swap (MB): 10420 | Free Page/Swap: 10420
|---------------------------------------------------------------------------------------|
2021-03-24 15:33:45 [5,613ms] [Warning] [carb.tasking.plugin] Counter 0x7f4574001080 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:45 [5,616ms] [Warning] [carb.tasking.plugin] Counter 0x7f45740051a0 released with 1 waiters; spinning until waiters finish
2021-03-24 15:33:45 [5,662ms] [Warning] [carb.tasking.plugin] Counter 0x7f4550001080 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:47 [7,570ms] [Warning] [carb.tasking.plugin] Counter 0x7f45440c9580 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:47 [7,585ms] [Warning] [carb.tasking.plugin] Counter 0x7f457cab3890 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:47 [7,669ms] [Warning] [carb.tasking.plugin] Counter 0x7f456037b610 released with 3 waiters; spinning until waiters finish
[7.842s] [ext: omni.kit.widget.live-0.1.0] startup
2021-03-24 15:33:48 [7,816ms] [Warning] [omni.ext-live.plugin] Unabled to detect Omniverse Cache Server. File /home/henry/.config/omniverse/Cache/Omniverse.toml is not found. Consider installing it (version >= 280) for better IO performance.
[7.845s] [ext: omni.convexdecomposition] startup
[7.852s] [ext: omni.debugdraw] startup
[7.859s] [ext: omni.isaac.about] startup
[7.864s] [ext: omni.isaac.manip] startup
[7.875s] [ext: omni.kit.primitive.mesh-0.1.0] startup
[7.883s] [ext: omni.kit.stage_templates-0.1] startup
[7.900s] [ext: omni.kit.window.console-0.1.0] startup
[7.902s] [ext: omni.kit.window.script_editor-0.1.0] startup
[7.903s] [ext: omni.kit.window.stats-0.1.0] startup
[7.909s] [ext: omni.kit.window.toolbar-0.1.0] startup
[7.921s] [ext: omni.kit.window.usd_paths-0.1.0] startup
[7.927s] [ext: omni.kit.window.viewport-0.1.0] startup
[7.953s] [ext: omni.syntheticdata] startup
2021-03-24 15:33:48 [8,022ms] [Warning] [carb.tasking.plugin] Counter 0x7f4558095b40 released with 3 waiters; spinning until waiters finish
2021-03-24 15:33:48 [8,036ms] [Warning] [carb.tasking.plugin] Counter 0x7f454c005550 released with 2 waiters; spinning until waiters finish
[8.087s] [ext: omni.tagging-0.1.0] startup
[8.101s] [ext: omni.kit.window.content-0.1.0] startup
[8.110s] [ext: omni.kit.window.tagging-0.1.0] startup
[8.115s] [ext: omni.isaac.synthetic_utils] startup
[8.533s] [ext: omni.kit.context_menu-0.1.0] startup
[8.659s] [ext: omni.kit.default-0.1] startup
[8.663s] [ext: omni.physx] startup
2021-03-24 15:33:48 [8,728ms] [Error] [omni.physx.plugin] PhysX error: Could not find registered CUDA function 'updateBodyExternalVelocitiesLaunch'., FILE /buildAgent/work/ce945f8f79bc260f/source/cudamanager/src/CudaKernelWrangler.cpp, LINE 101
/home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/linux-x86_64/release/isaac-sim.sh: line 4:  3367 Segmentation fault      (core dumped) "$SCRIPT_DIR/../../target-deps/kit_sdk_release/_build/linux-x86_64/release/omniverse-kit" --merge-config="$SCRIPT_DIR//experiences/isaac-sim.json" $@

Would really appreciate your help since I’ll be needing Isaac sim urgently, thanks!

Just checking, have you uninstalled the CUDA drivers too? See Installation Guide Linux :: CUDA Toolkit Documentation.

Yes I did. I have followed the instruction to uninstall and install the nvidia driver again, sadly the CUDA error is still there. Do I need to install cuda or the toolkit again after installing the 455.45 driver?

No, you do not need to install the CUDA toolkit to run Isaac Sim. CUDA runtime is included in the NVIDIA drivers.
Please try the 450.80 drivers instead. NVIDIA DRIVERS Linux x64 (AMD64/EM64T) Display Driver

Thanks. However it still didn’t work. I have removed everything and installed the 450.80 driver, it still produces the same error. Could there be something misconfigured other than the driver?

[5.454s] [ext: omni.physx] startup
2021-03-24 20:48:36 [5,512ms] [Error] [omni.physx.plugin] PhysX error: Could not find registered CUDA function 'updateBodyExternalVelocitiesLaunch'., FILE /buildAgent/work/ce945f8f79bc260f/source/cudamanager/src/CudaKernelWrangler.cpp, LINE 101
/home/henry/Documents/isaac-sim-2020.2.2007-linux-x86_64-release/_build/linux-x86_64/release/isaac-sim.sh: line 4:  3233 Segmentation fault      (core dumped) "$SCRIPT_DIR/../../target-deps/kit_sdk_release/_build/linux-x86_64/release/omniverse-kit" --merge-config="$SCRIPT_DIR//experiences/isaac-sim.json" $@
➜  Downloads nvidia-smi
Wed Mar 24 20:48:49 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.80.02    Driver Version: 450.80.02    CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 2070    Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   53C    P8     6W /  N/A |    262MiB /  7982MiB |     19%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1959      G   /usr/lib/xorg/Xorg                 27MiB |
|    0   N/A  N/A      2150      G   /usr/bin/gnome-shell               50MiB |
|    0   N/A  N/A      2603      G   /usr/lib/xorg/Xorg                103MiB |
|    0   N/A  N/A      2731      G   /usr/bin/gnome-shell               74MiB |
|    0   N/A  N/A      2739      G   ...mviewer/tv_bin/TeamViewer        2MiB |
+-----------------------------------------------------------------------------+

I have tried to install the container version which launches ok, however I can’t switch to it since I need the ROS bridge to be able to work. So since it works in docker, could there be some config that affects the physx plugin and cuda?

That looks like you still have a conflicting CUDA version installed. I think in a container, the CUDA version is set in the image itself.
Try running nvcc –version. It should say Command 'nvcc' not found.
Also check that CUDA is not in your LD_LIBRARY_PATH.

It still does not work. I have check CUDA is not in LD_LIBRARY_PATH, and nvcc --version will return Command ‘nvcc’ not found.

Here is my nvidia-smi output:
±----------------------------------------------------------------------------+
| NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 208… Off | 00000000:1A:00.0 Off | N/A |
| 18% 40C P8 14W / 250W | 10MiB / 11019MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 208… Off | 00000000:68:00.0 On | N/A |
| 18% 48C P8 20W / 250W | 366MiB / 11016MiB | 7% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

by the way, isaac sim works in the container.

@henry_z @frank_hu Do you have PyTorch installed? Please try running Isaac Sim again after removing PyTorch. We may have a workaround if PyTorch is the issue.

Hi @frank_hu @Sheikh_Dawood, sorry I didn’t reply earlier and thanks for the help, in my case it was indeed because of PyTorch. After days of struggle, reinstalling drivers and cuda, even reinstalling the ubuntu system in the root partition, the problem still exists. Then after I remove the newly installed python dependencies from .local folder in the home partition, Isaac finally launches. The error message is quite misleading!

1 Like

Thanks for confirming it. Yes, sorry it took long to track it down. I’ve marked this issue as [OM-29498] to keep track before the next release.

2 Likes

Another workaround is searching for import torch in your code base and add import omni.physx in front.

import omni.physx
import torch

1 Like

@HaiLocLu Thank you. Actually, it works. I think this should be the solution. Thanks