DGX Station A100
Ubuntu 20.04
Hi.
While trying to install Isaac SIM from Omniverse launcher, i got the following error:
Error occurred during installation of Isaac Sim: Command failed: "/home/userabc/.local/share/ov/pkg/isaac_sim-2021.2.1/omni.isaac.sim.post.install.sh"
ln: failed to create symbolic link 'extension_examples/examples': File exists
What should I do next before I install it again? Should I clean up something first?
Hi try uninstalling that version of Isaac Sim then remove the folder below if it exists.
/home/userabc/.local/share/ov/pkg/isaac_sim-2021.2.1
Next reinstall Isaac Sim.
Thanks @Sheikh_Dawood
BTW, does Isaac SIM require RTX series GPU to run?
It seems that I cannot host Isaac SIM on DGX A100 workstation according to this doc under Omniverse.
Yes.
It should be fine running Isaac Sim on an A100. Here is our System Requirements page:
https://docs.omniverse.nvidia.com/app_isaacsim/app_isaacsim/requirements.html
Can you please confirm that the A100 is connected using PCIE? This is version of A100 is required (compared to the non PCIE version) if you are using livestreaming from the DGX Station. It has NVENC support.
Driver version: 470.57.02
CUDA version 11.04
May I know how to check A100 is connected using PCIE?
After reinstalling Isaac SIM, I got the following error log, does that mean the GPU on my DGX A100 workstation is not connected using PCIE?
Error occurred during installation of Isaac Sim: Command failed: “/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/omni.isaac.sim.post.install.sh”
2022-01-12 02:11:58 [1,686ms] [Error] [gpu.foundation.plugin] No device could be created. Some known system issues:
- The driver is not installed properly and requires a clean re-install.
- Your GPUs do not support RayTracing: DXR or Vulkan ray_tracing, or hardware is excluded due to performance.
- The driver cannot enumerate any GPU: driver, display or a docker issue. For Vulkan, test it with Vulkaninfo tool from Vulkan SDK, instead of nvidia-smi.
- For Ubuntu, it requires server-xorg-core 1.20.7+ and a display to work without --no-window.
- For Linux dockers, the setup is not complete. Install the latest driver, xServer and NVIDIA container runtime.
2022-01-12 02:12:00 [3,906ms] [Error] [omni.kit.window.viewport.plugin] Failed to initialize resource manager
2022-01-12 02:12:00 [3,906ms] [Error] [omni.kit.window.viewport.plugin] Failed to prepare GPU Foundation
2022-01-12 02:12:00 [3,906ms] [Error] [omni.kit.window.viewport.plugin] Failed to startup required plugins
2022-01-12 02:12:00 [3,915ms] [Error] [carb.scripting-python.plugin] AttributeError: ‘NoneType’ object has no attribute ‘get_mouse_event_stream’At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/extscore/omni.kit.window.viewport/omni/kit/viewport/scripts/context_menu.py(27): on_startup
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(143): _startup_ext
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(174): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2):2022-01-12 02:12:00 [3,915ms] [Error] [omni.ext.plugin] [ext: omni.kit.window.viewport-0.0.0] Failed to process python module extension in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/extscore/omni.kit.window.viewport/.’.
2022-01-12 02:12:01 [4,167ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.core. Error: No module named ‘omni.kit.viewport’. Traceback:
Traceback (most recent call last):
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py”, line 65, in import_module
return importlib.import_module(name)
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/python/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1006, in _gcd_import
File “”, line 983, in _find_and_load
File “”, line 967, in _find_and_load_unlocked
File “”, line 677, in _load_unlocked
File “”, line 728, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/init.py”, line 10, in
from omni.isaac.core.simulation_context.simulation_context import SimulationContext
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/init.py”, line 9, in
from omni.isaac.core.simulation_context.simulation_context import SimulationContext
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py”, line 14, in
from omni.isaac.core.utils.viewports import set_camera_view
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/utils/viewports.py”, line 16, in
import omni.kit.viewport
ModuleNotFoundError: No module named ‘omni.kit.viewport’2022-01-12 02:12:01 [4,167ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: ‘omni.isaac.core’ in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/.’ failed to load.
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2):2022-01-12 02:12:01 [4,167ms] [Error] [omni.ext.plugin] [ext: omni.isaac.core-0.1.12] Failed to process python module extension in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/.’.
2022-01-12 02:12:01 [4,274ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.synthetic_utils.tests. Error: No module named ‘omni.isaac.core’. Traceback:
Traceback (most recent call last):
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py”, line 65, in import_module
return importlib.import_module(name)
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/python/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1006, in _gcd_import
File “”, line 983, in _find_and_load
File “”, line 967, in _find_and_load_unlocked
File “”, line 677, in _load_unlocked
File “”, line 728, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/tests/init.py”, line 10, in
from .test_synthetic_utils import *
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/tests/test_synthetic_utils.py”, line 31, in
from omni.isaac.core.utils.physics import simulate_async
ModuleNotFoundError: No module named ‘omni.isaac.core’2022-01-12 02:12:01 [4,274ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: ‘omni.isaac.synthetic_utils.tests’ in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/.’ failed to load.
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2):2022-01-12 02:12:01 [4,274ms] [Error] [omni.ext.plugin] [ext: omni.isaac.synthetic_utils-0.1.5] Failed to process python module extension in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/.’.
2022-01-12 02:12:01 [4,318ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.synthetic_recorder. Error: No module named ‘omni.isaac.synthetic_utils’. Traceback:
Traceback (most recent call last):
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py”, line 65, in import_module
return importlib.import_module(name)
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/python/lib/python3.7/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1006, in _gcd_import
File “”, line 983, in _find_and_load
File “”, line 967, in _find_and_load_unlocked
File “”, line 677, in _load_unlocked
File “”, line 728, in exec_module
File “”, line 219, in _call_with_frames_removed
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/omni/isaac/synthetic_recorder/init.py”, line 10, in
from .extension import *
File “/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/omni/isaac/synthetic_recorder/extension.py”, line 29, in
from omni.isaac.synthetic_utils import visualization as vis
ModuleNotFoundError: No module named ‘omni.isaac.synthetic_utils’2022-01-12 02:12:01 [4,318ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: ‘omni.isaac.synthetic_recorder’ in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/.’ failed to load.
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2):2022-01-12 02:12:01 [4,318ms] [Error] [omni.ext.plugin] [ext: omni.isaac.synthetic_recorder-0.1.0] Failed to process python module extension in ‘/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/.’.
2022-01-12 02:12:02 [5,135ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!carbOnPluginStartup
2022-01-12 02:12:02 [5,137ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::Scheduler::)(unsigned int, int, carb::cpp20::latch), carb::tasking::Scheduler*, unsigned int, int, carb::cpp20::latch*> > >::_M_run()
2022-01-12 02:12:02 [5,140ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2022-01-12 02:12:02 [5,140ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.physxui.plugin.so!carbOnPluginStartup
2022-01-12 02:12:02 [5,144ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!omniCoreStart
2022-01-12 02:12:02 [5,148ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!omniCoreStart
2022-01-12 02:12:02 [5,151ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!releaseFramework
2022-01-12 02:12:02 [5,155ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!releaseFramework
2022-01-12 02:12:02 [5,158ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!_init
2022-01-12 02:12:02 [5,162ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocatorstd::string >::_M_realloc_insertstd::string&(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocatorstd::string > >, std::string&)
2022-01-12 02:12:02 [5,165ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!carbOnPluginShutdown
2022-01-12 02:12:02 [5,168ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!char* std::string::_S_construct<char*>(char*, char*, std::allocator const&, std::forward_iterator_tag)
2022-01-12 02:12:02 [5,172ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Rb_tree<std::string, std::string, std::_Identitystd::string, std::lessstd::string, std::allocatorstd::string >::_M_erase(std::_Rb_tree_nodestd::string)
2022-01-12 02:12:02 [5,175ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Rb_tree<std::string, std::string, std::_Identitystd::string, std::lessstd::string, std::allocatorstd::string >::_M_erase(std::_Rb_tree_nodestd::string)
2022-01-12 02:12:02 [5,179ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Rb_tree<std::string, std::string, std::_Identitystd::string, std::lessstd::string, std::allocatorstd::string >::_M_erase(std::_Rb_tree_nodestd::string)
2022-01-12 02:12:02 [5,182ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Hashtable<std::string, std::string, std::allocatorstd::string, std::__detail::_Identity, std::equal_tostd::string, std::hashstd::string, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::operator=(std::_Hashtable<std::string, std::string, std::allocatorstd::string, std::__detail::_Identity, std::equal_tostd::string, std::hashstd::string, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> > const&)
2022-01-12 02:12:02 [5,186ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int ()(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, …)
2022-01-12 02:12:02 [5,189ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2022-01-12 02:12:02 [5,189ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
2022-01-12 02:12:02 [5,193ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2022-01-12 02:12:02 [5,193ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/omni.isaac.sim.post.install.sh: line 10: 3952241 Segmentation fault (core dumped) ${SCRIPT_DIR}/omni.isaac.sim.warmup.sh
Please share the output of nvidia-smi. You should be able to run Isaac Sim. Only the livestreaming feature may not be working if it is not a PCIE version.
You can also try updating the drivers.
Hi @Sheikh_Dawood
here it is.
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.57.02 Driver Version: 470.57.02 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA A100-SXM... On | 00000000:01:00.0 Off | 0 |
| N/A 40C P0 64W / 275W | 10517MiB / 81251MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA A100-SXM... On | 00000000:47:00.0 Off | 0 |
| N/A 40C P0 67W / 275W | 0MiB / 81251MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA A100-SXM... On | 00000000:81:00.0 Off | 0 |
| N/A 40C P0 62W / 275W | 0MiB / 81251MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA DGX Display On | 00000000:C1:00.0 On | N/A |
| 33% 47C P8 N/A / 50W | 321MiB / 3911MiB | 1% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA A100-SXM... On | 00000000:C2:00.0 Off | 0 |
| N/A 40C P0 63W / 275W | 0MiB / 81251MiB | 0% Default |
| | | Disabled |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 1200144 C tritonserver 10514MiB |
| 3 N/A N/A 5121 G /usr/lib/xorg/Xorg 39MiB |
| 3 N/A N/A 2867789 G /usr/lib/xorg/Xorg 99MiB |
| 3 N/A N/A 2867913 G /usr/bin/gnome-shell 101MiB |
| 3 N/A N/A 2871362 G ...AAAAAAAAA= --shared-files 72MiB |
+-----------------------------------------------------------------------------+
Try running Isaac Sim with this flag to select the GPU:
--/renderer/activeGpu=1
See this link for more info:
https://docs.omniverse.nvidia.com/prod_launcher/prod_kit/linux-troubleshooting.html#q9-how-to-specify-what-gpus-to-run-omniverse-apps-on
I have tried to launch Isaac SIM via this command
/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/isaac-sim.sh --/renderer/activeGpu=1
but it shows
2022-01-20 09:54:30 [1,025ms] [Error] [gpu.foundation.plugin] The chosen activeGpu index is higher than the available GPUs. An unsupported driver might also prevent it from enumerating your devices properly.
and
2022-01-20 09:54:30 [1,039ms] [Warning] [omni.gpu_foundation_factory.plugin] RT-capable GPU not found, switching to compatibility mode
2022-01-20 09:54:30 [1,039ms] [Error] [omni.gpu_foundation_factory.plugin] Failed to create GPU foundation!
2022-01-20 09:54:30 [1,039ms] [Error] [omni.kit.renderer.plugin] GPU Foundation is not initialized!
2022-01-20 09:54:30 [1,039ms] [Error] [omni.kit.imgui_renderer.plugin] Invalid device attached to the default window!
I don’t get it, it’s very obvious that I have 5 GPUs (index 0 ~ 4) according to nvidia-smi
here is the complete error log
[Info] [carb] Logging to file: /home/ryan_lin/.nvidia-omniverse/logs/Kit/Isaac-Sim/2021.2/kit_20220120_175429.log
[0.230s] [ext: omni.kit.pipapi-0.0.0] startup
[0.238s] [ext: omni.usd.config-1.0.0] startup
[0.239s] [ext: omni.usd.libs-1.0.0] startup
[0.528s] [ext: omni.kit.pip_archive-0.0.0] startup
[0.535s] [ext: omni.stats-0.0.0] startup
[0.537s] [ext: omni.gpu_foundation-0.0.0] startup
[0.552s] [ext: omni.isaac.pip_archive-0.1.0] startup
[0.689s] [ext: omni.usd.schema.semantics-0.0.0] startup
[0.695s] [ext: omni.usd.schema.audio-0.0.0] startup
[0.700s] [ext: omni.hydra.iray-0.1.0] startup
[0.707s] [ext: omni.assets.plugins-0.0.0] startup
[0.711s] [ext: omni.kit.async_engine-0.0.0] startup
[0.719s] [ext: omni.kit.loop-default-0.1.0] startup
[0.720s] [ext: omni.client-0.1.0] startup
[0.744s] [ext: omni.appwindow-1.0.0] startup
[0.758s] [ext: omni.kit.test-0.0.0] startup
[0.871s] [ext: omni.kit.renderer.core-0.0.0] startup
2022-01-20 09:54:30 [1,025ms] [Error] [gpu.foundation.plugin] The chosen activeGpu index is higher than the available GPUs. An unsupported driver might also prevent it from enumerating your devices properly.
|---------------------------------------------------------------------------------------------|
| Driver Version: 470.57 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA DGX Display | | | 4342 MB | 10de | 0 |
| | | | | | 1fb0 | 1fefbb2d.. |
|=============================================================================================|
| OS: Linux dgx-govtech-a100, Version: 5.4.0-81-generic
| Processor: AMD EPYC 7742 64-Core Processor | Cores: Unknown | Logical: 128
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515835 | Free Memory: 501896
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2022-01-20 09:54:30 [1,025ms] [Error] [gpu.foundation.plugin] No device could be created. Some known system issues:
- The driver is not installed properly and requires a clean re-install.
- Your GPUs do not support RayTracing: DXR or Vulkan ray_tracing, or hardware is excluded due to performance.
- The driver cannot enumerate any GPU: driver, display or a docker issue. For Vulkan, test it with Vulkaninfo tool from Vulkan SDK, instead of nvidia-smi.
- For Ubuntu, it requires server-xorg-core 1.20.7+ and a display to work without --no-window.
- For Linux dockers, the setup is not complete. Install the latest driver, xServer and NVIDIA container runtime.
2022-01-20 09:54:30 [1,039ms] [Error] [gpu.foundation.plugin] The chosen activeGpu index is higher than the available GPUs. An unsupported driver might also prevent it from enumerating your devices properly.
|---------------------------------------------------------------------------------------------|
| Driver Version: 470.57 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA DGX Display | | | 4342 MB | 10de | 0 |
| | | | | | 1fb0 | 1fefbb2d.. |
|=============================================================================================|
| OS: Linux dgx-govtech-a100, Version: 5.4.0-81-generic
| Processor: AMD EPYC 7742 64-Core Processor | Cores: Unknown | Logical: 128
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 515835 | Free Memory: 501895
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2022-01-20 09:54:30 [1,039ms] [Error] [gpu.foundation.plugin] No device could be created. Some known system issues:
- The driver is not installed properly and requires a clean re-install.
- Your GPUs do not support RayTracing: DXR or Vulkan ray_tracing, or hardware is excluded due to performance.
- The driver cannot enumerate any GPU: driver, display or a docker issue. For Vulkan, test it with Vulkaninfo tool from Vulkan SDK, instead of nvidia-smi.
- For Ubuntu, it requires server-xorg-core 1.20.7+ and a display to work without --no-window.
- For Linux dockers, the setup is not complete. Install the latest driver, xServer and NVIDIA container runtime.
2022-01-20 09:54:30 [1,039ms] [Warning] [omni.gpu_foundation_factory.plugin] RT-capable GPU not found, switching to compatibility mode
2022-01-20 09:54:30 [1,039ms] [Error] [omni.gpu_foundation_factory.plugin] Failed to create GPU foundation!
2022-01-20 09:54:30 [1,039ms] [Error] [omni.kit.renderer.plugin] GPU Foundation is not initialized!
2022-01-20 09:54:30 [1,039ms] [Error] [omni.kit.imgui_renderer.plugin] Invalid device attached to the default window!
[1.077s] [ext: omni.ui-2.5.3] startup
[1.104s] [ext: omni.kit.mainwindow-0.0.0] startup
[1.108s] [ext: omni.kit.uiapp-0.0.0] startup
[1.109s] [ext: omni.usd.schema.physics-1.0.0] startup
[1.119s] [ext: omni.usd.schema.isaac-0.1.1] startup
[1.154s] [ext: omni.usd.schema.anim-0.0.0] startup
[1.187s] [ext: omni.timeline-1.0.2] startup
[1.191s] [ext: omni.hydra.scene_delegate-0.1.0] startup
[1.208s] [ext: carb.audio-0.1.0] startup
[1.218s] [ext: omni.kit.commands-0.0.0] startup
[1.224s] [ext: omni.usd-1.4.5] startup
[1.253s] [ext: omni.kit.asset_converter-1.1.43] startup
[1.279s] [ext: omni.usd.schema.physx-0.0.0] startup
[1.307s] [ext: omni.services.facilities.base-1.0.0] startup
[1.310s] [ext: omni.services.core-1.1.1] startup
[1.436s] [ext: omni.services.transport.client.base-1.1.0] startup
[1.439s] [ext: omni.services.transport.server.base-1.0.0] startup
[1.442s] [ext: omni.services.client-0.2.3] startup
[1.469s] [ext: omni.kit.agent-0.1.0] startup
[1.472s] [ext: omni.services.transport.server.http-1.0.0] startup
[1.556s] [ext: omni.services.transport.client.http_async-1.0.0] startup
[1.742s] [ext: omni.kit.thumbnails.images-0.1.0] startup
[1.748s] [ext: omni.kit.widget.path_field-2.0.3] startup
[1.753s] [ext: omni.kit.window.popup_dialog-2.0.4] startup
[1.759s] [ext: omni.kit.window.commands-0.1.1] startup
[1.771s] [ext: omni.kit.widget.browser_bar-2.0.2] startup
[1.776s] [ext: omni.kit.menu.utils-1.0.2] startup
[1.783s] [ext: omni.kit.search_core-1.0.0] startup
[1.788s] [ext: omni.kit.material.library-1.2.3] startup
[1.920s] [ext: omni.kit.menu.create-1.0.1] startup
[1.926s] [ext: omni.kit.widget.filebrowser-2.2.20] startup
[1.937s] [ext: omni.kit.context_menu-1.3.2] startup
[1.967s] [ext: omni.kvdb-0.0.0] startup
[1.973s] [ext: omni.kit.window.toolbar-1.2.1] startup
[1.984s] [ext: omni.localcache-0.0.0] startup
[1.989s] [ext: omni.convexdecomposition-1.3.14] startup
[1.994s] [ext: omni.usdphysics-1.3.14] startup
[1.995s] [ext: omni.renderer-rtx-0.0.0] startup
[1.998s] [ext: omni.kit.window.drop_support-1.0.0] startup
[2.003s] [ext: omni.hydra.rtx-0.1.0] startup
[2.020s] [ext: omni.physx-1.3.14-5.1] startup
[2.049s] [ext: omni.kit.numpy.common-0.1.0] startup
[2.080s] [ext: omni.kit.window.viewport-0.0.0] startup
2022-01-20 09:54:31 [2,057ms] [Error] [omni.kit.window.viewport.plugin] renderer::IRenderer is invalid
2022-01-20 09:54:31 [2,057ms] [Error] [omni.kit.window.viewport.plugin] Failed to prepare GPU Foundation
2022-01-20 09:54:31 [2,057ms] [Error] [omni.kit.window.viewport.plugin] Failed to startup required plugins
2022-01-20 09:54:31 [2,066ms] [Error] [carb.scripting-python.plugin] AttributeError: 'NoneType' object has no attribute 'get_mouse_event_stream'
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/extscore/omni.kit.window.viewport/omni/kit/viewport/scripts/context_menu.py(27): on_startup
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(143): _startup_ext
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(174): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2): <module>
2022-01-20 09:54:31 [2,066ms] [Error] [omni.ext.plugin] [ext: omni.kit.window.viewport-0.0.0] Failed to process python module extension in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/extscore/omni.kit.window.viewport/.'.
[2.108s] [ext: omni.isaac.dynamic_control-0.1.8] startup
[2.127s] [ext: omni.isaac.lula-0.1.0] startup
[2.146s] [ext: omni.kit.notification_manager-1.0.0] startup
[2.153s] [ext: omni.kit.widget.versioning-1.3.4] startup
[2.161s] [ext: omni.syntheticdata-0.1.6] startup
[2.247s] [ext: omni.isaac.core-0.1.12] startup
2022-01-20 09:54:31 [2,225ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.core. Error: No module named 'omni.kit.viewport'. Traceback:
Traceback (most recent call last):
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py", line 65, in import_module
return importlib.import_module(name)
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/python/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/__init__.py", line 10, in <module>
from omni.isaac.core.simulation_context.simulation_context import SimulationContext
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/__init__.py", line 9, in <module>
from omni.isaac.core.simulation_context.simulation_context import SimulationContext
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/simulation_context/simulation_context.py", line 14, in <module>
from omni.isaac.core.utils.viewports import set_camera_view
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/omni/isaac/core/utils/viewports.py", line 16, in <module>
import omni.kit.viewport
ModuleNotFoundError: No module named 'omni.kit.viewport'
2022-01-20 09:54:31 [2,225ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.isaac.core' in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/.' failed to load.
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2): <module>
2022-01-20 09:54:31 [2,225ms] [Error] [omni.ext.plugin] [ext: omni.isaac.core-0.1.12] Failed to process python module extension in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.core/.'.
[2.271s] [ext: omni.kit.window.filepicker-2.4.12] startup
[2.331s] [ext: omni.isaac.synthetic_utils-0.1.5] startup
2022-01-20 09:54:31 [2,306ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.synthetic_utils.tests. Error: No module named 'omni.isaac.core'. Traceback:
Traceback (most recent call last):
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py", line 65, in import_module
return importlib.import_module(name)
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/python/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/tests/__init__.py", line 10, in <module>
from .test_synthetic_utils import *
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/omni/isaac/synthetic_utils/tests/test_synthetic_utils.py", line 31, in <module>
from omni.isaac.core.utils.physics import simulate_async
ModuleNotFoundError: No module named 'omni.isaac.core'
2022-01-20 09:54:31 [2,306ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.isaac.synthetic_utils.tests' in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/.' failed to load.
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2): <module>
2022-01-20 09:54:31 [2,306ms] [Error] [omni.ext.plugin] [ext: omni.isaac.synthetic_utils-0.1.5] Failed to process python module extension in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_utils/.'.
[2.351s] [ext: omni.kit.widget.stage-2.6.8] startup
[2.362s] [ext: omni.isaac.synthetic_recorder-0.1.0] startup
2022-01-20 09:54:31 [2,335ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.synthetic_recorder. Error: No module named 'omni.isaac.synthetic_utils'. Traceback:
Traceback (most recent call last):
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py", line 65, in import_module
return importlib.import_module(name)
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/python/lib/python3.7/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
File "<frozen importlib._bootstrap>", line 983, in _find_and_load
File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 728, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/omni/isaac/synthetic_recorder/__init__.py", line 10, in <module>
from .extension import *
File "/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/omni/isaac/synthetic_recorder/extension.py", line 29, in <module>
from omni.isaac.synthetic_utils import visualization as vis
ModuleNotFoundError: No module named 'omni.isaac.synthetic_utils'
2022-01-20 09:54:31 [2,335ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.isaac.synthetic_recorder' in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/.' failed to load.
At:
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(169): startup_all_extensions_in_module
/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/kit/plugins/bindings-python/omni/ext/impl/_internal.py(225): startup_all_extensions_in_module
PythonExtension.cpp::startup()(2): <module>
2022-01-20 09:54:31 [2,335ms] [Error] [omni.ext.plugin] [ext: omni.isaac.synthetic_recorder-0.1.0] Failed to process python module extension in '/raid/home/ryan_lin/.local/share/ov/pkg/isaac_sim-2021.2.1/exts/omni.isaac.synthetic_recorder/.'.
[2.380s] [ext: semantics.schema.editor-0.2.2] startup
[2.387s] [ext: omni.kit.widget.graph-1.1.4] startup
[2.425s] [ext: omni.kit.stage_templates-1.0.6] startup
[2.433s] [ext: omni.kit.window.property-1.6.2] startup
[2.441s] [ext: omni.kit.window.extensions-1.0.2] startup
[2.452s] [ext: omni.kit.window.file-1.3.5] startup
[2.528s] [ext: omni.kit.widget.live-0.1.0] startup
[2.530s] [ext: omni.isaac.ui-0.1.2] startup
[2.539s] [ext: omni.kit.window.content_browser-2.4.10] startup
[2.584s] [ext: omni.kit.tool.asset_exporter-1.0.8] startup
[2.596s] [ext: omni.command.usd-1.0.1] startup
[2.602s] [ext: omni.isaac.unit_converter-0.1.1] startup
[2.609s] [ext: omni.kit.property.usd-3.11.4] startup
[2.653s] [ext: omni.kit.window.status_bar-0.1.0] startup
[2.655s] [ext: omni.kit.menu.edit-1.0.3] startup
[2.664s] [ext: omni.kit.usd_undo-0.1.0] startup
[2.670s] [ext: omni.kit.tool.collect-2.0.8] startup
[2.683s] [ext: omni.kit.widget.settings-1.0.0] startup
[2.691s] [ext: omni.kit.property.skel-1.0.0] startup
[2.699s] [ext: omni.physx.commands-1.3.14-5.1] startup
[2.705s] [ext: omni.debugdraw-0.1.0] startup
[2.712s] [ext: omni.kit.profiler.window-1.3.3] startup
[2.724s] [ext: omni.kit.property.camera-1.0.3] startup
[2.732s] [ext: omni.kit.property.geometry-1.2.0] startup
[2.741s] [ext: omni.kit.property.light-1.0.4] startup
[2.749s] [ext: omni.kit.window.preferences-1.1.5] startup
[2.807s] [ext: omni.physx.ui-1.3.14-5.1] startup
2022-01-20 09:54:32 [3,022ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.events.plugin.so!carbOnPluginStartup
2022-01-20 09:54:32 [3,024ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.tasking.plugin.so!std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (carb::tasking::Scheduler::*)(unsigned int, int, carb::cpp20::latch*), carb::tasking::Scheduler*, unsigned int, int, carb::cpp20::latch*> > >::_M_run()
2022-01-20 09:54:32 [3,028ms] [Fatal] [carb.crashreporter-breakpad.plugin] libpthread.so.0!funlockfile
2022-01-20 09:54:32 [3,028ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.physxui.plugin.so!carbOnPluginStartup
2022-01-20 09:54:32 [3,032ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!omniCoreStart
2022-01-20 09:54:32 [3,035ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!omniCoreStart
2022-01-20 09:54:32 [3,038ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!releaseFramework
2022-01-20 09:54:32 [3,042ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!releaseFramework
2022-01-20 09:54:32 [3,045ms] [Fatal] [carb.crashreporter-breakpad.plugin] libcarb.so!_init
2022-01-20 09:54:32 [3,048ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!void std::vector<std::string, std::allocator<std::string> >::_M_realloc_insert<std::string&>(__gnu_cxx::__normal_iterator<std::string*, std::vector<std::string, std::allocator<std::string> > >, std::string&)
2022-01-20 09:54:32 [3,051ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!carbOnPluginShutdown
2022-01-20 09:54:32 [3,055ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!char* std::string::_S_construct<char*>(char*, char*, std::allocator<char> const&, std::forward_iterator_tag)
2022-01-20 09:54:32 [3,058ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Rb_tree<std::string, std::string, std::_Identity<std::string>, std::less<std::string>, std::allocator<std::string> >::_M_erase(std::_Rb_tree_node<std::string>*)
2022-01-20 09:54:32 [3,061ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Rb_tree<std::string, std::string, std::_Identity<std::string>, std::less<std::string>, std::allocator<std::string> >::_M_erase(std::_Rb_tree_node<std::string>*)
2022-01-20 09:54:32 [3,064ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Rb_tree<std::string, std::string, std::_Identity<std::string>, std::less<std::string>, std::allocator<std::string> >::_M_erase(std::_Rb_tree_node<std::string>*)
2022-01-20 09:54:32 [3,068ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.ext.plugin.so!std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> >::operator=(std::_Hashtable<std::string, std::string, std::allocator<std::string>, std::__detail::_Identity, std::equal_to<std::string>, std::hash<std::string>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, true, true> > const&)
2022-01-20 09:54:32 [3,071ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!std::string __gnu_cxx::__to_xstring<std::string, char>(int (*)(char*, unsigned long, char const*, __va_list_tag*), unsigned long, char const*, ...)
2022-01-20 09:54:32 [3,074ms] [Fatal] [carb.crashreporter-breakpad.plugin] libomni.kit.app.plugin.so!_init
2022-01-20 09:54:32 [3,074ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
2022-01-20 09:54:32 [3,078ms] [Fatal] [carb.crashreporter-breakpad.plugin] libc.so.6!__libc_start_main
2022-01-20 09:54:32 [3,078ms] [Fatal] [carb.crashreporter-breakpad.plugin] kit!_init
Segmentation fault (core dumped)
I have also tried other GPU index (0 ~ 4) to launch
GPU 0 gave me this error
2022-01-20 10:05:01 [1,153ms] [Warning] [gpu.foundation.plugin] Raytracing is not supported on GPU: 0. Skipping this device.
and other indexes gave me this error
2022-01-20 10:06:16 [680ms] [Error] [gpu.foundation.plugin] The chosen activeGpu index is higher than the available GPUs. An unsupported driver might also prevent it from enumerating your devices properly.
Hi, it could be caused by incompatible drivers. Please try this datacenter drivers:
https://www.nvidia.com/Download/driverResults.aspx/182617/en-us