Issue with Livestream example when running IssacSim on docker container in the server

andypark · February 7, 2023, 6:23pm

Hi, I am trying to setup IssacSim on my remote workstation server with docker container. IssacSim is running headlessly in the docker container, and I am accessing it using Omniverse Streaming Client. I was able to run it with runheadless.native.sh script, but I am not able to do it via Python interface. I tried running Livestream.py example but I have errors that I don’t know how to resolve. Would you be able to help me out?

Below is part of log and I also attached the full log. Any suggestion would be greatly appreciated!

Loading user config located at: ‘/root/.local/share/ov/data/Kit/Isaac-Sim/2022.2/user.config.json’
[Info] [carb] Logging to file: /root/.nvidia-omniverse/logs/Kit/Isaac-Sim/2022.2/kit_20230207_103003.log
2023-02-07 15:30:03 [45ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.lidar] Extensions config ‘extension.toml’ doesn’t exist ‘/isaac-sim/exts/omni.drivesim.sensors.nv.lidar’ or ‘/isaac-sim/exts/omni.drivesim.sensors.nv.lidar/config’
2023-02-07 15:30:03 [45ms] [Warning] [omni.ext.plugin] [ext: omni.drivesim.sensors.nv.radar] Extensions config ‘extension.toml’ doesn’t exist ‘/isaac-sim/exts/omni.drivesim.sensors.nv.radar’ or ‘/isaac-sim/exts/omni.drivesim.sensors.nv.radar/config’
[0.218s] [ext: omni.stats-0.0.0] startup
[0.246s] [ext: omni.rtx.shadercache-1.0.0] startup
[0.257s] [ext: omni.assets.plugins-0.0.0] startup
[0.258s] [ext: omni.gpu_foundation-0.0.0] startup
2023-02-07 15:30:03 [244ms] [Warning] [carb] FrameworkImpl::setDefaultPlugin(client: omni.gpu_foundation_factory.plugin, desc : [carb::graphics::Graphics v2.11], plugin : carb.graphics-vulkan.plugin) failed. Plugin selection is locked, because the interface was previously acquired by:
[0.263s] [ext: carb.windowing.plugins-1.0.0] startup
2023-02-07 15:30:03 [245ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-02-07 15:30:03 [245ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
[0.264s] [ext: omni.kit.renderer.init-0.0.0] startup
2023-02-07 15:30:04 [300ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2023-02-07 15:30:04 [300ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.3]) (impl: carb.windowing-glfw.plugin)
2023-02-07 15:30:04 [300ms] [Error] [carb.glinterop.plugin] GLInteropContext::init: carb::windowing is not available
2023-02-07 15:30:04 [616ms] [Warning] [gpu.foundation.plugin] Raytracing is not supported on GPU: 1. Skipping this device.

|---------------------------------------------------------------------------------------------|
| Driver Version: 525.85.12 | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name | Active | LDA | GPU Memory | Vendor-ID | LUID |
| | | | | | Device-ID | UUID |
|---------------------------------------------------------------------------------------------|
| 0 | NVIDIA RTX A5000 | Yes: 0 | | 24810 MB | 10de | 0 |
| | | | | | 2231 | c32f96f6… |
|---------------------------------------------------------------------------------------------|
| 1 | NVIDIA T400 4GB | | | 4342 MB | 10de | 0 |
| | | | | | 1ff2 | 7effd56f… |
|=============================================================================================|
| OS: Linux WS1, Version: 5.15.0-58-generic
| Processor: AMD Ryzen Threadripper PRO 5945WX 12-Cores | Cores: Unknown | Logical: 24
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 64132 | Free Memory: 59926
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] IOMMU is enabled. Found 62 items in /sys/kernel/iommu_groups/.
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] On Linux only, CUDA and the display driver does not support IOMMU-enabled bare-metal PCIe peer to peer memory copy.
However, CUDA and the display driver does support IOMMU via VM pass through. As a consequence, users on Linux,
when running on a native bare metal system, should disable the IOMMU. The IOMMU should be enabled and the VFIO driver
be used as a PCIe pass through for virtual machines.
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin]
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] !!! Local system validation failed! Incorrect configuration detected.
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] Summary below. Details above.
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin]
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] - ECC: OK
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] - IOMMU: FAILED
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin]
2023-02-07 15:30:04 [722ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------

IssacSim-log-error-020723.log (49.4 KB)

kruzland · March 19, 2023, 3:52pm

Hi, did you manage to solve the problem?

andypark · March 20, 2023, 3:15am

Hi thanks for asking. No I have not been able to resolve the issue, and am still waiting to hear from the support team. Do you have any clue?

kruzland · March 20, 2023, 9:43am

Hi, I am not sure with the Linux stuff, my understanding is that there might be an incorrectly installed nvidia driver.
See the “incorrect configuration detected” part where it also says IOMMU: Failed.

So I assume the FAQ part 1) of troubleshooting linux (link) where you purge the nvidia drivers and reinstall them given their tutorial could help. I figured Id better wait for a word by the developer before freestyling with graphic’s drivers.

What Ubuntu distro are you running on both server and the client? Just curious.

I made a similar topic, see here..

andypark · March 20, 2023, 10:10am

Thanks for sharing some suggestions. My workstation is running Ubuntu 22.04, and that is why we are using docker to set up 20.04 in the image for IsaacSim which complicated the setup. Since the simulation runs fine with headless native server script. I did not think I had to play with the graphics driver. I hope the developer says something soon. Thanks!

kruzland · March 20, 2023, 12:05pm

Well, I suppose the headless starts without a problem because the RTX does not need to run, strictly speaking. What I think is happening is that RTX needs some special driver blah blah, that you currently do not have set up. When you run it in headless mode, it perhaps does not start RTX right away and might produce errors later. RTX should be the service allowing the usage of RTX cores on certain RTX-enabled NVIDIA cards, which presumably needs a custom driver since they state they hate proprietary drivers :DD.

See their recommendation:

Do NOT install PPA drivers. PPA drivers are packaged with Linux distribution, and installed with add-apt or Software & Updates. These drivers are not easy to uninstall or clean up. It requires purging them and cleaning up Vulkan ICD files manually.

I have already ppa driver installed, I did it because it was easy to do with inbuilt Software & Updates > NVIDIA driver metapackage from nvidia-driver-515 (proprietary), which might be the cause for the problem.

Anyway, are you able to run the IsaacSim in the docker, also accessing the nvidia card in the docker? On the link to the FAQ above, there is the vulkaninfo, try if your workstation environment inside of the docker, while calling vulkaninfo as instructed in the FAQ, shows the system graphics cards. (mine does)

hhansen · March 31, 2023, 7:26pm

Adding to the chorus here that this is a problem I’m facing. I’m running an Isaac Sim instance inside of a docker on an Ubuntu 20.04 Google Cloud VM, following the instructions from the cloud deployment page but attempting to launch the sim from python bindings. I have attached my logs (at the bottom they’re interspersed with a ROS process, but that’s just a little bit at the end), which show a different hardware and driver configuration that the one of the original poster but has similar problems.

Would really appreciate any help, as this is a blocker for my organization. Thank you!

isaac_output.txt (29.7 KB)