IsaacSim livestreaming container with 525.125 driver

kzaitsev · October 19, 2023, 9:10pm

I’m running a Singularity container on a server with RTX 2080 Ti, CUDA 12.0, and the driver version 525.125.06. Having observed the Isaac Sim Headless Native App is loaded., I open the Omniverse livestreaming client and connect via an IP address of the server. After several seconds of waiting, the error code I observed was:

The only error in the logs is due to the failed creation of the NGX context:

2023-10-19 20:54:08 [87,945ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register LdrColor, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,945ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register DistanceToImagePlane, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,945ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register DistanceToCamera, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,945ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register Camera3dPosition, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,945ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register Normal, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,945ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register CrossCorrespondence, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register TargetMotion, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register InstanceIdSegmentation, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register InstanceSegmentation, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register SemanticSegmentation, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register SemanticIdSegmentation, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register BoundingBox2DTight, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register BoundingBox2DLoose, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register BoundingBox3D, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register <omni.kit.viewport.menubar.core.menu_item.viewport_menu_separator.ViewportMenuSeparator object at 0x7fcd160cd230>, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register Clear All, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [87,946ms] [Info] [omni.kit.viewport.menubar.core.viewport_menu_model] register Show Window, order: -1, parent: SyntheticData
2023-10-19 20:54:08 [88,034ms] [Warning] [omni.client.plugin]  117965: omniclient: Cannot stat root directory (URI: plus.40x40.png)
2023-10-19 20:54:08 [88,039ms] [Warning] [omni.client.plugin]  117956: omniclient: Cannot stat root directory (URI: save.40x40.png)
2023-10-19 20:54:08 [88,045ms] [Warning] [omni.client.plugin]  118018: omniclient: Cannot stat root directory (URI: settings.40x40.png)
2023-10-19 20:54:08 [88,073ms] [Info] [omni.kit.menu.utils.scripts.utils] omni.kit.menu.utils.refresh_menu_items Tools
2023-10-19 20:54:08 [88,123ms] [Warning] [omni.client.plugin]  117939: omniclient: Cannot stat root directory (URI: physics_placement_toolbar.56x56.png)
2023-10-19 20:54:08 [88,130ms] [Warning] [omni.client.plugin]  117996: omniclient: Cannot stat root directory (URI: paint_dark.56x56.png)
2023-10-19 20:54:08 [88,257ms] [Info] [carb] Initializing plugin: omni.gpucompute-cuda.plugin (interfaces: [omni::gpucompute::GpuCompute v0.1]) (impl: omni.gpucompute-cuda.plugin)
2023-10-19 20:54:08 [88,259ms] [Info] [carb] Plugin omni.anim.curve.plugin is already a dependency of omni.kit.app.plugin; not changing unload order
2023-10-19 20:54:08 [88,261ms] [Info] [carb] Initializing plugin: omni.hydra.scene_delegate.plugin (interfaces: [carb::fastcache::FastCache v0.4],[carb::fastcache::FastCacheHydra v0.2],[omni::hydra::IDebug v0.1],[omni::hydra::IOmniHydra v2.0],[omni::hydra::ISceneDelegateFactory v0.1]) (impl: omni.hydra.scene_delegate.plugin)
2023-10-19 20:54:08 [88,793ms] [Info] [carb.graphics-vulkan.plugin] Active GPU: NVIDIA GeForce RTX 2080 Ti
2023-10-19 20:54:09 [88,870ms] [Info] [carb] Initializing plugin: omni.gpucompute-d3dvk.plugin (interfaces: [omni::gpucompute::ComputeCompiler v0.1],[omni::gpucompute::GpuCompute v0.1]) (impl: omni.gpucompute-d3dvk.plugin)
2023-10-19 20:54:09 [88,870ms] [Info] [carb.flowusd.plugin] Attaching Flow USD plugin: 9223001
2023-10-19 20:54:09 [88,882ms] [Info] [carb] Initializing plugin: carb.volume.plugin (interfaces: [carb::volume::IVolume v0.1]) (impl: carb.volume.plugin)
2023-10-19 20:54:09 [88,925ms] [Info] [omni.usd.audio] refreshing the USD scene state
2023-10-19 20:54:09 [88,960ms] [Info] [carb.audio.output] {speakers = 0x00000003, map = 0x7ffeaecf1260}
2023-10-19 20:54:09 [88,963ms] [Info] [omni.usd.audio] initial audio volumes: (1.00, 1.00, 1.00)
2023-10-19 20:54:09 [89,156ms] [Info] [carb] Initializing plugin: carb.scenerenderer-rtx.plugin (interfaces: [carb::renderer::SceneRenderer v6.1]) (impl: carb.scenerenderer-rtx.plugin)
2023-10-19 20:54:09 [89,163ms] [Info] [carb] Initializing plugin: carb.scenerenderer-iray.plugin (interfaces: [carb::renderer::SceneRenderer v6.1]) (impl: carb.scenerenderer-iray.plugin)
2023-10-19 20:54:09 [89,192ms] [Info] [gpu.foundation.plugin] The list of unsupported driver ranges being verified with known bugs (RTX):
[0.0, 470.121)
[495.0, 510.58)
[515.0, 515.17)
The recommended drivers: 525.85.05 (GameReady, Studio, RTX/Quadro), 525.85.12 (Grid/vGPU)
The latest driver version may also work but is not fully tested.
2023-10-19 20:54:09 [89,192ms] [Info] [carb] Initializing plugin: rtx.multigpumanager.plugin (interfaces: [rtx::multigpumanager::MultiGpuManager v0.8]) (impl: rtx.multigpumanager.plugin)
2023-10-19 20:54:09 [89,195ms] [Info] [carb] Initializing plugin: rtx.scenedb.plugin (interfaces: [rtx::scenedb::SceneDb v1.2]) (impl: rtx.scenedb.plugin)
2023-10-19 20:54:09 [89,212ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/database/DirtyPageCopy.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,213ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/database/DirtyElementsCopy.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/SparseMemCopy.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/utils/PrefixScanBuffer.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/utils/PrefixScanBuffer.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/utils/PrefixScanBuffer.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/utils/PrefixScanTexture.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/utils/PrefixScanTexture.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,249ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/utils/PrefixScanTexture.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLong.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,251ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLongConversion.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,277ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLongConversion.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,277ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLongConversion.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,278ms] [Info] [carb.graphics-vulkan.plugin] plugins rev1 path: /isaac-sim/kit/extscore/omni.gpu_foundation/bin/deps/rev1
2023-10-19 20:54:09 [89,278ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingLatLongConversion.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,278ms] [Info] [carb] Initializing plugin: carb.graphics_optional.plugin (interfaces: [carb::graphics::IOptionalFeatures v2.0]) (impl: carb.graphics_optional.plugin)
2023-10-19 20:54:09 [89,278ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/filters/LowPass.cs.hlsl' does not exist
2023-10-19 20:54:09 [89,279ms] [Info] [carb.graphics-vulkan.plugin] carb.graphics optional features library path: /isaac-sim/kit/extscore/omni.gpu_foundation/bin/deps
2023-10-19 20:54:09 [89,281ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingApproxIBL.cs.hlsl' does not exist
2023-10-19 20:54:09 [89,288ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingApproxIBL.cs.hlsl' does not exist
2023-10-19 20:54:09 [89,289ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingApproxIBL.cs.hlsl' does not exist
2023-10-19 20:54:09 [89,289ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/lighting/EnvironmentPreprocessingApproxIBL.cs.hlsl' does not exist
2023-10-19 20:54:09 [89,289ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/TranslateInstancePosition.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,290ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/ValidateSbt.compute.hlsl' does not exist
2023-10-19 20:54:09 [89,290ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/ProceduralAabb.comp.hlsl' does not exist
2023-10-19 20:54:09 [89,290ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/Skinning.comp.hlsl' does not exist
2023-10-19 20:54:09 [89,290ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/SkinNormal.comp.hlsl' does not exist
2023-10-19 20:54:09 [89,291ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/NormalGeneration.comp.hlsl' does not exist
2023-10-19 20:54:09 [89,291ms] [Info] [carb] file '/isaac-sim/kit/shaders/rtx/scenedb/VertexPulling.comp.hlsl' does not exist
2023-10-19 20:54:09 [89,296ms] [Info] [carb.graphics-vulkan.plugin] Failed to initialize NGX on GPU: NVIDIA GeForce RTX 2080 Ti. Optional DLSS feature will be disabled for it.
2023-10-19 20:54:09 [89,296ms] [Error] [carb.graphics-vulkan.plugin] Could not get NGX parameters block because NGX isn't enabled.
2023-10-19 20:54:09 [89,296ms] [Error] [carb.graphics-vulkan.plugin] Failed to create NGX context.
2023-10-19 20:54:09 [89,296ms] [Warning] [carb.scenerenderer-rtx.plugin] Failed to create NGX context.

The original docker container works in a setup of 515.65.01 driver with CUDA 11.7 on a local machine. Could you help to troubleshoot the NGX error?

Sheikh_Dawood · October 19, 2023, 9:34pm

Hi. We currently do not support Singularity yet.
Can you provide instructions on how you setup and run the container?

Does nvidia-smi works in that container?

kzaitsev · October 20, 2023, 5:57am

nvidia-smi works, its output is:

I build it from an official image:

docker save nvcr.io/nvidia/isaac-sim:2022.2.1 -o isaacsim.tar
singularity build --sandbox isaacsim docker-archive://isaacsim.tar

and start the following way:

singularity exec --nv --home /user/isaacsim/home \
    --bind /user/isaacsim/cache/kit:/isaac-sim/kit/cache/Kit:rw \
    --bind /user/isaacsim/cache/ov:/root/.cache/ov:rw \
    --bind /user/isaacsim/cache/pip:/root/.cache/pip:rw \
    --bind /user/isaacsim/cache/glcache:/root/.cache/nvidia/GLCache:rw \
    --bind /user/isaacsim/cache/computecache:/root/.nv/ComputeCache:rw \
    --bind /user/isaacsim/logs:/root/.nvidia-omniverse/logs:rw \
    --bind /user/isaacsim/data:/root/.local/share/ov/data:rw \
    --bind /user/isaacsim/documents:/root/Documents:rw \
    isaacsim sh

kzaitsev · October 20, 2023, 9:44am

With 515.65.01 and CUDA 11.7, the same container launched on the localhost allows connecting to it via Omniverse Livestreaming. Although the NGX error remains.
The docker setup runs cleanly.

Sheikh_Dawood · October 23, 2023, 3:24pm

Thanks. Does the livestreaming work with the 515 drivers?

I am not sure if Vulkan is setup on our container to support Singularity. This could be the cause of the NGX errors.

Can you try install and run vulkaninfo?

Try this old topic too.

Topic		Replies	Views
Isaac Sim container failed to load (native & WebRTC) livestreaming server due to Net Stream Creation failed Isaac Sim	10	1913	August 6, 2024
Isaac Sim (2021.1.1) Wont Launch (Omniverse 1.3.4) Isaac Sim aws , setup	23	4142	April 5, 2024
Issues with starting Isaac Sim headless in a docker container and Workstation installation Isaac Sim containers , installation , setup	32	3795	March 4, 2024
Error occured during the installation of IsaaC SIM Isaac Sim setup	10	2828	April 5, 2024
Isaac Sim Headless native fails to initialize streaming Isaac Sim gstreamer	3	180	November 5, 2024
Isaac Sim / Windows Isaac Sim setup	12	5851	June 18, 2022
What's the role of NGX? Isaac Sim isaac-sim-v4-2-0	13	149	November 1, 2024
Simulation App Error with Isaac Sim 2023.1.1 Isaac Sim	3	625	June 11, 2024
Could not select device driver "" with capabilities: [[gpu]] Isaac Sim containers , setup	12	13062	April 5, 2024
Issues abot Installing Isaac Sim on Ubuntu20.04 on laptop Isaac Sim isaacsim	6	1024	April 5, 2024

IsaacSim livestreaming container with 525.125 driver

Related topics