Running OIGE or python standalone examples cause raytracing shaders to recompile every time

Omniverse Isaac Sim
1

Continuing the discussion from Running examples as Standalone Python takes exaggerated time in Windows 11:

Running python scripts which use omni.isaac.sim.python or omni.isaac.sim.gym.python kits cause raytracing shaders to recompile every time which takes a long time. Running Isaac Sim normally via launcher uses cached shaders and launches very quickly. I have Nucleus and Cache connected and working. When I run examples from Isaac Sim GUI (from launcher) everything loads quickly.

This happens when I run
PYTHON_PATH .\standalone_examples\api\omni.isaac.franka\stacking.py
or from OIGE
PYTHON_PATH scripts/rlgames_train.py task=Cartpole

From logs I see ray tracing shaders compilation which is the cause of the delay:

2024-02-08 13:42:25 [17,450ms] [Info] [omni.usd] [MBVH]: raytracingMotion/enable: OFF, for indexForEngineInstance: 0
2024-02-08 13:42:55 [47,493ms] [Info] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders for RTX renderer main raytracing pipeline by GPU driver: 29 seconds so far
2024-02-08 13:43:25 [77,493ms] [Info] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders for RTX renderer main raytracing pipeline by GPU driver: 59 seconds so far
2024-02-08 13:43:55 [107,493ms] [Info] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders for RTX renderer main raytracing pipeline by GPU driver: 89 seconds so far
2024-02-08 13:44:25 [137,493ms] [Info] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders for RTX renderer main raytracing pipeline by GPU driver: 119 seconds so far
2024-02-08 13:44:55 [167,494ms] [Info] [gpu.foundation.plugin] Waiting for compilation of ray tracing shaders for RTX renderer main raytracing pipeline by GPU driver: 149 seconds so far
2024-02-08 13:45:23 [195,421ms] [Info] [gpu.foundation.plugin] Ray tracing shader compilation for RTX renderer main raytracing pipeline finished after 177 seconds

Steps already tried:

  • Reinstall Isaac Sim
  • Clear/remove shader cache
  • Restart system multiple times

My environment:
Isaac Sim version: 2023.1.1
Windows 10
RTX 4060 GPU with drivers 546.09 (I cannot reinstall as I don’t have admin rights on my work laptop)

2

I seem to have fixed the issue by adding “–vulkan” argument. To use it when running OIGE or Python scripts, I modified _start_app method in

isaac_sim-2023.1.1\exts\omni.isaac.kit\omni\isaac\kit\simulation_app.py

def _start_app(self) -> None:
      ...
    if sys.platform.startswith("linux") and os.geteuid() == 0 and "--allow-root" not in unknown_args:
        args.append("--allow-root")
    
    args.append("--vulkan") # <- I added this line
    
    # pass all extra arguments onto the main kit app
    print("Starting kit application with the following args: ", args)
    ...

I’m not sure what this argument does and why the problem only happens when Isaac Sim is launched by Python scripts and not when launched via Omniverse Launcher. Also, are there drawbacks when using --vulkan?

4

Hi. The shader cache may have different locations when running from Python instead of the Launcher. The path should be in the log file.
Usually, clearing the shader cache and re-running should solve most issues. The shader caching issue could also be caused by other configs or driver version too.

By default, Isaac Sim on Windows uses DirectX to render but enabling the “–vulkan” flag will force it to use the Vulkan API instead (Note: this Vulkan runtime library is included in the NVIDIA GPU drivers). Vulkan is used by default on Linux.
There are some differences between Vulkan and DirectX rendering paths but we have not seen any major issues. DirectX is preferred natively on Windows.

5

Hi. I found following cache folders from the log:

2024-02-08 17:37:02 [453ms] [Info] [omni.gpu_foundation_factory.impl.foundation_extension] appShaderCachePathList: c:/users/user/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.rtx.shadercache.d3d12/cache/shadercache, c:/users/user/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.rtx.shadercache.vulkan/cache/shadercache
2024-02-08 17:37:02 [453ms] [Info] [omni.gpu_foundation_factory.impl.foundation_extension] appShaderCachePath: c:/users/user/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.gpu_foundation/cache/shadercache/vulkan
2024-02-08 17:37:02 [453ms] [Info] [omni.gpu_foundation_factory.impl.foundation_extension] shaderCachePath: c:/users/user/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/cache/shadercache
2024-02-08 17:37:02 [453ms] [Info] [omni.gpu_foundation_factory.impl.foundation_extension] driverAppShaderCachePath: c:/users/user/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.gpu_foundation/cache/nv_shadercache
2024-02-08 17:37:02 [453ms] [Info] [omni.gpu_foundation_factory.impl.foundation_extension] driverShaderCachePath: c:/users/user/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/cache/nv_shadercache

I tried to clear these caches by renaming them but that lead to errors when running. I removed the whole isaac sim install folder (also removing these cache folders) when I reinstalled Isaac sim, thus the shader cache problem with DirectX might be due to my drivers.

1 Like
6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.