I’m trying to run an Issac Sim RL training on an HPC. I’ve been using the Issac Lab repo: Isaac Lab.
The training should be using Isaac Sim in headless mode, but I get this error:
2024-08-02 14:13:36 [2,290ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-08-02 14:13:36 [2,290ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-08-02 14:13:36 [2,291ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-08-02 14:13:36 [2,291ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-08-02 14:13:36 [2,291ms] [Error] [omni.appwindow.plugin] Failed to acquire IWindowing interface
followed immediately by a Python crash:
[2.315s] [ext: omni.kit.renderer.core-1.0.1] startup
Fatal Python error: Segmentation fault
Current thread 0x00007f734c319b80 (most recent call first):
File "/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py", line 350 in _start_app
File "/isaac-sim/exts/omni.isaac.kit/omni/isaac/kit/simulation_app.py", line 197 in __init__
File "/isaac-sim/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 66 in __init__
File "/workspace/isaaclab/omniisaacgymenvssitcore/omniisaacgymenvs/scripts/rlgames_train.py", line 98 in parse_hydra_configs
File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 160 in run_job
File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 98 in run
File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 378 in <lambda>
File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 211 in run_and_report
File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 377 in _run_hydra
File "/isaac-sim/kit/python/lib/python3.10/site-packages/hydra/main.py", line 48 in decorated_main
File "/workspace/isaaclab/omniisaacgymenvssitcore/omniisaacgymenvs/scripts/rlgames_train.py", line 143 in <module>
Extension modules: yaml._yaml, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, torch._C, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, google.protobuf.pyext._message, psutil._psutil_linux, psutil._psutil_posix (total: 24)
/isaac-sim/python.sh: line 41: 33 Segmentation fault (core dumped) $python_exe "$@" $args
There was an error running python
Here’s the line in simulation_app.py where it appears to be crashing:
# get the effective uid of this process, if its root, then we automatically add the allow root flag
# if the flag is already in unknown_args, we don't need to add it again.
if sys.platform.startswith("linux") and os.geteuid() == 0 and "--allow-root" not in unknown_args:
args.append("--allow-root")
I’m running a custom RL env using OmniIsaacGymEnvs and rl_games. It does run OK on my local machine, outside of the container.
I have some questions:
- Does the error “Failed to acquire IWindowing interface” mean that I am not successfully running headlessly?
- Is the IWindowing error what’s crashing the sim?
- Any other ideas as to what I may be doing wrong here.
Thanks,
Matt