Extremely Slow Loading Times - Isaac Sim 2023.1.0

james_usevitch · November 9, 2023, 11:55pm

I just installed Isaac Sim 2023.1.0, and am experiencing very slow loading times for the tutorials and when creating worlds. Loading the Franka example hangs for almost 5 minutes before the robot and room appear. Loading the simple room environment into a new stage also takes several minutes.

The computer I am using has a 32-core Threadripper with 256 GB of RAM and two 4090 GPUs, which I believe is more than sufficient for running the simulator. Will these slow loading times be typical for using Isaac Sim? The long waiting times are a bit of a pain, and I would love to shorten them if at all possible.

(I’m running Isaac Sim on Ubuntu 22.04. My NVIDIA driver version is 535.113.01.)

sujitvasanth · November 10, 2023, 1:30pm

I had the same issue…the problem is server timeout settings… specifically you can launch a local nucleus server or alter the nucleus server code
please see explanation here

github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs

Cartpole is slow in 2023.1.0

opened 09:11AM - 29 Oct 23 UTC

sujitvasanth

I just upgraded to Isaac Sim 2023.1.0 and therefore recloned the omnisaacgymenvs… repo and reinstalled omniisaacgym through git as per the repo instructions. It does work but it has some strange/slow behaviour. I get some omniverse gui windows transiently popping up and then omniverse hangs for 1-2 minutes at this point before running the rl example. I have taken a screenshot of the hanging: ![Screenshot from 2023-10-29 09-01-05](https://github.com/NVIDIA-Omniverse/OmniIsaacGymEnvs/assets/18464444/6822983d-91eb-4664-8c15-2cb9559ebdce) the wanings in the screen shot are ``` Sim Device: GPU Task Device: cuda:0 RL device: cuda:0 2023-10-29 09:00:04 [16,119ms] [Warning] [omni.client.python] Detected a blocking function. This will cause hitches or hangs in the UI. Please switch to the async version: File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/scripts/rlgames_train.py", line 150, in <module> File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/main.py", line 94, in decorated_main File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 457, in _run_app File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 220, in run_and_report File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/utils.py", line 458, in <lambda> File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/_internal/hydra.py", line 119, in run File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/python/lib/python3.10/site-packages/hydra/core/utils.py", line 186, in run_job File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/scripts/rlgames_train.py", line 121, in parse_hydra_configs File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/utils/task_util.py", line 103, in initialize_task File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/envs/vec_env_rlgames.py", line 47, in set_task File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.gym/omni/isaac/gym/vec_env/vec_env_base.py", line 126, in set_task File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/world/world.py", line 276, in reset File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/tasks/cartpole.py", line 65, in set_up_scene File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/tasks/cartpole.py", line 83, in get_cartpole File "/home/sujit/Desktop/OmniIsaacGymEnvs2023/omniisaacgymenvs/robots/articulations/cartpole.py", line 54, in __init__ File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 504, in get_assets_root_path File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/exts/omni.isaac.core/omni/isaac/core/utils/nucleus.py", line 191, in check_server File "/home/sujit/.local/share/ov/pkg/isaac_sim-2023.1.0/kit/extscore/omni.client/omni/client/__init__.py", line 161, in stat ``` then after hanging for 1-2 minutes, i get this error and cartpole starts normally at the normal speed ``` 2023-10-29 09:02:05 [137,263ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found 2023-10-29 09:02:07 [139,036ms] [Warning] [omni.isaac.core.utils.nucleus] /persistent/app/omniverse/mountedDrives setting not found ~*~*~* ~*~*~* Direct GPU Helper: ~*~*~* 512 articulations, maxLinks=3 ~*~*~* self.seed = 42 Started to train Exact experiment name requested from command line: Cartpole Box(-1.0, 1.0, (1,), float32) Box(-inf, inf, (4,), float32) seq_length: 4 current training device: cuda:0 build mlp: 4 RunningMeanStd: (1,) RunningMeanStd: (4,) [2023-10-29 09:02:09] Running RL reset fps step: 30103 fps step and policy inference: 27797 fps total: 25915 epoch: 1/100 frames: 0 fps step: 35992 fps step and policy inference: 33289 fps total: 30690 epoch: 2/100 frames: 8192 fps step: 34550 fps step and policy inference: 31696 fps total: 29390 epoch: 3/100 frames: 16384 fps step: 34697 fps step and policy inference: 31565 fps total: 29357 epoch: 4/100 frames: 24576 fps step: 34479 fps step and policy inference: 31280 fps total: 29116 epoch: 5/100 frames: 32768 fps step: 34432 fps step and policy inference: 31383 fps total: 29321 epoch: 6/100 frames: 40960 fps step: 34310 fps step and policy inference: 31217 fps total: 29131 epoch: 7/100 frames: 49152 fps step: 34603 fps step and policy inference: 31570 fps total: 29363 epoch: 8/100 frames: 57344 fps step: 33794 fps step and policy inference: 30878 fps total: 28656 epoch: 9/100 frames: 65536 fps step: 33828 fps step and policy inference: 30696 fps total: 28656 epoch: 10/100 frames: 73728 fps step: 33990 fps step and policy inference: 31037 fps total: 28968 epoch: 11/100 frames: 81920 fps step: 34108 fps step and policy inference: 31240 fps total: 29119 epoch: 12/100 frames: 90112 ``` my system is ubuntu 20.04, RTX 3090, 64 Gb RAM How can I resolve the system hanging? also what is with the gui windows appearing for import and also the simulation window is not the same as 2021.2.1, why? Opening files directly in Isaac Sim 2023.1.0 works fine and it has fixed issues I was having with usd robot file errors so I definitely need to upgrade

james_usevitch · November 11, 2023, 1:23am

Thank you! Your solution worked for me–startup time is now under 10 seconds.

For those who come across this post later, the solution can be found at this link.

system · November 25, 2023, 1:24am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.