Isaac Sim Startup Error (cannot load cublas dll)

When I launch Isaac Sim from the Omniverse launcher, I get the following error:

2022-06-29 09:00:01 [5,663ms] [Error] [omni.ext.impl._internal] Failed to import python module omni.isaac.core from c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core. Error: [WinError 1114] A dynamic link library (DLL) initialization routine failed. Error loading "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.ml_archive\pip_prebundle\torch\lib\cublas64_11.dll" or one of its dependencies.. Traceback:
Traceback (most recent call last):
  File "c:/nvidia omniverse/pkg/isaac_sim-2022.1.0/kit/plugins/bindings-python\omni\ext\impl\_internal.py", line 151, in _custom_importer
    return _import_public(ext_module.path, ext_module.name)
  File "c:/nvidia omniverse/pkg/isaac_sim-2022.1.0/kit/plugins/bindings-python\omni\ext\impl\_internal.py", line 78, in _import_public
    module = import_module(module_name)
  File "c:/nvidia omniverse/pkg/isaac_sim-2022.1.0/kit/plugins/bindings-python\carb\profiler\__init__.py", line 78, in wrapper
    r = f(*args, **kwds)
  File "c:/nvidia omniverse/pkg/isaac_sim-2022.1.0/kit/plugins/bindings-python\omni\ext\impl\custom_importer.py", line 76, in import_module
    return importlib.import_module(name)
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\kit\python\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core\omni\isaac\core\__init__.py", line 10, in <module>
    from omni.isaac.core.simulation_context.simulation_context import SimulationContext
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core\omni\isaac\core\simulation_context\__init__.py", line 9, in <module>
    from omni.isaac.core.simulation_context.simulation_context import SimulationContext
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core\omni\isaac\core\simulation_context\simulation_context.py", line 24, in <module>
    import omni.isaac.core.utils.torch as torch_utils
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core\omni\isaac\core\utils\torch\__init__.py", line 1, in <module>
    from .rotations import *
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core\omni\isaac\core\utils\torch\rotations.py", line 5, in <module>
    import torch
  File "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.ml_archive\pip_prebundle\torch\__init__.py", line 127, in <module>
    raise err
OSError: [WinError 1114] A dynamic link library (DLL) initialization routine failed. Error loading "c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.ml_archive\pip_prebundle\torch\lib\cublas64_11.dll" or one of its dependencies.

2022-06-29 09:00:01 [5,665ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.isaac.core' in 'c:\nvidia omniverse\pkg\isaac_sim-2022.1.0\exts\omni.isaac.core' failed to load.

Apparently, he cublas64_11.dll fails to load when importing torch in some files of the isaac core extension. Isaac Sim starts nevertheless and I can use it to some extent, but I can’t use, for example, writers as they depend on the core module.
I’m working on a Windows 10 operating system with a RTX 3090. Here is the first line of the output of nvidia-smi:
NVIDIA-SMI 511.79 Driver Version: 511.79 CUDA Version: 11.6

I’ve got CUDA 11.6 installed, as I need it for other applications. Could there be a problem with conflicting versions? If yes, how could I resolve this?

Using a regular python install (or anaconda) if you install
torch==1.11.0+cu113 does it import correctly?

Yes, if I use anaconda (python 3.7 environment, conda install pytorch==1.11.0 cudatoolkit=11.3 -c pytorch) and open a python shell, I can use import torch without problems.

I’m not sure what the issue is currently, will see if I can find a better way to reproduce it on our end

@ole.wegen does this issue persist with 2022.1.1?

Unfortunately, yes. However, maybe it has something to do with my Windows version. I will update it and try again.

We can reproduce this on Windows server 2019 but its not clear if its an older windows SDK issue being incompatible with pytorch somehow.
If you run the winver command what version does it report?

Windows 10 Version 1809 (OS Build 17763.316)

Is updating the windows version an option?

interestingly the windows server version we can reproduce this on is also a very similar version to yours
Windows Server 2019 version 1809 (OS Build 17762.3046)

Unfortunately I’m bound by institute guidelines regarding the use of specific OS versions, which makes updating windows challenging in my specific context. Nevertheless, I will see what I can do.

Hi @ole.wegen - Can you please confirm if this issue still happening with latest Isaac Sim release? (Isaac Sim 2022.2.1)

Still does not work. Now the cudart_64_110.dll fails to load and Isaac Sim even crashes on startup.
Unfortunately I was not able to upgrade my Windows version due to internal issues.

Hi @ole.wegen - You can alternatively try docker container if it helps in your case.
https://docs.omniverse.nvidia.com/isaacsim/2022.2.1/install_container.html