WinError 127 when loading extensions

[axel.goedrich]

Hello,

I recently encountered an issue that could be related to the new Isaac Sim version (2023.1.1). Initially, everything worked fine, but after closing and reopening the simulator, I started facing errors related to extensions using PyTorch. The same issue now also appears with older versions of Isaac Sim, suggesting that the root cause might be a change affecting the system more globally.

Here are some additional context details:

• At the time of the issue’s first occurrence, I was not running any programs besides VSCode, Omniverse, Outlook, MS Teams and my browser.
• I was working on a UI for an Isaac Sim extension (so nothing with pytorch).
• Isaac Sim was closed only for a brief period, just a few seconds.

Here’s a part of the error message I receive for each extension:

2023-12-20 12:10:33 [21,339ms] [Error] [omni.ext.plugin] [ext: omni.isaac.gain_tuner-0.1.3] Failed to startup python extension.
2023-12-20 12:10:34 [21,771ms] [Error] [omni.ext.python] OSError: [WinError 127] Die angegebene Prozedur wurde nicht gefunden. Error loading "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.wx64/torch-2-0-1/torch\lib\cublas64_11.dll" or one of its dependencies. 

Steps I’ve taken to troubleshoot:

• Verified that cublas64_11.dll is in the correct directory.
• Checked dependencies of cublas64_11.dll using ldd in Git Bash on Windows 10. All dependencies are correctly located.
• In the Isaac Sim Python environment, PyTorch imports successfully, and torch.cuda.is_available() returns true.
• Reinstalled Isaac Sim version 2023.1.1.

Despite these efforts, the issue persists. If anyone has experienced a similar problem or has any insights, your assistance would be greatly appreciated.

Thank you in advance for your help.

Kind regards

Axel

Here is the whole error message for reference:

2023-12-20 12:10:33 [21,337ms] [Error] [omni.ext._impl._internal] Failed to import python module omni.isaac.gain_tuner from c:\users\goedrichax\appdata\local\ov\pkg\isaac_sim-2023.1.1\exts\omni.isaac.gain_tuner. Error: [WinError 127] Die angegebene Prozedur wurde nicht gefunden. Error loading "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.wx64/torch-2-0-1/torch\lib\cublas64_11.dll" or one of its dependencies.. Traceback:
Traceback (most recent call last):
  File "c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\omni\ext\_impl\_internal.py", line 187, in _custom_importer
    return _import_public(ext_module.path, ext_module.name, reload_enabled)
  File "c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\carb\profiler\__init__.py", line 83, in wrapper
    r = f(*args, **kwds)
  File "c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\omni\ext\_impl\_internal.py", line 94, in _import_public
    module = import_module(module_name)
  File "c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\carb\profiler\__init__.py", line 83, in wrapper
    r = f(*args, **kwds)
  File "c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\omni\ext\_impl\custom_importer.py", line 76, in import_module
    return importlib.import_module(name)
  File "c:\users\goedrichax\appdata\local\ov\pkg\isaac_sim-2023.1.1\kit\python\lib\importlib\__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.gain_tuner/omni/isaac/gain_tuner/__init__.py", line 10, in <module>
    from .extension import *
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.gain_tuner/omni/isaac/gain_tuner/extension.py", line 21, in <module>
    from omni.isaac.core.articulations import Articulation
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/__init__.py", line 11, in <module>
    from omni.isaac.core.physics_context.physics_context import PhysicsContext
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/physics_context/__init__.py", line 9, in <module>
    from omni.isaac.core.physics_context.physics_context import PhysicsContext
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/physics_context/physics_context.py", line 15, in <module>
    from omni.isaac.core.utils.prims import get_prim_at_path, get_prim_path, is_prim_path_valid
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/utils/prims.py", line 24, in <module>
    from omni.isaac.core.utils.types import SDF_type_to_Gf
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/utils/types.py", line 12, in <module>
    import torch
  File "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.wx64/torch-2-0-1/torch/__init__.py", line 122, in <module>
    raise err
OSError: [WinError 127] Die angegebene Prozedur wurde nicht gefunden. Error loading "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.wx64/torch-2-0-1/torch\lib\cublas64_11.dll" or one of its dependencies.

2023-12-20 12:10:33 [21,338ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.isaac.gain_tuner' in 'c:\users\goedrichax\appdata\local\ov\pkg\isaac_sim-2023.1.1\exts\omni.isaac.gain_tuner' failed to load.

At:
  c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\omni\ext\_impl\_internal.py(213): startup
  c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\omni\ext\_impl\_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>

2023-12-20 12:10:33 [21,339ms] [Error] [omni.ext.plugin] [ext: omni.isaac.gain_tuner-0.1.3] Failed to startup python extension.
2023-12-20 12:10:34 [21,771ms] [Error] [omni.ext.python] OSError: [WinError 127] Die angegebene Prozedur wurde nicht gefunden. Error loading "c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.wx64/torch-2-0-1/torch\lib\cublas64_11.dll" or one of its dependencies.

At:
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/extscache/omni.pip.torch-2_0_1-2.0.2+105.1.wx64/torch-2-0-1/torch/__init__.py(122): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/utils/types.py(12): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/utils/prims.py(24): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/physics_context/physics_context.py(15): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/physics_context/__init__.py(9): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(992): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.core/omni/isaac/core/__init__.py(11): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap>(992): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.gain_tuner/omni/isaac/gain_tuner/extension.py(21): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/exts/omni.isaac.gain_tuner/omni/isaac/gain_tuner/__init__.py(10): <module>
  <frozen importlib._bootstrap>(241): _call_with_frames_removed
  <frozen importlib._bootstrap_external>(883): exec_module
  <frozen importlib._bootstrap>(703): _load_unlocked
  <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
  <frozen importlib._bootstrap>(1027): _find_and_load
  <frozen importlib._bootstrap>(1050): _gcd_import
  c:\users\goedrichax\appdata\local\ov\pkg\isaac_sim-2023.1.1\kit\python\lib\importlib\__init__.py(126): import_module
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.graph/omni/graph/core/_impl/_registration/extension_management.py(154): extension_management_factory
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.graph/omni/graph/core/_impl/_registration/register_python_ogn.py(70): register_python_ogn
  c:\users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/kernel/py\carb\profiler\__init__.py(85): wrapper
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.graph/omni/graph/core/_impl/registration.py(34): __init__
  c:/users/goedrichax/appdata/local/ov/pkg/isaac_sim-2023.1.1/kit/exts/omni.graph/omni/graph/core/_impl/extension.py(120): on_after_ext_enabled

[rthaker]

Hi @axel.goedrich - The error message that you’re encountering appears to suggest that there is a problem with loading the cublas64_11.dll library or one of its dependencies when trying to start a Python extension that uses PyTorch within Isaac Sim. This kind of error could be due to:

1. Missing dependencies: Although you’ve checked the dependencies of cublas64_11.dll using ldd, it’s possible that there are still runtime discrepancies or conflicts that aren’t being caught at that stage.

2. DLL conflicts: There could be a conflict with different versions of CUDA or cuBLAS that are installed on your system. This can happen if another application has updated these libraries or if there are multiple copies of the libraries that can be found by the system.

3. Environment variables: Changes to system environment variables, especially those related to CUDA (CUDA_PATH, CUDA_HOME), can affect the ability of Isaac Sim to find and load the correct libraries.

4. Isaac Sim cache: Sometimes corrupted cache files can cause strange issues, so clearing the cache of Isaac Sim might help.

5. PyTorch version mismatch: The version of PyTorch within Isaac Sim might not be compatible with the system CUDA or cuBLAS libraries. Ensure that the correct version of PyTorch that is compatible with the CUDA version installed on your system is being used within Isaac Sim.

6. System updates: An operating system or driver update could have caused changes that lead to the library not being found or loaded properly.

Here are a few additional troubleshooting steps you might consider:

1. Clear Isaac Sim’s cache and configuration files to ensure that there is no retained state causing the issue.

2. Verify that the latest compatible graphics drivers and CUDA toolkit are installed.

3. Check the environment variables for any changes that might affect the search paths for the required DLLs.

4. Try creating a fresh environment with the same PyTorch and CUDA versions as Isaac Sim expects and see if that works outside of Isaac Sim. If it does, there might indeed be some change in the application’s environment that is causing the issue.

5. If you have recently updated any software on your system (such as graphics drivers, CUDA toolkit, or other related libraries), try rolling back to the versions that were known to work.

Let us know if none of these suggestions work for you.

[axel.goedrich]

Hi @rthaker,

thank you for your suggestions. Here’s what I’ve done so far:

• I’ve cleared the cache.
• Updated the GPU drivers and also uninstalled, then reinstalled the specific version of the CUDA toolkit required by Isaac Sim.
• Uninstalled all Omniverse applications through the Omniverse Launcher.

Unfortunately, none of these steps fixed the problem. I also created a fresh environment with the same PyTorch and CUDA version, as you suggested and pytorch worked fine in this environment.

So as a last resort, I uninstalled every omniverse application (also the launcher) and manually deleted all folders related to omniverse installations. After a clean installation of the launcher and Isaac Sim, the problem was resolved.

In retrospect, perhaps just deleting the configuration files might have been enough, but I did not find them to be honest.

Kind regards

Axel