Failed to import python module - pip dependencies

[gavin.stevens]

I am trying to create an extension that can connect to a Microsoft Azure Instance and load resources into OmniVerse. I’m having some trouble importing the azure.identity library on module load. It’s pretty easy to reproduce this.

Steps to reproduce:
1.; Install Azure Python libraries:
pip install azure-mgmt-resource
pip install azure-identity

2. add these libraries to the extension.toml:

...
[python.pipapi]
requirements = [
    "azure-identity",
    "azure-mgmt-resource"
]

use_online_index = true
...

then in a python script (test.py) enter:

# Import the needed credential and management objects from the libraries.
from azure.identity import AzureCliCredential
from azure.mgmt.resource import ResourceManagementClient
import os

# Acquire a credential object using CLI-based authentication.
credential = AzureCliCredential()

# Retrieve subscription ID from environment variable.
subscription_id = os.environ["AZURE_SUBSCRIPTION_ID"]

# Obtain the management object for resources.
resource_client = ResourceManagementClient(credential, subscription_id)

# Retrieve the list of resource groups
group_list = resource_client.resource_groups.list()

# Show the groups in formatted output
column_width = 40

print("Resource Group".ljust(column_width) + "Location")
print("-" * (column_width * 2))

for group in list(group_list):
    print(f"{group.name:<{column_width}}{group.location}")

The script runs correctly if I run “python.exe test”
You don’t even need to execute it, to demonstrate the extension load failure.
The extension fails to load with:

2022-06-29 06:46:49  [Info] [omni.ext.plugin] [ext: omni.hello.test-1.0.0] registered (path: c:/users/.../kit-exts-project/exts/omni.hello.test)
2022-06-29 06:46:49  [Info] [omni.ext.plugin] [ext: omni.hello.test-1.0.0] deps:
2022-06-29 06:46:49  [Info] [omni.ext.impl._internal] Adding c:\users\...\kit-exts-project\exts\omni.hello.test to sys.path
2022-06-29 06:46:49  [Info] [omni.ext.plugin] About to startup: [ext: omni.hello.test-1.0.0] (order: 0) Triggered by API/CLI/Config. (path: c:/users/.../kit-exts-project/exts/omni.hello.test)
2022-06-29 06:46:49  [Error] [omni.ext.impl.custom_importer] Failed to import python module omni.hello.test. Error: No module named 'azure.identity'. Traceback:
2022-06-29 06:46:49  [Error] [omni.ext.impl.custom_importer]   File "c:\users\...\kit-exts-project\exts\omni.hello.test\omni\hello\test\__init__.py", line 1, in <module>
2022-06-29 06:46:49  [Error] [omni.ext.impl.custom_importer]   File "c:\users\...\kit-exts-project\exts\omni.hello.test\omni\hello\test\extension.py", line 5, in <module>
2022-06-29 06:46:49  [Error] [omni.ext.impl.custom_importer]     from azure.identity import DefaultAzureCredential
2022-06-29 06:46:49  [Error] [omni.ext.impl.custom_importer] ModuleNotFoundError: No module named 'azure.identity'
2022-06-29 06:46:49  [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.hello.test' in 'c:\users\...\kit-exts-project\exts\omni.hello.test' failed to load.

If you comment out this line, the extension loads fine.

from azure.identity import DefaultAzureCredential 

So, something in the extension load can’t find the azure-identity package.?

I don’t think this is library specific as it runs fine in python. So, it seems it’s something with packaging these dependencies into the embedded kit python library. The package is clearly installed in my local python library, but probably not getting added to the embedded python.

[gavin.stevens]

I was able to progress a bit… I added this:

try:
    import azure.identity
except:
    omni.kit.pipapi.install("azure-identity")

Logs show:

[Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] Attempting to install 'azure.identity' from online index
[Info] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] run process: ['c:\\users\\...\\ov\\pkg\\code-2022.1.1\\kit\\python/python.exe', '-m', 'pip', '--isolated', 'install', '--target=F:\\OmniVerseNucleus\\Kit\\Code\\2022.1\\pip3-envs\\default', 'azure.identity']
[Error] [omni.kit.pipapi.pipapi] Failed to import python module azure.identity. Error: No module named 'azure.identity'

It seems to me that there might be something with the “azure-identity” vs “azure.identity” references here. The pip package is azure-identity,

Package             Version
------------------- ---------
azure-common        1.1.28
azure-core          1.24.1
azure-identity      1.10.0
azure-mgmt-core     1.3.1
azure-mgmt-resource 21.1.0

Name: azure-identity
Version: 1.10.0
Summary: Microsoft Azure Identity Library for Python
Home-page: https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/identity/azure-identity
Author: Microsoft Corporation
Author-email: azpysdkhelp@microsoft.com
License: MIT License
Location: c:\users\gavin\appdata\local\packages\pythonsoftwarefoundation.python.3.10_qbz5n2kfra8p0\localcache\local-packages\python310\site-packages
Requires: azure-core, cryptography, msal, msal-extensions, six
Required-by:

So “Attempting to install ‘azure.identity’ from online index” Shouldn’t this be installing azure-identity?

I havent seen any other samples with library names that differ from the imports.

Maybe the Omni importer is making as assumption based on the “import” name?

[gavin.stevens]

Still not working right:

print(sys.modules.keys())

if "azure.identity" not in sys.modules.keys():
    try:
        from azure.identity import ClientSecretCredential
    except ImportError:
        import omni.kit.pipapi

        omni.kit.pipapi.install("azure-identity", module="azure.identity", ignore_import_check=True, ignore_cache=True, surpress_output=False,use_online_index=True )

looking at the output of that sys.modules.keys print, there are 3543 items in the library, including some azure libraries, BUT azure.identity is NOT in there?
I took the output and sorted it by library name to see.

The relevant section:
...
azure.core.utils._connection_string_parser
azure.core.utils._pipeline_transport_rest_shared
azure.core.utils._pipeline_transport_rest_shared_async
azure.core.utils._utils
azure.mgmt
azure.mgmt.core
azure.mgmt.core._async_pipeline_client
azure.mgmt.core._pipeline_client
azure.mgmt.core._version
...

the extension load log cant find that Credential type, I added a button and an onclick()

                def on_click():
                    
                    print("hello")
                    tenant = '' //want to get these from ui.StringField
                    client = ''  //want to get these from ui.StringField
                    secret = ''  //want to get these from ui.StringField
                    authority = 'https://login.microsoftonline.com'
                    credential = ClientSecretCredential(tenant, client, secret, authority)

this fails with:

[Info] [omni.kit.app.impl] [py stdout]: hello
[Info] [omni.kit.app.impl] 
[Error] [omni.ui.python] NameError: name 'ClientSecretCredential' is not defined
[Error] [omni.ui.python] 
[Error] [omni.ui.python] At:
[Error] [omni.ui.python]   c:\users\...\extension.py(45): on_click
[Error] [omni.ui.python]   c:\users\...\extension.py(107): <lambda>

So the pip import and missing azure.identity library is still a thing… ;(
What else can I try ?

[mati-nvidia]

Hey @gavin.stevens! I’ll report this to the dev team. I see two issues:

1. Like you mentioned, there’s an error where it seems like it’s trying to import the package name as a module at the end of the install.
   
   
2. It seems like the two pip install commands are clobbering each other. After I install azure-mgmt-resource, azure.identity is gone.

As a workaround for now, you could either run Kit with a different python interpreter that has the python packages you want already installed or just add the paths to the python packages using sys.path.

[gavin.stevens]

Thanks for your reply. I’m not sure how to do either of those workarounds?
The documentation is vague…

“[run Kit with a different python interpreter]”
I don’t really “run kit”. I’m just writing an extension, am I using it under the covers?
How do I do this in the context of my code extension?

" [add the paths to the python packages using sys.path]"
I can get the local path to my site-packages with pip show azure.identity.
your link says, “use kit.exe --/app/python/extractPaths…” I don’t ever “run kit.exe” to run my extension?

or in kit file:

[settings]
app.python.extraPaths = ["C:/temp"]

What kit file? add this to extension.toml ? doesn’t seem right…

I tried to use the archiveDirs:

[python.pipapi]
# List of additional directories with pip achives to be passed into pip using ``--find-links`` arg.
# Relative paths are relative to extension root. Tokens can be used.
archiveDirs = ["path/to/pip_archive"]

this seemed to crash my extension and it had to be re-added to the extension list to reload.

[mati-nvidia]

All the OV app are built on Kit. They use the kit.exe with some config file that says, “only use these extensions, with this ui style, etc.” to make each app unique.

Avoiding tweaking app configurations, the easiest path forward for you to continue developing/testing your extension would be to:

1. pip install the libraries you want into another Python 3.7 interpreter.
2. Add the path to the site-packages for your separate Python 3.7 interpreter with sys.path.append at the top of your extension module.

If you want to know more about Kit app configuration, I recommend checking out this documentation: Configuration — kit-sdk 103.1 documentation

[anovoselov]

All we do in omni.kit.pipapi is calling pip and passing a dir to install into. That dir is added to sys.path. From here it is up to individual package if it works or not. Source code is quite simple and short there. If you have a working installation you can instead manually add that dir to sys.path

[anovoselov]

I tried that script

import omni.kit.pipapi
omni.kit.pipapi.install("azure-identity azure-mgmt-resource", ignore_import_check=True)

import azure.identity
import azure.mgmt.resource

run as: kit.exe --enable omni.kit.pipapi --exec script.py --clear-data

it fails with:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
flake8 4.0.1 requires importlib-metadata<4.3; python_version < "3.8", but you have importlib-metadata 4.10.1 which is incompatible.

I don’t know how to resolve it. importlib-metadata is something python comes with.

[gavin.stevens]

Hmm, thats strange, I was able to work around the issue as @mati-nvidia suggested. I installed a new Python 3.7 instance and updated my env vars to point at it. I pip installed the packages I needed and then added:

sys.path.append("D:/python37/lib/site-packages")

from azure.mgmt.resource import ResourceManagementClient
from azure.identity import ClientSecretCredential

This seems to work, but it wasn’t working with the embedded python.