Pip packages that share a namespace not being installed correctly

[josh.parker1]

Having issue with pip packages that share a namespace not being installed correctly when using pipapi as described here. For example, azure packages get installed into a azure namespace. If I install multiple packages, e.g. azure-storage-blob and azure-data-tables, only the first one gets installed.

Reproduction steps:

1. add the following pip dependancies to extension.toml:

[python.pipapi]
use_online_index = true
requirements = [
    "azure-storage-blob --ignore-installed",
    "azure-data-tables --ignore-installed",
]

2. start service on local machine:

kit app.kit

3. start service in container:

docker compose up

What I expect:

1. Packages installed successfully and service starts when running locally
2. Packages installed successfully and service starts when running in container

What happens:

1. Installation fails when running locally because one of the packages can not be found.

2023-03-31 21:08:13 [36,121ms] [Error] [omni.ext._impl.custom_importer] Failed to import python module poc.services.azure. Error: No module named 'azure.data'. Traceback:
Traceback (most recent call last):
  File "c:/users/nycjyp/appdata/local/ov/pkg/kit-104.1.0/kernel/py\omni\ext\_impl\custom_importer.py", line 76, in import_module
    return importlib.import_module(name)
  File "c:\users\nycjyp\appdata\local\ov\pkg\kit-104.1.0\python\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\users\nycjyp\code\sandbox\omniverse\poc.extensions\service-with-pip-deps\exts\poc.services.azure\poc\services\azure\__init__.py", line 1, in <module>
    from .extension import *
  File "c:\users\nycjyp\code\sandbox\omniverse\poc.extensions\service-with-pip-deps\exts\poc.services.azure\poc\services\azure\extension.py", line 4, in <module>
    from azure.data.tables import TableServiceClient
ModuleNotFoundError: No module named 'azure.data'

2023-03-31 21:08:13 [36,122ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'poc.services.azure' in 'c:\users\nycjyp\code\sandbox\omniverse\poc.extensions\service-with-pip-deps\exts\poc.services.azure' failed to load.

At:
  c:/users/nycjyp/appdata/local/ov/pkg/kit-104.1.0/kernel/py\omni\ext\_impl\_internal.py(188): startup
  c:/users/nycjyp/appdata/local/ov/pkg/kit-104.1.0/kernel/py\omni\ext\_impl\_internal.py(280): startup_extension
  PythonExtension.cpp::startup()(2): <module>

2023-03-31 21:08:13 [36,122ms] [Error] [omni.ext.plugin] [ext: poc.services.azure-1.0.0] Failed to startup python extension.

2. Installation fails when running in container:

service-with-pip-deps-ov-farm-connector-1  | 2023-03-31 20:37:42 [11,648ms] [Error] [omni.ext.impl.custom_importer] Failed to import python module poc.services.azure. Error: cannot import name 'Self' from 'typing_extensions' (/opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.pip_archive/pip_prebundle/typing_extensions.py). Traceback:
service-with-pip-deps-ov-farm-connector-1  | Traceback (most recent call last):
service-with-pip-deps-ov-farm-connector-1  |   File "/opt/nvidia/omniverse/kit-sdk-launcher/plugins/bindings-python/omni/ext/impl/custom_importer.py", line 76, in import_module
service-with-pip-deps-ov-farm-connector-1  |     return importlib.import_module(name)
service-with-pip-deps-ov-farm-connector-1  |   File "/opt/nvidia/omniverse/kit-sdk-launcher/python/lib/python3.7/importlib/__init__.py", line 127, in import_module
service-with-pip-deps-ov-farm-connector-1  |     return _bootstrap._gcd_import(name[level:], package, level)
service-with-pip-deps-ov-farm-connector-1  |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
service-with-pip-deps-ov-farm-connector-1  |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
service-with-pip-deps-ov-farm-connector-1  |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
service-with-pip-deps-ov-farm-connector-1  |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
service-with-pip-deps-ov-farm-connector-1  |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
service-with-pip-deps-ov-farm-connector-1  |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
service-with-pip-deps-ov-farm-connector-1  |   File "/exts/poc.services.azure/poc/services/azure/__init__.py", line 1, in <module>
service-with-pip-deps-ov-farm-connector-1  |     from .extension import *
service-with-pip-deps-ov-farm-connector-1  |   File "/exts/poc.services.azure/poc/services/azure/extension.py", line 3, in <module>
service-with-pip-deps-ov-farm-connector-1  |     from azure.storage.blob import BlobServiceClient
service-with-pip-deps-ov-farm-connector-1  |   File "/root/.local/share/ov/data/Kit/kit/103.5/pip3-envs/default/azure/storage/blob/__init__.py", line 10, in <module>
service-with-pip-deps-ov-farm-connector-1  |     from ._blob_client import BlobClient
service-with-pip-deps-ov-farm-connector-1  |   File "/root/.local/share/ov/data/Kit/kit/103.5/pip3-envs/default/azure/storage/blob/_blob_client.py", line 17, in <module>
service-with-pip-deps-ov-farm-connector-1  |     from typing_extensions import Self
service-with-pip-deps-ov-farm-connector-1  | ImportError: cannot import name 'Self' from 'typing_extensions' (/opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.pip_archive/pip_prebundle/typing_extensions.py)
service-with-pip-deps-ov-farm-connector-1  |
service-with-pip-deps-ov-farm-connector-1  | 2023-03-31 20:37:42 [11,648ms] [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'poc.services.azure' in '/exts/poc.services.azure' failed to load.
service-with-pip-deps-ov-farm-connector-1  |
service-with-pip-deps-ov-farm-connector-1  | At:
service-with-pip-deps-ov-farm-connector-1  |   /opt/nvidia/omniverse/kit-sdk-launcher/plugins/bindings-python/omni/ext/impl/_internal.py(177): startup
service-with-pip-deps-ov-farm-connector-1  |   /opt/nvidia/omniverse/kit-sdk-launcher/plugins/bindings-python/omni/ext/impl/_internal.py(268): startup_extension
service-with-pip-deps-ov-farm-connector-1  |   PythonExtension.cpp::startup()(2): <module>
service-with-pip-deps-ov-farm-connector-1  |
service-with-pip-deps-ov-farm-connector-1  | 2023-03-31 20:37:42 [11,648ms] [Error] [omni.ext.plugin] [ext: poc.services.azure-1.0.0] Failed to startup python extension.

Notes:

• Able to successfully install either package by itself (just not both).
• Able to manually install both packages to install location, and run kit app.kit without clearing cache and everything works.

Step taken:

1. Tried adding --ignore-installed option. Has no effect.

requirements = [
    "azure-storage-blob --ignore-installed",
    "azure-data-tables --ignore-installed",
]

2. Tried installing dependancies during container init and adding location to sys.path

Removed [python.pipapi] from extension.toml and updated dockerfile to include steps to install azure dependancies to /usr/local/lib/python3.7/dist-packages:

RUN echo "deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal main" | tee /etc/apt/sources.list.d/deadsnakes-ppa.list

# install pip via python
RUN apt-get update \
    && apt-get install -y python3.7 python3-pip python3.7-distutils \
    && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1

# Install azure dependancies
RUN pip3 install azure-storage-blob azure-data-tables

Able to log into container and verify that packages were installed to /usr/local/lib/python3.7/dist-packages:

Then tried to add path to sys.path from .kit file:

[settings.app]
python.sysPaths.'++' = ["/usr/local/lib/python3.7/dist-packages"]
python.extraPaths.'++' = ["/usr/local/lib/python3.7/dist-packages"]

But sys.path is not getting updated. Finally resorted to updating sys.path from script using sys.path.append() and that updated sys.path, but now getting strage ImportError: cannot import name 'Self' from 'typing_extensions' error:

[josh.parker1]

minimal example that reproduces issue: omniverse-sandbox/poc.extensions/azure-service at main · parkerjgit/omniverse-sandbox · GitHub

[mati-nvidia]

Hi Josh. Thanks for the excellent write-up. We are tracking this issue of shared installs with shared namespaces with internal issue OM-86666.

For the setting sys.path, extraPaths is correct. Where did you see sysPaths? I wonder if you have [settings.path] under your settings section and it resolves to the wrong setting path?

For the typing_extensions error you’re hitting now, a possible workaround could be turning off the pip_prebundleand cherry picking just the Python requirements that you really need. I think the issue is a version mismatch between what pip_prebundle uses and what azure wants.

[josh.parker1]

Hi Mati,

How do you recommend turning off the pip_prebundle? Do you mean, instead of building from nvcr.io/nvidia/omniverse/kit base image, manually installing all kit-bundled dependancies.

[josh.parker1]

Looks like nvcr.io/nvidia/omniverse/kit:4.0.0 bundles version 3.7.5 of typing_extensions, which is a 2+ yrs old. Possible to bump this to 4.x?

[josh.parker1]

Tried to prepend /usr/local/lib/python3.7/dist-packages to sys.path to give it lookup precendance over prebundle /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.pip_archive/pip_prebundle/ but script is still using version 3.7.5 from prebundle for some reason.

Current work around is to manually delete version 3.7.5, then re-install version 4.5.0 into prebundle.

#dockerfile

# remove from prebundle
RUN rm -rf /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.pip_archive/pip_prebundle/typing_extensions.py \
    && rm -rf /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.pip_archive/pip_prebundle/typing_extensions-3.7.4.3.dist-info

# re-install
RUN pip3 install typing-extensions azure-storage-blob azure-data-tables --target /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.pip_archive/pip_prebundle/

@mati-nvidia do you happen to know if version3.7.5 required by any kit packages?

[mati-nvidia]

I haven’t looked at our full dependencies, but it’s likely reasonable to update that.

For now, I do think we need to find a workaround for you until OM-86666 can fully address these issues. You have a solid understanding of the issue and thanks for powering through this.

Prepending to sys.path didn’t work because I’m guessing the pip_prebundle extension imported the bad typing_extensions before you got to add yours to the path. Editing omni.kit.pip_archive would work, but I think we can do something a little less hacky.

Two options:

1. app.path.extraPaths settings is adding the path for me. Idk if you could try it with my same TOML formatting to see if it works.
   
   
2. omni.kit.pip_archive has a extension load order number = -1000. You could create your own “pip_archive” that with order=-1001 so that it loads before and adds your dependencies and imports them first.
   
   
   
   I realized disabling omni.kit.pip_archive won’t work because other extensions explicitly depend on it.

Let me know how it goes.

[josh.parker1]

Thanks @mati-nvidia! I was able to get extraPaths working.

when running locally with kit app.kit:

[settings.app]
python.extraPaths.'++' = ["/usr/local/lib/python3.7/dist-packages"]

when running in container:

# Install external pip dependancies   
RUN echo "deb http://ppa.launchpad.net/deadsnakes/ppa/ubuntu focal main" | tee -a /etc/apt/sources.list.d/deadsnakes-ppa.list
RUN apt-get update \
    && apt-get install -y python3.7 python3-pip python3.7-distutils \
    && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1 \
    && pip3 install azure-storage-blob azure-data-tables 

...

ENTRYPOINT [ \
  "/opt/nvidia/omniverse/kit-sdk-launcher/kit", \
  "--/app/python/extraPaths/0=/usr/local/lib/python3.7/dist-packages", \
  ...
]

Agree, this is improvement over editing omni.kit.pip_archive.