Linux vs Windows - load order issue - Kit104 - No Module named omni - Git extension

Omniverse Developer
1

I have been interested in developing a cpu driven cicd pipeline for custom extensions. I have seen significant progress and am able to run pipelines both locally (Windows), and in CICD (linux). The linux pipeline is a modification of the kit-104 container ( test scripts, kit file, rootless folder structure, etc.) . This sufficed until I needed to add other extensions as a dependency.

The odd part is it windows works… for the most part.
By adding the following lines into my .kit file

[settings.app.exts]
folders.“++” = [
“git://github.com/NVIDIA-Omniverse/kit-extension-sample-ui-scene?branch=main&dir=exts”,
]

I am able to, 90% of the time pull down any git dependency (slightly modified).

Windows Success LOG

…
2023-06-21 15:46:50 [280ms] [Info] [omni.ext.plugin] [ext: omni.kit.extpath.git-0.0.0] applying settings
2023-06-21 15:46:50 [417ms] [Info] [carb] Plugin carb.scripting-python.plugin is already a dependency of omni.kit.app.plugin; not changing unload order
2023-06-21 15:46:50 [417ms] [Info] [omni.ext._impl._internal] Adding c:.…\MY_AWESOME_EXTENSION\app\kit\extscore\omni.kit.extpath.git to sys.path
2023-06-21 15:46:50 [417ms] [Info] [omni.ext.plugin] About to startup: [ext: omni.kit.extpath.git-0.0.0] (order: -1000) Triggered by API/CLI/Config. (path: c:/…/MY_AWESOME_EXTENSION/app/kit/extscore/omni.kit.extpath.git)
2023-06-21 15:46:50 [417ms] [Info] [omni.kit.app._impl] python GC: gc.disable()
2023-06-21 15:46:50 [417ms] [Info] [omni.kit.app.plugin] [0.517s] [ext: omni.kit.extpath.git-0.0.0] startup
2023-06-21 15:46:50 [425ms] [Info] [omni.ext._impl._internal] Searching for classes derived from ‘omni.ext.IExt’ in ‘omni.kit.extpath.git’ (‘c:\users.…\MY_AWESOME_EXTENSION\app\kit\extscore\omni.kit.extpath.git’)
2023-06-21 15:46:50 [426ms] [Info] [omni.ext._impl._internal] Found class ExtPathGitExt derived from omni.ext.IExt in module: ‘omni.kit.extpath.git’ in ‘c:…\MY_AWESOME_EXTENSION\app\kit\extscore\omni.kit.extpath.git’. Calling on_startup.
2023-06-21 15:46:50 [426ms] [Info] [omni.kit.extpath.git.utils] [omni.kit.extpath.git]: run process: [‘git’, ‘–version’]
2023-06-21 15:46:50 [498ms] [Info] [omni.ext.plugin] Registered protocol provider for scheme: git
2023-06-21 15:46:50 [498ms] [Info] [omni.ext.plugin] Registered protocol provider for scheme: git+https
2023-06-21 15:46:50 [498ms] [Info] [omni.ext.plugin] Registered protocol provider for scheme: git+http
2023-06-21 15:46:50 [498ms] [Info] [omni.ext.plugin] Registered protocol provider for scheme: git+ssh
2023-06-21 15:46:50 [498ms] [Info] [omni.ext.plugin] [ext: omni.kit.extpath.git-0.0.0] started, startup time: 80 (ms)
2023-06-21 15:46:50 [498ms] [Info] [omni.kit.extpath.git.utils] [omni.kit.extpath.git]: ExtPathGitExt::_on_add_path: git:/github.com/NVIDIA-Omniverse/kit-extension-sample-ui-scene?branch=main&dir=exts
2023-06-21 15:46:50 [499ms] [Info] [omni.kit.app._impl] [py stdout]: hello git:/github.com/NVIDIA-Omniverse/kit-extension-sample-ui-scene?branch=main&dir=exts, C:/Users/…/AppData/Local/ov/cache/gitexts
…

However, when making a new modification to the …/app/kit/extscore/omni.kit.extpath.git file, I am able to manually trigger an error similarly to what I see every run on the linux side.

Linux Fail LOG

…
2023-06-22 18:36:34 [91ms] [Info] [omni.ext.plugin] [ext: omni.usd.libs-1.0.0] registered (path: /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.usd.libs)
2023-06-22 18:36:34 [91ms] [Info] [omni.ext.plugin] Extensions dependency solver solution:
[ext: omni.kit.extpath.git-0.0.0] no deps

2023-06-22 18:36:34 [91ms] [Info] [omni.ext.plugin] [ext: omni.kit.extpath.git-0.0.0] applying settings
2023-06-22 18:36:34 [91ms] [Info] [carb] Initializing plugin: carb.scripting-python.plugin (interfaces: [carb::scripting::IScripting v1.0]) (impl: carb.scripting-python.plugin)
2023-06-22 18:36:34 [92ms] [Info] [carb.scripting-python.plugin] Attempting to initialize Python interpreter. Failure of doing it will lead to the termination of the application.
2023-06-22 18:36:34 [128ms] [Info] [carb.scripting-python.plugin] Python interpreter was successfully initialized.
2023-06-22 18:36:34 [128ms] [Error] [carb.scripting-python.plugin] ModuleNotFoundError: No module named ‘omni’

At:
PythonExtension.cpp::prestartup()(1):

2023-06-22 18:36:34 [128ms] [Error] [omni.ext.plugin] [ext: omni.kit.extpath.git-0.0.0] Failed to process python module extension in ‘/opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.extpath.git/.’.
2023-06-22 18:36:34 [128ms] [Info] [omni.ext.plugin] About to startup: [ext: omni.kit.extpath.git-0.0.0] (order: -1000) Triggered by API/CLI/Config. (path: /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.extpath.git)
2023-06-22 18:36:34 [128ms] [Info] [omni.kit.app.plugin] [0.154s] [ext: omni.kit.extpath.git-0.0.0] startup
2023-06-22 18:36:34 [128ms] [Error] [carb.scripting-python.plugin] ModuleNotFoundError: No module named ‘omni’

At:
PythonExtension.cpp::shutdown()(1):

2023-06-22 18:36:34 [128ms] [Error] [omni.ext.plugin] Failed while calling shutdown_extensions function: /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.extpath.git
2023-06-22 18:36:34 [128ms] [Info] [omni.ext.plugin] [ext: omni.kit.extpath.git-0.0.0] started, startup time: 0 (ms)
2023-06-22 18:36:34 [128ms] [Error] [omni.ext.plugin] Failed to add extension path: ‘git:/github.com/NVIDIA-Omniverse/kit-extension-sample-ui-scene?branch=main&dir=exts’. Local path is empty or scheme is not supported.
2023-06-22 18:36:34 [363ms] [Info] [omni.kit.app._impl] python GC: gc.disable()
2023-06-22 18:36:34 [363ms] [Info] [carb] Plugin carb.scripting-python.plugin is already a dependency of omni.kit.app.plugin; not changing unload order
…

I added some logging to the extension itself, but I believe the issue to be a unix specific race condition. I have tried playing with the load order and adding dependencies to no avail.

Additionally, I have been looking at task manager for performance clues and on this fail it seemed to touch 100% CPU usage, memory looks fine at about 50%)

Does anyone have insights? Perhaps the git code has an undocumented dependency? Let me know if i can provide additional information.

2

Hi @michael.s1.ellis. Thanks for bringing this to our attention. Checking with the dev team.

3

Yes please let me know if there is any development. It may be worth noting that I have only tried the kit 104 app as a back end (for linux, it was the only docker container other than isaac-sim available) whereas the local tests on windows are linked to code. Seeing as the underline kit version is the same i do not know why this would matter.

4

Is there any update on this? Perhaps it is a known issue being worked into kit 105?

5

I went ahead and made my own kit 105 image based off the beta branch and it is providing the same behavior.

6

The intent is to test extensions in a cicd pipeline. Some tests are dependent on other repos, and while I’d like to go back and write the unit tests in a way so that external dependencies like my utilities repo are mocked, I do like the idea of having some sort of integration test capacity.

Docker file creation: I have created a docker image of kit 105, following the kit 104 kitSDK as an example.

I have created a test_ext.kit file and an entry point test_ext_rootless.sh entry point which is a lightly modified of the startup_no_gpu.sh script shipped with most images.

The only delta being I needed to add in VULKAN_SDK=/opt/nvidia/omniverse/vkapiversion/bin to the LD_LIBRARY_PATH

I then call test.kit via

 exec "/opt/nvidia/omniverse/kit-sdk-launcher/kit" /opt/nvidia/omniverse/kit-sdk-launcher/**_test_ext.kit --no-window \
 --/exts/omni.kit.test/testExts/0="${EXT_TO_TEST}" \
 --/exts/omni.kit.test/stdoutFailPatterns/exclude='["*GLFW*","*[carb.graphics-vulkan.plugin]*","*[gpu.foundation.plugin]*","*[omni.gpu_foundation_factory.plugin]*","*
[omni.appwindow.plugin]*","*[omni.kit.renderer.plugin]*"]' \
 --portable-root "$SDK_DIR/" \
 --/app/enableStdoutOutput=0 \
 --/telemetry/mode=test -- "$@"

where the excluded errors are vulkan related, SDK_DIR=/opt/nvidia/omniverse/kit-sdk-launcher/, and ‘–’ is added before “$@” so that flags are actually passed through.

This works, and very well for independent extensions which i mount to /mnt/

When adding any git folder though e.g:

[settings.app.exts]
folders.'++' = [
    "/mnt/exts",                                        # Add mounted extension to path
    "/opt/nvidia/omniverse/kit-sdk-launcher/exts",      # Add base kit extensions to path
    "/opt/nvidia/omniverse/kit-sdk-launcher/apps",      # Add shipped kit files to path
    "/opt/nvidia/omniverse/kit-sdk-launcher/extscache", # Add Create and Code specific files to path (OGC tile converter, ui- testing, etc.)
    "git+ssh://gitlab******.com/****/omniverse/extensions/utilities.git?branch=develop&dir=****-utilities/exts"
]

What follows is a list of flags allowing me to run tests in a “cpu mode” and the resulting error when bringing in a git repo:

./test_ext_rootless.sh --/exts/omni.kit.test/testExts/0="${EXT_TO_TEST}" --/exts/omni.appwindow/enabled=false --/exts/omni.kit.renderer.core/autostartRenderer=false --/exts/omni.kit.renderer.core/autostartImGuiRenderer=false --/exts/omni.appwindow/autocreateAppWindow=false --/exts/omni.kit.renderer.core/compatibilityMode=true --/app/window/dpiScaleOverride=1.0 --/app/window/scaleToMonitor=false --/app/file/ignoreUnsavedOnExit=true --/exts/omni.kit.viewport.window/startup/windowName=Viewport

…

At:
PythonExtension.cpp::prestartup()(1):

2023-09-01 17:55:31 [103ms] [Error] [omni.ext.plugin] [ext: omni.kit.extpath.git-0.0.0] Failed to process python module extension in ‘/opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.extpath.git/.’.
2023-09-01 17:55:31 [103ms] [Error] [carb.scripting-python.plugin] ModuleNotFoundError: No module named ‘omni’

At:
PythonExtension.cpp::shutdown()(1):

2023-09-01 17:55:31 [103ms] [Error] [omni.ext.plugin] Failed while calling shutdown_extensions function: /opt/nvidia/omniverse/kit-sdk-launcher/extscore/omni.kit.extpath.git
2023-09-01 17:55:31 [103ms] [Error] [omni.ext.plugin] Failed to add extension path: ‘git+ssh://gitlab..com//omniverse/extensions/utilities.git?branch=develop&dir=***-utilities/exts’. Local path is empty or scheme is not supported.

Again, this sporadically happens on windows, when a change to the git extension is made, but always happens on Linux. This extensions seems to ignore changes to load order.

I believe this is most of the pertinent information.