Ucf_app_builder_cli not found k8s secret

Omniverse General Discussion
1

I’m trying to run a tokkio workflow following this quick start guide here. I got stuck when building the tokkio workflow llm-rag/tokkio-llm-app.yaml.

The error stated that:

$ ucf_app_builder_cli app build llm-rag/tokkio-llm-app.yaml llm-rag/tokkio-llm-app-params.yaml
2024-09-26 22:58:46,603 - AppBuilder - INFO - Syncing any missing service versions to cache...
2024-09-26 22:58:46,605 - AppBuilder - WARNING - secret 'ngc-api-key-secret' not found in 'animation-graph' (ucf.svc.ia-animation-graph-microservice)
2024-09-26 22:58:46,606 - AppBuilder - WARNING - secret 'ngc-api-key-secret' not found in 'avatar-renderer' (ucf.svc.ia-omniverse-renderer-microservice)
2024-09-26 22:58:46,675 - AppBuilder - INFO - Validating application ...
2024-09-26 22:58:46,723 - AppBuilder - INFO - Building application ucs-tokkio-audio-video-llm-app-4.1.0 ...
Output directory llm-rag/ucs-tokkio-audio-video-llm-app-4.1.0 is not empty. All contents will be removed. Proceed? [y/n] y
2024-09-26 22:59:06,389 - AppBuilder - ERROR - Failed to get values from 'https://helm.ngc.nvidia.com/eevaigoeixww/tokkio-4-0/charts/ds-sdr-envoy-0.1.22.tgz' - Error: failed to fetch https://helm.ngc.nvidia.com/eevaigoeixww/tokkio-4-0/charts/ds-sdr-envoy-0.1.22.tgz

This clearly means that the ucf_app_builder_cli doesn’t found the secret on my kubernetes.

I’ve created the secret using this command: kubectl create secret generic ngc-api-key-secret --from-literal=NGC_CLI_API_KEY=$NGC_API_KEY (I’ve also make sure that NGC_API_KEY is set). I’ve also validated that the key exists by running kubectl get secrets -o yaml and here’s the result:

$ kubectl get secrets -o yaml
apiVersion: v1
items:
- apiVersion: v1
  data:
    NGC_CLI_API_KEY: xxxx
  kind: Secret
  metadata:
    creationTimestamp: "2024-09-26T15:57:48Z"
    name: ngc-api-key-secret
    namespace: default
    resourceVersion: "34257"
    uid: yyy
  type: Opaque
kind: List
metadata:
  resourceVersion: ""

I’ve been stuck for a while now, any help would be appreciated.

Here’s my additional info on my operating system:

  • Ubuntu 22.04.4 LTS
  • Running kubectl on microk8s
2

Let me see if I can find you some help on this. Thanks.

3

Some update, so I reinstall my k8s again while also ensuring that all ports from docker are closed (making sure that there’s no port collision making k8s communication inaccessible)

I tried to build the ucf yaml again but this time for animation_pipeline using this command:

ucf_app_builder_cli app build ucs_apps/animation_pipeline.yaml ucs_apps/animation_pipeline_params.yaml -o _build/animation_pipeline

The ucf_app_builder failed again with the following error:

2024-10-04 12:56:14,328 - AppBuilder - INFO - Syncing any missing service versions to cache...
2024-10-04 12:56:15,291 - Registry - INFO - Service list fetched successfully
2024-10-04 12:56:16,949 - Registry - ERROR - Service not found in NGC : ucf.svc.audio2face
2024-10-04 12:56:16,949 - Registry - ERROR - {
    "requestStatus": {
        "statusCode": "UNAUTHORIZED",
        "statusDescription": "Authentication Failed",
        "requestId": "282a5686-12440"
    }
}
2024-10-04 12:56:17,955 - AppBuilder - ERROR - Failed to find microservice 'ucf.svc.audio2face' in cache or '*' repo.

So I then check whether my key have proper permission to private registry by running docker login nvcr.io using the configured API key for ucf app builder and try to pull audio2face with this command:

docker pull nvcr.io/eevaigoeixww/animation/audio2face:1.0.11

The docker can be pulled successfully without any issue.

4

Ok thanks. Let me add that to the dev thread. Can you give me all the details of your hardware and environment setup please. As much detail as possible.

5

Some more update,

animation-pipeline workflow build

I was able to built the _build directory via the following command:

$ ucf_app_builder_cli app build ucs_apps/animation_pipeline.yaml ucs_apps/animation_pipeline_params.yaml -o _build/animation_pipeline
2024-10-06 21:49:31,216 - AppBuilder - INFO - Syncing any missing service versions to cache...
2024-10-06 21:49:31,230 - AppBuilder - INFO - Validating application ...
2024-10-06 21:49:31,238 - AppBuilder - INFO - Building application animation-pipeline-1.0.1 ...
2024-10-06 21:51:04,628 - AppBuilder - INFO - Application Information written to _build/animation_pipeline/app_info.yaml
2024-10-06 21:51:04,765 - AppBuilder - INFO - Application compliance report generated at _build/animation_pipeline/compliance_report.json
Application Helm Chart generated in _build/animation_pipeline
2024-10-06 21:51:04,766 - MsBuilder - INFO - App built

The good thing is that the ucf_app_builder_cli was able to find my secrets for ngc-api-key-secret in animation-pipeline namespace.

However, upon my k8s deployment with helm using this command:

$ helm upgrade --install --cleanup-on-fail --namespace $names
pace $name _build/animation_pipeline/ -f _build/animation_pipeline/values.yaml -f ucs_apps/animation_pipeline_values.yaml
WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
WARNING: Kubernetes configuration file is world-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
Release "animation-pipeline" has been upgraded. Happy Helming!
NAME: animation-pipeline
LAST DEPLOYED: Sun Oct  6 21:54:06 2024
NAMESPACE: animation-pipeline
STATUS: deployed
REVISION: 2
TEST SUITE: None

The pod avatar-renderer-a-deployment-0 (also b deployment) was always restarting and I’m not sure why.

Here's the pods full logs: 

	+ ldconfig -p
	+ grep libGLX_nvidia.so.0
		libGLX_nvidia.so.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.0
	+ [[ -v NOTFOUND ]]
	+ export VK_ICD_FILENAMES=/tmp/nvidia_icd.json
	+ VK_ICD_FILENAMES=/tmp/nvidia_icd.json
	+ export LD_LIBRARY_PATH=:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/nvidia/omniverse/kit-sdk-launcher/plugins/carb_gfx
	+ LD_LIBRARY_PATH=:/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/nvidia/omniverse/kit-sdk-launcher/plugins/carb_gfx
	+ /opt/nvidia/omniverse/vkapiversion/bin/vkapiversion /tmp/nvidia_icd.json
	Writing disposable ICD file (/tmp/tmp_icd_B0uViW.json)...
	Writing ICD file to (/tmp/nvidia_icd.json)
	GPU0
		apiVersion     = 1.3.277
		driverVersion  = 550.54.15
		vendorID       = 0x10de
		deviceID       = 0x2330
		deviceName     = NVIDIA H100 80GB HBM3

++ echo False
++ tr '[:upper:]' '[:lower:]'
+ lowercase_gpu_disable_alloc=false
+ '[' false == true ']'
+ '[' false == 1 ']'
+ echo 'Dynamic GPU allocation disabled'
Dynamic GPU allocation disabled
+ export CUDA_VISIBLE_DEVICES=0
+ CUDA_VISIBLE_DEVICES=0
+ echo Running: --/exts/omni.services.transport.server.http/port=8021 --/app/printConfig=False --/app/window/width=1280 --/app/window/height=720 --enable omni.kit.telemetry --/crashreporter/enabled=true --/crashreporter/data/serviceName=ia-omniverse-renderer-microservice-a-deployment --/crashreporter/url=https://services.nvidia.com/submit --/crashreporter/alwaysUpload=true --/crashreporter/dumpDir=/home/ace/asset
Running: --/exts/omni.services.transport.server.http/port=8021 --/app/printConfig=False --/app/window/width=1280 --/app/window/height=720 --enable omni.kit.telemetry --/crashreporter/enabled=true --/crashreporter/data/serviceName=ia-omniverse-renderer-microservice-a-deployment --/crashreporter/url=https://services.nvidia.com/submit --/crashreporter/alwaysUpload=true --/crashreporter/dumpDir=/home/ace/asset
+ exec /opt/nvidia/omniverse/renderer/kit/kit /opt/nvidia/omniverse/renderer/apps/omni.omniverse_renderer_ms.app.kit --/persistent/app/viewport/defaults/tickRate=30 --ext-folder /opt/nvidia/omniverse/kit-livestream/ --no-window --/exts/omni.services.transport.server.http/port=8021 --/app/printConfig=False --/app/window/width=1280 --/app/window/height=720 --enable omni.kit.telemetry --/crashreporter/enabled=true --/crashreporter/data/serviceName=ia-omniverse-renderer-microservice-a-deployment --/crashreporter/url=https://services.nvidia.com/submit --/crashreporter/alwaysUpload=true --/crashreporter/dumpDir=/home/ace/asset
[Info] [carb] Logging to file: /home/ace/.nvidia-omniverse/logs/Kit/omni.omniverse_renderer_ms.app/0.0/kit_20241006_150416.log
[0.488s] [ext: omni.kit.async_engine-0.0.0] startup
[0.569s] [ext: omni.stats-0.0.0] startup
[0.571s] [ext: omni.assets.plugins-0.0.0] startup
[0.572s] [ext: omni.rtx.shadercache.vulkan-1.0.0] startup
[0.573s] [ext: omni.client-1.0.1] startup
[0.585s] [ext: omni.gpu_foundation-0.0.0] startup
[0.604s] [ext: carb.windowing.plugins-1.0.0] startup
2024-10-06 15:04:17 [583ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:17 [583ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[0.605s] [ext: omni.kit.renderer.init-0.0.0] startup
2024-10-06 15:04:17 [636ms] [Warning] [omni.platforminfo.plugin] failed to open the default display.  Can't verify X Server version.

|---------------------------------------------------------------------------------------------|
| Driver Version: 550.54.15     | Graphics API: Vulkan
|=============================================================================================|
| GPU | Name                             | Active | LDA | GPU Memory | Vendor-ID | LUID       |
|     |                                  |        |     |            | Device-ID | UUID       |
|     |                                  |        |     |            | Bus-ID    |            |
|---------------------------------------------------------------------------------------------|
| 0   | NVIDIA H100 80GB HBM3            | Yes: 0 |     | 81559   MB | 10de      | 0          |
|     |                                  |        |     |            | 2330      | 61a1dd9b.. |
|     |                                  |        |     |            | 19        |            |
|=============================================================================================|
| OS: 20.04.6 LTS (Focal Fossa) ubuntu, Version: 20.04.6, Kernel: 5.15.0-122-generic
| Processor: Intel(R) Xeon(R) Platinum 8462Y+ | Cores: 64 | Logical: 128
|---------------------------------------------------------------------------------------------|
| Total Memory (MB): 2063919 | Free Memory: 1957812
| Total Page/Swap (MB): 0 | Free Page/Swap: 0
|---------------------------------------------------------------------------------------------|
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin] ECC is enabled for device 0. This will reduce rendering performance.
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin]  
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin] !!!!! Local system validation failed! Incorrect configuration detected.
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin]       Summary below. Details above.
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin]  
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin]  - ECC:                         FAILED
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin]  - IOMMU:                       OK
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin]  
2024-10-06 15:04:19 [2,684ms] [Warning] [gpu.foundation.plugin] -----------------------------------------------------------------------
[3.113s] [ext: omni.kit.pipapi-0.0.0] startup
[3.118s] [ext: omni.kit.pip_archive-0.0.0] startup
[3.118s] [ext: omni.kit.telemetry-0.5.0] startup
[3.161s] [ext: omni.mtlx-0.1.0] startup
[3.162s] [ext: omni.usd.config-1.0.3] startup
[3.168s] [ext: omni.gpucompute.plugins-0.0.0] startup
[3.168s] [ext: omni.usd.libs-1.0.0] startup
[3.267s] [ext: omni.kit.loop-default-0.2.0] startup
[3.268s] [ext: omni.appwindow-1.1.5] startup
2024-10-06 15:04:20 [3,248ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:20 [3,248ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.272s] [ext: omni.kit.renderer.core-0.0.0] startup
2024-10-06 15:04:20 [3,254ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:20 [3,254ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-10-06 15:04:20 [3,263ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:20 [3,263ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.286s] [ext: omni.kit.renderer.capture-0.0.0] startup
[3.289s] [ext: omni.kit.renderer.imgui-0.0.0] startup
2024-10-06 15:04:20 [3,271ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:20 [3,271ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-10-06 15:04:20 [3,272ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:20 [3,272ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
2024-10-06 15:04:20 [3,273ms] [Warning] [carb.windowing-glfw.plugin] GLFW initialization failed.
2024-10-06 15:04:20 [3,273ms] [Warning] [carb] Failed to startup plugin carb.windowing-glfw.plugin (interfaces: [carb::windowing::IGLContext v1.0],[carb::windowing::IWindowing v1.4]) (impl: carb.windowing-glfw.plugin)
[3.439s] [ext: carb.audio-0.1.0] startup
[3.464s] [ext: omni.ui-2.18.6] startup
[3.485s] [ext: omni.uiaudio-1.0.0] startup
[3.487s] [ext: omni.kit.mainwindow-1.0.1] startup
[3.490s] [ext: omni.kit.uiapp-0.0.0] startup
[3.490s] [ext: omni.usd.schema.omniscripting-1.0.0] startup
[3.616s] [ext: omni.usd.schema.anim-0.0.0] startup
[3.658s] [ext: omni.usd.schema.omnigraph-1.0.0] startup
[3.670s] [ext: omni.usd.schema.audio-0.0.0] startup
[3.675s] [ext: omni.usd.schema.semantics-0.0.0] startup
[3.679s] [ext: omni.usd.schema.geospatial-0.0.0] startup
[3.683s] [ext: omni.usd.schema.physics-0.0.0] startup
[3.706s] [ext: omni.usd_resolver-1.0.1] startup
[3.711s] [ext: omni.usd.core-1.1.8] startup
[3.714s] [ext: omni.resourcemonitor-105.0.0] startup
[3.717s] [ext: omni.activity.core-1.0.1] startup
[3.719s] [ext: omni.hydra.usdrt_delegate-7.2.39] startup
[3.732s] [ext: omni.kit.audiodeviceenum-1.0.1] startup
[3.734s] [ext: omni.kit.widget.path_field-2.0.8] startup
[3.743s] [ext: omni.kit.actions.core-1.0.0] startup
[3.745s] [ext: omni.graph.exec-0.3.0] startup
[3.746s] [ext: omni.kit.widget.browser_bar-2.0.9] startup
[3.750s] [ext: omni.kit.commands-1.4.6] startup
[3.783s] [ext: omni.kit.exec.core-0.5.0] startup
[3.786s] [ext: omni.kit.viewport.registry-104.0.5] startup
[3.788s] [ext: omni.kit.menu.utils-1.5.7] startup
[3.816s] [ext: omni.kit.window.popup_dialog-2.0.23] startup
[3.826s] [ext: omni.timeline-1.0.9] startup
[3.829s] [ext: omni.kit.widget.nucleus_connector-1.1.4] startup
[3.842s] [ext: omni.hydra.scene_delegate-0.3.2] startup
[3.849s] [ext: omni.iray.libs-0.0.0] startup
[3.854s] [ext: omni.usd-1.10.21] startup
[3.965s] [ext: omni.mdl.neuraylib-0.2.0] startup
[3.967s] [ext: omni.kit.helper.file_utils-0.1.6] startup
[4.058s] [ext: omni.kit.widget.nucleus_info-1.0.2] startup
[4.060s] [ext: omni.volume-0.4.0] startup
[4.062s] [ext: omni.kit.widget.filebrowser-2.3.35] startup
[4.092s] [ext: omni.kit.search_core-1.0.5] startup
[4.095s] [ext: omni.hydra.rtx-0.1.0] startup
[4.111s] [ext: omni.kit.widget.search_delegate-1.0.4] startup
[4.118s] [ext: omni.kit.widget.version------------------SETTINGS------------------
scene_skel_root_prim_path: /World/SkelRoots/Rig_Retarget/SkelRoot
scene_skeleton_prim_path: /World/SkelRoots/Rig_Retarget/SkelRoot/Skeleton
scene_skel_animation_prim_path: /World/SkelRoots/Rig_Retarget/SkelRoot/Skeleton/ACE_Animation_Target
scene_camera_prim_path: /World/SkelRoots/Rig_Camera/SkelRoot/Skeleton/root/camera_location/camera_body/camera_main
scene_skel_roots_scope_prim_path: /World/SkelRoots
scene_character_anim_pose_prim_path: /World/SkelRoots/Rig_Retarget/SkelRoot/Skeleton/AnimGraphOutputPose
scene_path: /home/ace/asset/Avatar_Scene.usda
scene_validation_mode: SceneValidationMode.DEFAULT_SCENE_VALIDATION
animation_source_host: ia-animation-graph-microservice-deployment-service
animation_source_port: 51000
animation_source_buffer_size: 0.1
debug_log_enabled: True
debug_log_each_received_animation_data_message: False
debug_log_frame_interval: 30
debug_show_wheel: False
rtp_negotiation_host: 0.0.0.0
rtp_negotiation_port: 32667
rtp_negotiation_host_mocking_enabled: True
livestream_host: 127.0.0.1
livestream_video_port: 9020
livestream_video_bit_rate: 32777216
livestream_video_iframe_interval: 30
livestream_video_frame_rate: 30
livestream_audio_port: 9021
livestream_audio_sample_rate: 16000
livestream_audio_bit_rate: 32
/exts/omni.services.transport.server.http/port: 8021
--------------------------------------------
2024-10-06 15:04:21 [5,079ms] [Error] [carb] [Plugin: libomni.livestream-rtp_udp.plugin.so] Could not load the dynamic library from /opt/nvidia/omniverse/kit-livestream/omni.kit.livestream.rtp_udp/plugins/libomni.livestream-rtp_udp.plugin.so. Error: /opt/nvidia/omniverse/kit-livestream/omni.kit.livestream.rtp_udp/plugins/libssl.so.1.1: undefined symbol: EVP_idea_cbc, version OPENSSL_1_1_0 (Additional information may be available by running the process with the LD_DEBUG environment variable set)
2024-10-06 15:04:21 [5,079ms] [Error] [omni.ext.plugin] [ext: omni.kit.livestream.rtp_udp-1.0.0] failed to load native plugin: /opt/nvidia/omniverse/kit-livestream/omni.kit.livestream.rtp_udp/plugins/libomni.livestream-rtp_udp.plugin.so
ing-1.4.6] startup
[4.130s] [ext: omni.kit.hydra_texture-1.1.12] startup
[4.135s] [ext: omni.ui.scene-1.7.0] startup
[4.142s] [ext: omni.kit.notification_manager-1.0.6] startup
[4.148s] [ext: omni.kit.clipboard-1.0.3] startup
[4.149s] [ext: omni.kit.widget.viewport-105.1.10] startup
[4.171s] [ext: omni.kit.window.filepicker-2.10.14] startup
[4.249s] [ext: omni.kit.viewport.window-105.1.13] startup
[4.301s] [ext: omni.kit.window.file_importer-1.0.23] startup
[4.305s] [ext: omni.omniverse_renderer_ms.dependencies.pip-0.0.0] startup
[4.342s] [ext: omni.kit.viewport.utility-1.0.16] startup
[4.347s] [ext: omni.services.facilities.base-1.0.4] startup
[4.348s] [ext: omni.kit.widget.searchable_combobox-1.0.5] startup
[4.353s] [ext: omni.services.core-1.9.0] startup
[4.566s] [ext: omni.kit.window.file_exporter-1.0.23] startup
[4.569s] [ext: omni.kit.widget.settings-1.0.6] startup
[4.580s] [ext: omni.services.transport.server.base-1.1.1] startup
[4.581s] [ext: omni.rtx.window.settings-0.6.9] startup
[4.593s] [ext: omni.services.transport.server.http-1.3.0] startup
[4.688s] [ext: omni.kit.viewport.rtx-104.0.0] startup
[4.688s] [ext: omni.omniverse_renderer_ms.window-0.0.0] startup
[5.023s] [ext: omni.kit.usd.layers-2.1.27] startup
[5.060s] [ext: omni.kit.menu.create-1.0.11] startup
[5.064s] [ext: omni.kit.context_menu-1.6.8] startup
[5.077s] [ext: omni.kit.window.preferences-1.3.23] startup
[5.094s] [ext: omni.kit.renderer.cuda_interop-1.0.1] startup
[5.096s] [ext: omni.kit.livestream.core-1.0.4] startup
[5.097s] [ext: omni.kit.livestream.rtp_udp-1.0.0] startup
2024-10-06 15:04:21 [5,079ms] [Warning] [carb] Potential plugin preload failed: /opt/nvidia/omniverse/kit-livestream/omni.kit.livestream.rtp_udp/plugins/libomni.livestream-rtp_udp.plugin.so
2024-10-06 15:04:21 [5,079ms] [Error] [carb] [Plugin: omni.kit.livestream.plugin] Dependency: [carb::livestream::Vision v0.4] failed to be resolved.
2024-10-06 15:04:21 [5,080ms] [Error] [carb] [Plugin: omni.kit.livestream.plugin] Dependency: [carb::livestream::Vision v0.4] failed to be resolved.
2024-10-06 15:04:21 [5,080ms] [Error] [carb.scripting-python.plugin] RuntimeError: Failed to acquire interface: omni::kit::livestream::ILivestream (pluginName: nullptr)

At:
  /opt/nvidia/omniverse/kit-livestream/omni.kit.livestream.rtp_udp/omni/kit/livestream/rtp_udp/extension.py(25): on_startup
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(164): _startup_ext
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/carb/profiler/__init__.py(85): wrapper
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(224): startup
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>
  /opt/nvidia/omniverse/renderer/exts/omni.omniverse_renderer_ms.window/omni/omniverse_renderer_ms/window/livestream.py(64): initialize
  /opt/nvidia/omniverse/renderer/exts/omni.omniverse_renderer_ms.window/omni/omniverse_renderer_ms/window/extension.py(254): on_startup
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(164): _startup_ext
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/carb/profiler/__init__.py(83): wrapper
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(224): startup
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>

2024-10-06 15:06:22 [5,823ms] [Error] [omni.ext.plugin] [ext: omni.kit.livestream.rtp_udp-1.0.0] Failed to startup python extension.
2024-10-06 15:06:22 [5,863ms] [Error] [carb] [Plugin: omni.kit.livestream.plugin] Dependency: [carb::livestream::Vision v0.4] failed to be resolved.
2024-10-06 15:06:22 [5,863ms] [Error] [carb] [Plugin: omni.kit.livestream.plugin] Dependency: [carb::livestream::Vision v0.4] failed to be resolved.
2024-10-06 15:06:22 [5,863ms] [Error] [carb.scripting-python.plugin] RuntimeError: Failed to acquire interface: omni::kit::livestream::ILivestream (pluginName: nullptr)

At:
  /opt/nvidia/omniverse/renderer/exts/omni.omniverse_renderer_ms.window/omni/omniverse_renderer_ms/window/livestream.py(69): initialize
  /opt/nvidia/omniverse/renderer/exts/omni.omniverse_renderer_ms.window/omni/omniverse_renderer_ms/window/extension.py(254): on_startup
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(164): _startup_ext
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/carb/profiler/__init__.py(85): wrapper
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(224): startup
  /opt/nvidia/omniverse/kit-sdk-launcher/kernel/py/omni/ext/_impl/_internal.py(328): startup_extension
  PythonExtension.cpp::startup()(2): <module>

2024-10-06 15:06:22 [5,863ms] [Error] [omni.ext.plugin] [ext: omni.omniverse_renderer_ms.window-0.0.0] Failed to startup python extension.
[SceneLoader] Attempting to delete USD cache directory: /home/ace/asset/Avatar_Scene.usda-cache.
[SceneLoader] Directory '/home/ace/asset/Avatar_Scene.usda-cache' not found.
[SceneLoader] SCENE LOADED: /home/ace/asset/Avatar_Scene.usda
[SceneLoader] Default scene validation completed!
[SceneLoader] Assets loaded.
[5.931s] [ext: omni.omniverse_renderer_ms.app-0.0.0] startup
[6.024s] app ready
2024-10-06 15:06:23 [6,319ms] [Warning] [rtx.neuraylib.plugin] [IRAY:RENDER]   1.1   IRAY   rend warn : CUDA device 0 (NVIDIA H100 80GB HBM3): ECC is enabled, this will reduce rendering performance (it is strongly recommended to disable ECC with iray)
2024-10-06 15:06:57 [40,302ms] [Warning] [carb.scenerenderer-rtx.plugin] Scene has 4 lights and may render slowly. Enabling Sampled Direct Lighting is recommended
2024-10-06 15:06:58 [41,541ms] [Warning] [gpu.foundation.plugin] Requesting texture to use texture streaming, but the service isn't available.The texture will be created as a regular resource.
[114.765s] [ext: omni.omniverse_renderer_ms.app-0.0.0] shutdown
[114.765s] [ext: omni.kit.livestream.rtp_udp-1.0.0] shutdown
[114.767s] [ext: omni.kit.livestream.core-1.0.4] shutdown
[114.776s] [ext: omni.kit.window.preferences-1.3.23] shutdown
2024-10-06 15:08:11 [114,798ms] [Warning] [omni.ext._impl._internal] omni.kit.window.preferences-1.3.23 -> <class 'omni.kit.window.preferences.scripts.preferences_window.PreferencesExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'method'>, id: 140052192450368", "[1]:type: <class 'cell'>, id: 140054967696736", "[2]:type: <class 'frame'>, id: 140052193415712", "[3]:type: <class 'frame'>, id: 140052192352320", "[4]:type: <class 'frame'>, id: 140052192096432"]
[114.828s] [ext: omni.kit.context_menu-1.6.8] shutdown
[114.829s] [ext: omni.omniverse_renderer_ms.window-0.0.0] shutdown
[114.831s] [ext: omni.kit.viewport.rtx-104.0.0] shutdown
[114.831s] [ext: omni.rtx.window.settings-0.6.9] shutdown
[114.835s] [ext: omni.kit.window.file_importer-1.0.23] shutdown
[114.837s] [ext: omni.kit.window.file_exporter-1.0.23] shutdown
[114.838s] [ext: omni.kit.menu.create-1.0.11] shutdown
[114.839s] [ext: omni.kit.usd.layers-2.1.27] shutdown
[114.845s] [ext: omni.kit.viewport.utility-1.0.16] shutdown
[114.846s] [ext: omni.kit.viewport.window-105.1.13] shutdown
2024-10-06 15:08:11 [114,914ms] [Warning] [omni.ext._impl._internal] omni.kit.viewport.window-105.1.13 -> <class 'omni.kit.viewport.window.extension.ViewportWindowExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'frame'>, id: 150331824", "[1]:type: <class 'frame'>, id: 149617424", "[2]:type: <class 'frame'>, id: 140079357588128"]
[114.943s] [ext: omni.kit.widget.viewport-105.1.10] shutdown
[114.945s] [ext: omni.kit.hydra_texture-1.1.12] shutdown
[114.946s] [ext: omni.kit.window.filepicker-2.10.14] shutdown
[114.947s] [ext: omni.kit.widget.search_delegate-1.0.4] shutdown
[114.948s] [ext: omni.kit.widget.filebrowser-2.3.35] shutdown
[114.950s] [ext: omni.kit.helper.file_utils-0.1.6] shutdown
[114.951s] [ext: omni.usd-1.10.21] shutdown

[116.132s] [ext: omni.kit.uiapp-0.0.0] shutdown
[116.132s] [ext: omni.uiaudio-1.0.0] shutdown
[116.133s] [ext: omni.kit.audiodeviceenum-1.0.1] shutdown
[116.133s] [ext: carb.audio-0.1.0] shutdown
[116.133s] [ext: omni.kit.renderer.cuda_interop-1.0.1] shutdown
[116.134s] [ext: omni.ui.scene-1.7.0] shutdown
[116.135s] [ext: omni.kit.mainwindow-1.0.1] shutdown
[116.135s] [ext: omni.kit.widget.settings-1.0.6] shutdown
[116.135s] [ext: omni.kit.widget.browser_bar-2.0.9] shutdown
[116.136s] [ext: omni.kit.menu.utils-1.5.7] shutdown
[116.136s] [ext: omni.kit.notification_manager-1.0.6] shutdown
[116.137s] [ext: omni.kit.viewport.registry-104.0.5] shutdown
[116.137s] [ext: omni.kit.widget.path_field-2.0.8] shutdown
[116.137s] [ext: omni.kit.widget.versioning-1.4.6] shutdown
[116.138s] [ext: omni.kit.widget.searchable_combobox-1.0.5] shutdown
[116.138s] [ext: omni.kit.widget.nucleus_connector-1.1.4] shutdown
2024-10-06 15:08:13 [116,122ms] [Warning] [omni.ext._impl._internal] omni.kit.widget.nucleus_connector-1.1.4 -> <class 'omni.kit.widget.nucleus_connector.extension.NucleusConnectorExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'method'>, id: 140056081763712"]
[116.152s] [ext: omni.kit.search_core-1.0.5] shutdown
[116.152s] [ext: omni.kit.widget.nucleus_info-1.0.2] shutdown
[116.152s] [ext: omni.kit.window.popup_dialog-2.0.23] shutdown
[116.152s] [ext: omni.ui-2.18.6] shutdown
[116.152s] [ext: omni.kit.renderer.imgui-0.0.0] shutdown
[116.172s] [ext: omni.kit.renderer.capture-0.0.0] shutdown
[116.172s] [ext: omni.kit.renderer.core-0.0.0] shutdown
[116.179s] [ext: omni.kit.renderer.init-0.0.0] shutdown
[116.179s] [ext: omni.appwindow-1.1.5] shutdown
[116.179s] [ext: carb.windowing.plugins-1.0.0] shutdown
[116.180s] [ext: omni.hydra.usdrt_delegate-7.2.39] shutdown
[116.180s] [ext: omni.activity.core-1.0.1] shutdown
[116.181s] [ext: omni.resourcemonitor-105.0.0] shutdown
[116.182s] [ext: omni.hydra.scene_delegate-0.3.2] shutdown
[116.182s] [ext: omni.hydra.rtx-0.1.0] shutdown
[116.191s] [ext: omni.volume-0.4.0] shutdown
[116.191s] [ext: omni.usd.core-1.1.8] shutdown
[116.191s] [ext: omni.usd.schema.omnigraph-1.0.0] shutdown
[116.191s] [ext: omni.kit.exec.core-0.5.0] shutdown
[116.192s] [ext: omni.usd.schema.anim-0.0.0] shutdown
[116.192s] [ext: omni.usd.schema.geospatial-0.0.0] shutdown
[116.192s] [ext: omni.usd.schema.audio-0.0.0] shutdown
[116.192s] [ext: omni.usd.schema.omniscripting-1.0.0] shutdown
[116.192s] [ext: omni.usd.schema.semantics-0.0.0] shutdown
[116.192s] [ext: omni.usd.schema.physics-0.0.0] shutdown
[116.192s] [ext: omni.usd_resolver-1.0.1] shutdown
[116.192s] [ext: omni.usd.libs-1.0.0] shutdown
[116.195s] [ext: omni.gpucompute.plugins-0.0.0] shutdown
[116.195s] [ext: omni.gpu_foundation-0.0.0] shutdown
[116.196s] [ext: omni.graph.exec-0.3.0] shutdown
[116.196s] [ext: omni.mdl.neuraylib-0.2.0] shutdown
[116.196s] [ext: omni.iray.libs-0.0.0] shutdown
[116.196s] [ext: omni.kit.commands-1.4.6] shutdown
[116.197s] [ext: omni.kit.actions.core-1.0.0] shutdown
[116.197s] [ext: omni.kit.clipboard-1.0.3] shutdown
[116.198s] [ext: omni.services.transport.server.http-1.3.0] shutdown
2024-10-06 15:08:13 [116,281ms] [Warning] [omni.ext._impl._internal] omni.services.transport.server.http-1.3.0 -> <class 'omni.services.transport.server.http.server.HTTPServerExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'frame'>, id: 158262368", "[1]:type: <class 'frame'>, id: 140054972120144", "[2]:type: <class 'frame'>, id: 155659584", "[3]:type: <class 'frame'>, id: 140055071953328"]
[116.310s] [ext: omni.services.transport.server.base-1.1.1] shutdown
[116.311s] [ext: omni.services.core-1.9.0] shutdown
2024-10-06 15:08:13 [116,290ms] [Warning] [omni.ext._impl._internal] omni.services.core-1.9.0 -> <class 'omni.services.core.main.ServicesCoreExtension'>: extension object is still alive, something holds a reference on it. References: ["[0]:type: <class 'method'>, id: 140056066930112"]
[116.319s] [ext: omni.kit.loop-default-0.2.0] shutdown
[116.320s] [ext: omni.kit.telemetry-0.5.0] shutdown
[116.321s] [ext: omni.kit.pip_archive-0.0.0] shutdown
[116.321s] [ext: omni.kit.pipapi-0.0.0] shutdown
[116.321s] [ext: omni.usd.config-1.0.3] shutdown
[116.321s] [ext: omni.mtlx-0.1.0] shutdown
[116.321s] [ext: omni.rtx.shadercache.vulkan-1.0.0] shutdown
[116.321s] [ext: omni.services.facilities.base-1.0.4] shutdown
[116.321s] [ext: omni.stats-0.0.0] shutdown
[116.321s] [ext: omni.timeline-1.0.9] shutdown
[116.321s] [ext: omni.omniverse_renderer_ms.dependencies.pip-0.0.0] shutdown
[116.322s] [ext: omni.client-1.0.1] shutdown
[116.323s] [ext: omni.assets.plugins-0.0.0] shutdown
[116.323s] [ext: omni.kit.async_engine-0.0.0] shutdown
2024-10-06 15:08:13 [116,097ms] [Warning] [carb.assets.plugin] Evicting asset type c1237a43d569db21 v0.1 orphaned snapshot handle 4294967296 (/home/ace/asset/Scenes/Empty Scene/studio_small_09_2k.exr)
... [repeated line showing asset snapshot]
2024-10-06 15:08:15 [118,662ms] [Error] [carb.graphics-vulkan.plugin] pipelineLayouts: 3 or descriptorSetLayouts: 10 is leaking memory. Missing call to destroyResourceBindingSignature()
2024-10-06 15:08:15 [118,907ms] [Warning] [carb] [Plugin: omni.ui] Module /opt/nvidia/omniverse/renderer/extscache/omni.ui-2.18.6+09b309e7.lx64.r.cp310/bin/libomni.ui.so remained loaded after unload request
2024-10-06 15:08:15 [118,910ms] [Warning] [carb] [Plugin: omni.fabric.plugin] Module /opt/nvidia/omniverse/renderer/extscache/omni.usd.libs-1.0.0+09b309e7.lx64.r.cp310/bin/deps/libomni.fabric.plugin.so remained loaded after unload request
2024-10-06 15:08:15 [118,911ms] [Warning] [carb] [Plugin: gpu.foundation.plugin] Module /opt/nvidia/omniverse/renderer/extscache/omni.gpu_foundation-0.0.0+09b309e7.lx64.r.cp310/bin/deps/libgpu.foundation.plugin.so remained loaded after unload request
2024-10-06 15:08:15 [118,916ms] [Warning] [carb] [Plugin: carb.dictionary.plugin] Module /opt/nvidia/omniverse/kit-sdk-launcher/kernel/plugins/libcarb.dictionary.plugin.so remained loaded after unload request
2024-10-06 15:08:15 [118,918ms] [Warning] [omni.core.ITypeFactory] Module /opt/nvidia/omniverse/renderer/extscache/omni.activity.core-1.0.1+09b309e7.lx64.r.cp310/bin/libomni.activity.core.plugin.so remained loaded after unload request.
</pre></code>

some side note, I think this tutorial here skipped the persistent volume mount part in the tutorial.

tokkio k8s built with ucf_app_builder_cli

For tokkio workflow, I have reinstalled k0s cluster again while making sure that there’s also no port collision (I’ve successfully setup Ingress and deploy my application successfully there so this should validate my claim that k0s actually works). I’ve also created the ngc-api-key-secret on both default and ucs-tokkio-audio-video-llm-app namespace.

Validate that secret exists in `default` namespace 

$ k0s kubectl get secrets ngc-api-key-secret -o yaml
apiVersion: v1
data:
  NGC_CLI_API_KEY: XXX
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"NGC_CLI_API_KEY": XXX},"kind":"Secret","metadata":{"annotations":{},"creationTimestamp":null,"name":"ngc-api-key-secret","namespace":"default"}}
  creationTimestamp: "2024-10-04T05:51:34Z"
  name: ngc-api-key-secret
  namespace: default
  resourceVersion: "803116"
  uid: 7c631613-7836-447e-84ce-876ad7dea8fb
type: Opaque
Validate that secret exists in `ucs-tokkio-audio-video-llm-app` namespace 

$ k0s kubectl get secrets ngc-api-key-secret -o yaml -n ucs-tokkio-audio-video-llm-app
apiVersion: v1
data:
  NGC_CLI_API_KEY: xxx
kind: Secret
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"NGC_CLI_API_KEY":xxx},"kind":"Secret","metadata":{"annotations":{},"name":"ngc-api-key-secret","namespace":"ucs-tokkio-audio-video-llm-app"}}
  creationTimestamp: "2024-10-06T15:48:43Z"
  name: ngc-api-key-secret
  namespace: ucs-tokkio-audio-video-llm-app
  resourceVersion: "1114016"
  uid: 2d972c6a-032b-4e3f-8e1c-c5418edaef32
type: Opaque

However, I still can’t build the yaml file with the same error message:

$ ucf_app_builder_cli app build llm-rag/tokkio-llm-app.yaml llm-rag/tokkio-llm-app-params.yaml 
2024-10-06 21:46:17,973 - AppBuilder - INFO - Syncing any missing service versions to cache...
2024-10-06 21:46:17,975 - AppBuilder - WARNING - secret 'ngc-api-key-secret' not found in 'animation-graph' (ucf.svc.ia-animation-graph-microservice)
2024-10-06 21:46:17,976 - AppBuilder - WARNING - secret 'ngc-api-key-secret' not found in 'avatar-renderer' (ucf.svc.ia-omniverse-renderer-microservice)
2024-10-06 21:46:18,054 - AppBuilder - INFO - Validating application ...
2024-10-06 21:46:18,088 - AppBuilder - INFO - Building application ucs-tokkio-audio-video-llm-app-4.1.0 ...
Output directory llm-rag/ucs-tokkio-audio-video-llm-app-4.1.0 is not empty. All contents will be removed. Proceed? [y/n] 
Output directory llm-rag/ucs-tokkio-audio-video-llm-app-4.1.0 is not empty. All contents will be removed. Proceed? [y/n] y
2024-10-06 21:46:34,512 - AppBuilder - ERROR - Failed to get values from 'https://helm.ngc.nvidia.com/eevaigoeixww/tokkio-4-0/charts/ds-sdr-envoy-0.1.22.tgz' - WARNING: Kubernetes configuration file is group-readable. This is insecure. Location: /var/lib/k0s/pki/admin.conf
Error: failed to fetch https://helm.ngc.nvidia.com/eevaigoeixww/tokkio-4-0/charts/ds-sdr-envoy-0.1.22.tgz : 403 Forbidden

I’m not sure why it doesn’t read the secrets here again.

Hardware and environment

According to @Richard3D mentioned, here’s my details for my hardware and environment:

Operating System: Ubuntu 22.04.4 LTS
GPUs: 8x NVIDIA H100 80GB (4 of them are used to deploy LLMs so only 4 available)
Kubernetes: I’m using k0s for kubernetes
Docker: there’s no docker running
Legacy python: 3.10.12

No modification was done on both build app (llm-rag/tokkio-llm-app.yaml) and params (llm-rag/tokkio-llm-app-params.yaml)
6

Upon further investigation, we found the solution for each problem as follows:

animation-pipeline fails to start pods

For animation-pipeline, the animation renderer pod (running image name nvcr.io/eevaigoeixww/animation/ia-omniverse-renderer-microservice:1.0.1) fails can be fixed via setting environment variable LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/opt/nvidia/omniverse/kit-livestream/omni.kit.livestream.rtp_udp/plugins in values.yaml file inside _build directory.

Adding this environmental variable fix this issue and the pod was able to run successfully.

On tokkio fails to build

It seems that I haven’t registered to NVAIE yet, so this resulted in 403 error.

PS. This solution was provided by Ken Liao, kudos for providing this solution!

7

Great ! I am so glad you found the solution !

8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.