I’m developing a extension for A2F that connects to a LLM.
Once i recieve the response from the LLM in text form i want to use NVIDIA Riva to synthesize speech. I already have a non-extension test environment where i have this working. But i would like to have a extension built so that i can enable this directly in Audio 2 Face.
However i am getting this error:
2024-04-02 13:18:56 [Error] [omni.ui.python] NameError: name 'riva' is not defined
2024-04-02 13:18:56 [Error] [omni.ui.python]
2024-04-02 13:18:56 [Error] [omni.ui.python] At:
2024-04-02 13:18:56 [Error] [omni.ui.python] d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/extension.py(49): ui_wavefile
2024-04-02 13:18:56 [Error] [omni.ui.python]
I have tried to include riva or riva.client in the toml dependencies but it always fail to install. My thought here is that i’m not using the correct wording.
2024-04-02 13:43:21 [Warning] [omni.kit.pipapi.pipapi] 'riva.client' failed to install.
Tried: nvidia-riva-client based on this reference. It also fails even if i replace the - with .
2024-04-02 13:57:28 [Warning] [omni.kit.pipapi.pipapi] 'nvidia-riva-client' failed to install.
Whenever i add riva.client it breaks the ui. I’m assuming this is because it fails to install at this point
Any help would be greatly appriciated. I have everything working in A2F except this.
I am using my extension in both Omnvierse Code(2023.1.1) and A2F(2023.2.0).
Just before you wrote i tried this in the main extension.py file which did not work.
implemented your code in the extension.py file and when i run it i get these errors in the console
2024-04-02 19:56:21 [Warning] [omni.kit.helper.file_utils.extension] Failed to deserialize file event uri: bad query field: ''
2024-04-02 19:56:21 [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2024-04-02 19:56:21 [Warning] [omni.ext._impl._internal] There is already a module 's3transfer' in 'd:\nvidia\librarypath\code-2023.1.1\extscache\omni.pip.compute-1.1.0+wx64\pip_prebundle\s3transfer\__init__.py'. Can't import another module with the same name in public space from 'd:\nvidia\librarypath\code-2023.1.1\extscache\omni.replicator.core-1.9.8+105.0.wx64.r.cp310\pip_prebundle'.
2024-04-02 19:56:21 [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
2024-04-02 19:57:06 [Warning] [omni.kit.pipapi.pipapi] 'nvidia.riva.client' failed to install.
2024-04-02 19:57:49 [Warning] [omni.kit.pipapi.pipapi] 'nvidia-riva-client' failed to install.
2024-04-02 19:57:50 [Warning] [omni.kit.viewport.menubar.core.menu_item.viewport_menu_item] ViewportMenuItem order argument is deprecated, use order_setting_path
2024-04-02 19:57:54 [Warning] [carb.tasking.plugin] carb.tasking:pinToCurrentThread(): Task 326417514660 pinned to current thread. Thread pinning is inefficient and should be avoided!
2024-04-02 19:57:54 [Warning] [omni.fabric.plugin] getType called on non-existent attribute /Render/RenderProduct_omni_kit_widget_viewport_ViewportTexture_0 camera
2024-04-02 19:57:54 [Warning] [omni.fabric.plugin]
2024-04-02 20:01:13 [Warning] [omni.kit.pipapi.pipapi] 'nvidia.riva.client' failed to install.
2024-04-02 20:01:59 [Warning] [omni.kit.pipapi.pipapi] 'nvidia-riva-client' failed to install.
I get this when i try to run the particular section that needs riva.client
2024-04-02 20:02:35 [Error] [omni.ui.python] NameError: name 'riva' is not defined
2024-04-02 20:02:35 [Error] [omni.ui.python]
2024-04-02 20:02:35 [Error] [omni.ui.python] At:
2024-04-02 20:02:35 [Error] [omni.ui.python] d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/extension.py(48): ui_wavefile
2024-04-02 20:02:35 [Error] [omni.ui.python]
That tutorial is great. I’ve been following that way for most of my plugins for my extension.
I’ve managed to try out names, random, and verify that other plugins of my script are working. Like for instance grpc.
Maybe i can try to uninstall Isaac Sim / Replicator?
Based on this:
2024-04-02 19:56:21 [Warning] [omni.ext._impl._internal] There is already a module 's3transfer' in 'd:\nvidia\librarypath\code-2023.1.1\extscache\omni.pip.compute-1.1.0+wx64\pip_prebundle\s3transfer\__init__.py'. Can't import another module with the same name in public space from 'd:\nvidia\librarypath\code-2023.1.1\extscache\omni.replicator.core-1.9.8+105.0.wx64.r.cp310\pip_prebundle'
I’m wondering if we can use the example from Code somehow
# omni.kit.pipapi extension is required
import omni.kit.pipapi
# It wraps `pip install` calls and reroutes package installation into user specified environment folder.
# That folder is added to sys.path.
# Note: This call is blocking and slow. It is meant to be used for debugging, development. For final product packages
# should be installed at build-time and packaged inside extensions.
omni.kit.pipapi.install(
package="semver",
version="2.13.0",
module="semver", # sometimes module is different from package name, module is used for import check
ignore_import_check=False,
ignore_cache=False,
use_online_index=True,
surpress_output=False,
extra_args=[]
)
# use
import semver
ver = semver.VersionInfo.parse('1.2.3-pre.2+build.4')
print(ver)
I’ve tried this for two separate extensions now. Using Code 2023.1.1. Both fail to install riva.
I am using both VS Code and Omniverse + Omniverse Code as Administrator
Here is the whole log from the console when starting Code 2023.1.1. Here i have just added the “nvidia-riva-client” to the .toml requirements. I have not added the import part.
2024-04-03 17:18:17 [Warning] [omni.kit.helper.file_utils.extension] Failed to deserialize file event uri: bad query field: ''
2024-04-03 17:18:17 [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2024-04-03 17:18:17 [Warning] [omni.replicator.core.scripts.annotators] Annotator PostProcessDispatch is already registered, overwriting annotator template
2024-04-03 17:18:55 [Warning] [omni.kit.pipapi.pipapi] 'nvidia-riva-client' failed to install.
2024-04-03 17:19:36 [Warning] [omni.kit.pipapi.pipapi] 'nvidia-riva-client' failed to install.
2024-04-03 17:19:37 [Warning] [omni.kit.viewport.menubar.core.menu_item.viewport_menu_item] ViewportMenuItem order argument is deprecated, use order_setting_path
2024-04-03 17:19:40 [Warning] [carb.tasking.plugin] carb.tasking:pinToCurrentThread(): Task 764504178704 pinned to current thread. Thread pinning is inefficient and should be avoided!
2024-04-03 17:19:40 [Warning] [omni.fabric.plugin] getType called on non-existent attribute /Render/RenderProduct_omni_kit_widget_viewport_ViewportTexture_0 camera
2024-04-03 17:19:40 [Warning] [omni.fabric.plugin]
Can’t think of anything else that i have trouble installing.
Do you have an example of something i could try that is similar to nvidia-riva-client?
[package]
# Semantic Versioning is used: https://semver.org/
version = "1.0.0"
# Lists people or organizations that are considered the "authors" of the package.
authors = ["NVIDIA"]
# The title and description fields are primarily for displaying extension info in UI
title = "MY EXTENSION TITLE"
description="A simple python extension example to use as a starting point for your extensions."
# Path (relative to the root) or content of readme markdown file for UI.
readme = "docs/README.md"
# URL of the extension source repository.
repository = ""
# One of categories for UI.
category = "Example"
# Keywords for the extension
keywords = ["kit", "example"]
# Location of change log file in target (final) folder of extension, relative to the root.
# More info on writing changelog: https://keepachangelog.com/en/1.0.0/
changelog="docs/CHANGELOG.md"
# Preview image and icon. Folder named "data" automatically goes in git lfs (see .gitattributes file).
# Preview image is shown in "Overview" of Extensions window. Screenshot of an extension might be a good preview image.
preview_image = "data/preview.png"
# Icon is shown in Extensions window, it is recommended to be square, of size 256x256.
icon = "data/icon.png"
# Use omni.ui to build simple UI
[dependencies]
"omni.kit.uiapp" = {}
# Main python module this extension provides, it will be publicly available as "import MY EXTENSION NAME".
[[python.module]]
name = "MY EXTENSION NAME"
[python.pipapi]
requirements = ["names", "random", "requests", "scipy.io.wavfile", "subprocess", "json", "os", "numpy", "IPython"]
use_online_index = true
[[test]]
# Extra dependencies only to be used during test run
dependencies = [
"omni.kit.ui_test" # UI testing extension
]
Re-install complete of Omniverse Code 2023.1.1
I only have nvidia-riva-client in the .toml at this point.
I only have my extension active.
Re-activation of extension gives this error:
2024-04-03 18:20:02 [Error] [omni.kit.app._impl] [py stderr]: d:\nvidia\librarypath\code-2022.3.3\extscache\omni.isaac.core_archive-2.0.1+104.1.wx64\pip_prebundle\requests\__init__.py:104: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (None)/charset_normalizer (3.0.1) doesn't match a supported version!
2024-04-03 18:20:02 [Error] [omni.kit.app._impl] RequestsDependencyWarning)
2024-04-03 18:20:02 [Error] [omni.kit.app._impl]
2024-04-03 18:22:40 [Error] [omni.kit.pipapi.pipapi] Failed to import python module nvidia-riva-client. Error: No module named 'nvidia-riva-client'
2024-04-03 18:22:40 [Warning] [omni.kit.pipapi.pipapi] [omni.kit.pipapi] 'nvidia-riva-client' failed to install.
Solved
It seems that some files that i had in my extension folder were causing havoc. Cleaning my folder up i am now able to get it to work in Omniverse Code.
Thank you Ehsan.HM for you assistance and patience. It has been invaluable!
Superb, if you get more errors regarding protobuf, it’s because the protobuf version that Audio2Face uses is older. The workaround is to load your extension in a newer app e.g. USD Composer, then copy the generated protobuf folder into audio2face folder. In my case it’s like this:
From C:\Users\{USER}\AppData\Local\ov\data\Kit\USD.Composer\2023.2\pip3-envs\default\google\protobuf
To C:\Users\{USER}\AppData\Local\ov\pkg\prod-audio2face-2023.2.0\exts\omni.audio2face.player_deps\deps\py\google\protobuf
and now it works
I have yet to try this. I got sidetracked by debugging other stuff. Will test this today.
Thank you for your insights and quick reply. :)
I have tried this now without any success. I’m thinking i’m not installing protobuf correctly.
I have this in my A2F folder: ov\nucleus\Kit\Audio2Face\2023.2\pip3-envs\default
this is how it looks inside the protobuf folders(some folders do not have the.txt files):
this is how the inside of my system protobuf 4.25.3 folder looks like
i just want to emphasize that i am able to run riva services on my system and through docker without any complications(protobuf or grpc errors). I also have a working python package that is working with this exact code.
Another question that i have is why is it working in Code and not any other application?
I don’t know what i did differently. But i re-installed A2F and restarted my computer a couple of times.
Copied the protobuf folder from Code to A2F when both apps were shut down.
Started A2F up again and now it works except for Audio 2 Face…
I think the question here is rather how do i get my extension to downgrade Riva to Protobuf 3.17.3 which is the same as A2F. → Tried this. Didn’t work.
