Audio 2 Face Extension - PiP install

W1r3d · April 5, 2024, 7:30am

Hello

I’m trying to install the required modules through my python only extension for A2F 2023.2.0 as indicated here. I have the extension fully working in Omniverse Code 2022.3.3 so i can confirm that i am able to run the extension successfully and stream audio to Audio 2 Face. The problem is when i try to load the extension in Audio 2 Face.

Required pip packages to be installed for A2F 2023.1
grpcio==1.51.3
numpy==1.22.4
scipy==1.8.1
sounddevice==0.4.2
protobuf==3.17.3

I can verify that protobuf 3.17.3 and grpcio 1.51.3 is already installed on my system and that is the version that is detected in A2F when i use the script editor in A2F.

import google.protobuf
print(google.protobuf.__version__)

import grpc
print(grpc.__version__)

“protobuf==3.17.3” and “grpcio==1.51.3” for my A2F extension.
I i use the above in my .toml file like this

[python.pipapi]
requirements = ["requests", "scipy.io.wavfile", "subprocess", "json", "os", "numpy", "IPython", "nvidia-riva-client", "protobuf==3.17.3", "grpc==1.51.3", "sounddevice==0.4.2"]
use_online_index = true

I get these errors

2024-04-05 07:26:08  [Error] [omni.kit.pipapi.pipapi] Failed to import python module protobuf. Error: No module named 'protobuf'
2024-04-05 07:26:08  [Warning] [omni.kit.pipapi.pipapi] 'protobuf==3.17.3' failed to install.
2024-04-05 07:26:18  [Error] [omni.kit.pipapi.pipapi] Failed to import python module grpcio. Error: No module named 'grpcio'
2024-04-05 07:26:18  [Warning] [omni.kit.pipapi.pipapi] 'grpcio==1.51.3' failed to install.
...

Here are some references from NVIDIA that i’m currently looking at to see if i am doing something wrong:

From what i gather i do not need to build the extension since it is a python only extension.
But if i want the extension to load quicker or if i want to publish it i will need to build it.

W1r3d · April 5, 2024, 3:33pm

If i remove both protobuf and grcpio from python.pipapi in .toml file i get this error in A2F instead. Which results ini a failiure to load for the extension. Happy to provide more detailed error msg in DM.

2024-04-05 15:06:29  [Error] [omni.ext._impl.custom_importer] ImportError: cannot import name 'builder' from 'google.protobuf.internal' (d:/nvidia/librarypath/audio2face-2023.2.0/exts/omni.audio2face.player_deps/deps/py/google/protobuf/internal/__init__.py)

W1r3d · April 8, 2024, 8:52am

Still working to solve this for both Machinema and for Audio 2 Face.
I have no errors in Code

Version 1
I install in runtime in my extension.py. No istall errors but the startup is so fast so i’m not sure it does anything.

omni.kit.pipapi.install(
    package="google.protobuf",
    version="3.17.3",
    module="google.protobuf", # sometimes module is different from package name, module is used for import check
    ignore_import_check=False,
    ignore_cache=False,
    use_online_index=True,
    surpress_output=False,
    extra_args=[]
)

import google.protobuf
ver = google.protobuf('3.17.3')

Error:

2024-04-08 08:47:01  [Error] [omni.ext._impl.custom_importer] Failed to import python module curiousxr.avatar.extension. Error: cannot import name 'builder' from 'google.protobuf.internal' (d:/nvidia/librarypath/audio2face-2023.2.0/exts/omni.audio2face.player_deps/deps/py/google/protobuf/internal/__init__.py). Traceback:

Version 2
I install through .toml file by adding it as a requirement. Install errors ensue. It does however “think” a while before giving me the answer.

[python.pipapi]
requirements = ["requests", "scipy.io.wavfile", "subprocess", "json", "os", "numpy", "IPython", "nvidia-riva-client", "grpc==1.51.3", "protobuf==3.17.3"]
use_online_index = true

Error

2024-04-08 08:49:46  [Warning] [omni.kit.pipapi.pipapi] 'protobuf==3.17.3' failed to install.

Version 3
If i instead use google.protobuf in the .toml i get no errors. But the startup is so fast that i don’t think it actually does anyhing.

[python.pipapi]
requirements = ["requests", "scipy.io.wavfile", "subprocess", "json", "os", "numpy", "IPython", "nvidia-riva-client", "grpc==1.51.3", "google.protobuf==3.17.3"]
use_online_index = true

W1r3d · April 8, 2024, 2:23pm

Even when i just use import riva.client in the extension.py i still get a protobuf error.
I’m a bit confused by this.

import riva.client

If i disable these lines the extension loads. But nothing works.

import riva.client
import IPython.display as ipd

Error

2024-04-08 17:20:04  [Error] [omni.ui.python] AttributeError: 'CuriousxrAvatarExtensionExtension' object has no attribute 'string_model_result'
2024-04-08 17:20:04  [Error] [omni.ui.python] 
2024-04-08 17:20:04  [Error] [omni.ui.python] At:
2024-04-08 17:20:04  [Error] [omni.ui.python]   d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/extension.py(187): on_end_edit_string
2024-04-08 17:20:04  [Error] [omni.ui.python] 
2024-04-08 17:20:04  [Error] [omni.ui.python] AttributeError: 'CuriousxrAvatarExtensionExtension' object has no attribute 'string_model_result'
2024-04-08 17:20:04  [Error] [omni.ui.python] 
2024-04-08 17:20:04  [Error] [omni.ui.python] At:
2024-04-08 17:20:04  [Error] [omni.ui.python]   d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/extension.py(187): on_end_edit_string
2024-04-08 17:20:04  [Error] [omni.ui.python]   d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/extension.py(103): ui_llmpostrequest
2024-04-08 17:20:04  [Error] [omni.ui.python]

I can confirm that my streaming to A2F still works.
It’s the sending question to LLM and riva synthesize speech that are causing some kind of havoc…

but nvidia-riva-client installs through .toml without errors…

I can confirm that IPython is installed on my system but not in Audio 2 Face.

Ehsan.HM · April 8, 2024, 6:31pm

Regarding cannot import name 'builder' from 'google.protobuf.internal' this link might help:

python - ImportError: cannot import name ‘builder’ from ‘google.protobuf.internal’ - Stack Overflow

W1r3d · April 8, 2024, 6:40pm

I can already confirm that both grpc and protobuf are installed and they themselves are not generating any errors. However when i activate import riva.client THATS when i get the error.

I am using the specified version of protobuf(3.17.3) and grpc(1.51.3)

I’m not sure Riva and IPython gets correctly imported and i’m not sure what to do about it.
I can confirm that i have IPython on my system but A2F does not have it.
I’m not sure how to verify riva services in A2F yet.

If i try to install ipython through .toml

2024-04-08 18:42:49  [Error] [omni.kit.pipapi.pipapi] Failed to import python module ipython. Error: No module named 'ipython'
2024-04-08 18:42:49  [Warning] [omni.kit.pipapi.pipapi] 'ipython==8.23.0' failed to install.

or if i use IPython instead. Tried a few different versions.

2024-04-08 18:48:14  [Error] [omni.kit.pipapi.pipapi] Failed to import python module IPython. Error: cannot import name 'Self' from 'typing_extensions' (d:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.pip_archive-0.7.0+wx64/pip_prebundle/typing_extensions.py)
2024-04-08 18:48:14  [Warning] [omni.kit.pipapi.pipapi] 'IPython==8.22.2' failed to install.
2024-04-08 18:51:28  [Error] [omni.kit.pipapi.pipapi] Failed to import python module IPython. Error: cannot import name 'Self' from 'typing_extensions' (d:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.pip_archive-0.7.0+wx64/pip_prebundle/typing_extensions.py)
2024-04-08 18:51:28  [Warning] [omni.kit.pipapi.pipapi] 'IPython==6.5.0' failed to install.
2024-04-08 18:52:18  [Error] [omni.kit.pipapi.pipapi] Failed to import python module IPython. Error: cannot import name 'Self' from 'typing_extensions' (d:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.pip_archive-0.7.0+wx64/pip_prebundle/typing_extensions.py)
2024-04-08 18:52:18  [Warning] [omni.kit.pipapi.pipapi] 'IPython==5.8.0' failed to install.
2024-04-08 18:52:51  [Error] [omni.kit.pipapi.pipapi] Failed to import python module IPython. Error: cannot import name 'Self' from 'typing_extensions' (d:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.pip_archive-0.7.0+wx64/pip_prebundle/typing_extensions.py)
2024-04-08 18:52:51  [Warning] [omni.kit.pipapi.pipapi] 'IPython==3.2.3' failed to install.
2024-04-08 18:53:39  [Error] [omni.kit.pipapi.pipapi] Failed to import python module IPython. Error: cannot import name 'Self' from 'typing_extensions' (d:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.pip_archive-0.7.0+wx64/pip_prebundle/typing_extensions.py)
2024-04-08 18:53:39  [Warning] [omni.kit.pipapi.pipapi] 'IPython==2.4.0' failed to install.
2024-04-08 18:54:09  [Warning] [omni.kit.pipapi.pipapi] 'IPython==1.0.0' failed to install.

tried installing nvidia-riva-client 2.14.0 but it failed.
Pip install on my system worked though. The numbers in the mmiddle are Protobuf and below Grpc

IPython is installed.
4.25.3
1.60.1
NVIDIA Riva Client is installed.

W1r3d · April 9, 2024, 8:53am

I can verify that i can install IPython, Protobuf, GRPC and Nvidia Riva client in Omniverse Code when i start up the extension. Using this in the script editor:

import google.protobuf
import grpc
try:
    import IPython
    print("IPython is installed.")
except ImportError:
    print("IPython is not installed.")

print(google.protobuf.__version__)
print(grpc.__version__)

try:
    import riva.client
    print("NVIDIA Riva Client is installed.")
except ImportError:
    print("NVIDIA Riva Client is not installed.")

Extension system cache settings between Code and A2F is different.
Code points to: Nvidia\omniverseCache\exts
A2F points to: ov\nucleus\exts

Ran the link_app.bat got a warning for the extension path to Audio 2 Face but it said that it fixed it and printed Success! in the end. Still can’t install or enable IPython or nvidia-riva-client

Tried some of the advice in the link that youposted Ehsan.HM: python - ImportError: cannot import name ‘builder’ from ‘google.protobuf.internal’ - Stack Overflow
No success since i can’t upgrade or downgrade the A2F Protobuf version. I even tried overwriting the files manually. But it just created more problems. Re-installing A2F now to make sure it’s a clean install.

Ruling out IPython as something that i need. It is NOT a dependency.

I have now also tried what you suggested here Ehsan: Audio 2 Face - Extension - #19 by W1r3d

W1r3d · April 11, 2024, 1:22pm

I actually can’t make sense of this.
If i don’t have the correct protobuf installed on my system. I am unable to send a audio stream to A2F.
This makes sense. It tells me to downgrade to something below 3.20(I used 3.17.3). If i do that i can send a stream of audio from Omniverse Code to audio to face and it will play.
I will have no errors with my extension in Code. I can use a locally deployed docker riva quickstart 2.14.0 server to synthesize the audio.

If i try to activate the extension in Audio 2 Face i get this error:

2024-04-11 14:55:08  [Error] [omni.ext._impl.custom_importer] Failed to import python module curiousxr.avatar.extension. Error: cannot import name 'builder' from 'google.protobuf.internal' (d:/nvidia/librarypath/audio2face-2023.2.0/exts/omni.audio2face.player_deps/deps/py/google/protobuf/internal/__init__.py). Traceback:

I have already tried to copy the protobuf folder from a more recent project into the protobuf folder for A2F.
The extenion will load then. But NONE of the A2F extensions will load. So that doesn’t work.
3.17.3 is the protobuf version A2F is using as i’ve listed before in the beginning of this post

Ehsan.HM · April 12, 2024, 3:51pm

Can you try installing iPython 7.34.0 instead of 8.23.0?

W1r3d · April 12, 2024, 8:40pm

Downgrading system IPython is successful
If i try to do this through the .toml file i get this when enabling the extension in A2F

2024-04-12 20:39:08  [Error] [omni.kit.pipapi.pipapi] Failed to import python module IPython. Error: cannot import name 'Self' from 'typing_extensions' (d:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.pip_archive-0.7.0+wx64/pip_prebundle/typing_extensions.py)
2024-04-12 20:39:08  [Warning] [omni.kit.pipapi.pipapi] 'IPython==7.34.0' failed to install.

Ehsan.HM · April 12, 2024, 8:42pm

what if you install it using C:\Users\{USER}\AppData\Local\ov\pkg\prod-audio2face-2023.2.0\kit\python.bat -m pip install ipython==7.34

W1r3d · April 12, 2024, 8:48pm

Windows Terminal log

Installing collected packages: wcwidth, traitlets, parso, pygments, prompt-toolkit, pickleshare, matplotlib-inline, jedi, decorator, backcall, ipython
  WARNING: The script pygmentize.exe is installed in 'D:\Nvidia\LibraryPath\audio2face-2023.2.0\kit\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts iptest.exe, iptest3.exe, ipython.exe and ipython3.exe are installed in 'D:\Nvidia\LibraryPath\audio2face-2023.2.0\kit\python\Scripts' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed backcall-0.2.0 decorator-5.1.1 ipython-7.34.0 jedi-0.19.1 matplotlib-inline-0.1.6 parso-0.8.4 pickleshare-0.7.5 prompt-toolkit-3.0.43 pygments-2.17.2 traitlets-5.14.2 wcwidth-0.2.13
WARNING: You are using pip version 21.2.1+nv1; however, version 24.0 is available.
You should consider upgrading via the 'D:\Nvidia\LibraryPath\audio2face-2023.2.0\kit\python\python.exe -m pip install --upgrade pip' command.

I’m not getting any IPython errors in A2F when activating my Extension.
I’m back to just getting this error

2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer] Failed to import python module curiousxr.avatar.extension. Error: cannot import name 'builder' from 'google.protobuf.internal' (d:/nvidia/librarypath/audio2face-2023.2.0/exts/omni.audio2face.player_deps/deps/py/google/protobuf/internal/__init__.py). Traceback:
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer] Traceback (most recent call last):
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "d:\nvidia/librarypath/audio2face-2023.2.0/kit/kernel/py\omni\ext\_impl\custom_importer.py", line 76, in import_module
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     return importlib.import_module(name)
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "d:\nvidia\librarypath\audio2face-2023.2.0\kit\python\lib\importlib\__init__.py", line 126, in import_module
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     return _bootstrap._gcd_import(name[level:], package, level)
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/__init__.py", line 1, in <module>
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     from .extension import *
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/myextensions/curiousxr-exts-avatar/exts/curiousxr.avatar.extension/curiousxr/avatar/extension/extension.py", line 10, in <module>
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     import riva.client
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "D:\ov\nucleus\Kit\Audio2Face\2023.2\pip3-envs\default\riva\client\__init__.py", line 4, in <module>
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     from riva.client.asr import (
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "D:\ov\nucleus\Kit\Audio2Face\2023.2\pip3-envs\default\riva\client\asr.py", line 16, in <module>
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     import riva.client.proto.riva_asr_pb2 as rasr
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]   File "D:\ov\nucleus\Kit\Audio2Face\2023.2\pip3-envs\default\riva\client\proto\riva_asr_pb2.py", line 9, in <module>
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer]     from google.protobuf.internal import builder as _builder
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer] ImportError: cannot import name 'builder' from 'google.protobuf.internal' (d:/nvidia/librarypath/audio2face-2023.2.0/exts/omni.audio2face.player_deps/deps/py/google/protobuf/internal/__init__.py)
2024-04-12 20:51:07  [Error] [omni.ext._impl.custom_importer] 
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'curiousxr.avatar.extension' in 'd:\nvidia\myextensions\curiousxr-exts-avatar\exts\curiousxr.avatar.extension' failed to load.
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin] 
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin] At:
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:\nvidia/librarypath/audio2face-2023.2.0/kit/kernel/py\omni\ext\_impl\_internal.py(213): startup
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:\nvidia/librarypath/audio2face-2023.2.0/kit/kernel/py\omni\ext\_impl\_internal.py(328): startup_extension
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   PythonExtension.cpp::startup()(2): <module>
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/audio2face-2023.2.0/kit/exts/omni.kit.window.extensions/omni/kit/window/extensions/ext_commands.py(29): do
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/audio2face-2023.2.0/kit/exts/omni.kit.commands/omni/kit/undo/undo.py(80): execute
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/audio2face-2023.2.0/kit/exts/omni.kit.commands/omni/kit/commands/command.py(463): execute
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/audio2face-2023.2.0/kit/exts/omni.kit.window.extensions/omni/kit/window/extensions/common.py(288): toggle_extension
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/audio2face-2023.2.0/kit/exts/omni.kit.window.extensions/omni/kit/window/extensions/ext_components.py(93): toggle
2024-04-12 20:51:07  [Error] [carb.scripting-python.plugin] 
2024-04-12 20:51:07  [Error] [omni.ext.plugin] [ext: curiousxr.avatar.extension-1.0.0] Failed to startup python extension.