Connecting Audio 2 Face <> Omniverse Code

[W1r3d]

Hello

I’ve set up the A2F workflow with some characters now.
I’ve enables the Metahuman connection and can stream to UE5 this way.
I’ve deployed locally a Riva quickstart server and can run ASR, TTS, NLP

What i’m wondering is if i can connect to another Omniverse Code or Composer app like i connect to UE5 and control a character in the same way?

This would remove a lot of the overhead of having 3 heads(at least) to control one character.
Any suggestions on where to look or references would be greatly appriciated.

[W1r3d]

Is this close to the answer?

Bring in the A2F file into Code as a separate Layer?

Streaming Audio2Face with Audio2Gesture

So this must be possible, but I can’t figure it out. How do I get audio streaming to Audio2Face with Audio2Gesture. They seem sadly so close together and yet so far apart. I can stream audio to do the face in Audio2Face - but I can’t seem to get gestures in there. Trying to add the extension then wants anim.graph extension as a dependency and I start getting permission errors from there. It doesn’t feel like it’s meant to happen. And gestures are in Machinima, but what I’ve seen says you export…

Hmm ok, Interesting.

• I can import the A2F file as a layer.
• If i press play it plays the animation(idle)
• The A2F player is not present so can’t send audio to it.
  – Q: Is is possible to enable A2F Extensions in Omniverse Code?
  – There is no A2F player in the scene so testing it with this is not possible:

D:\Nvidia\LibraryPath\audio2face-2023.2.0\exts\omni.audio2face.player\omni\audio2face\player\scripts\streaming_server\test_client.py C:\Windows\System32\riva_quickstart_v2.14.0\synthesized_audio.wav /World/audio2face/Player 0

[W1r3d]

Using inspiration from this thread: Is this combining possible to do in create? - #2 by pekka.varis
Trying to enable A2F extension by adding it to Extension Search Paths in Code gives these errors:
Path: d:/nvidia/librarypath/audio2face-2023.2.0/extscache

2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: apps] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/apps' or 'd:/nvidia/librarypath/audio2face-2023.2.0/apps/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: PACKAGE-LICENSES] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/PACKAGE-LICENSES' or 'd:/nvidia/librarypath/audio2face-2023.2.0/PACKAGE-LICENSES/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: exts] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/exts' or 'd:/nvidia/librarypath/audio2face-2023.2.0/exts/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: dev] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/dev' or 'd:/nvidia/librarypath/audio2face-2023.2.0/dev/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: logs] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/logs' or 'd:/nvidia/librarypath/audio2face-2023.2.0/logs/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: site] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/site' or 'd:/nvidia/librarypath/audio2face-2023.2.0/site/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: kit] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/kit' or 'd:/nvidia/librarypath/audio2face-2023.2.0/kit/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: ue-plugins] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/ue-plugins' or 'd:/nvidia/librarypath/audio2face-2023.2.0/ue-plugins/config'
2024-05-04 23:08:53  [Warning] [omni.ext.plugin] [ext: extscache] Extensions config 'extension.toml' doesn't exist 'd:/nvidia/librarypath/audio2face-2023.2.0/extscache' or 'd:/nvidia/librarypath/audio2face-2023.2.0/extscache/config'
2024-05-04 23:09:06  [Error] [carb.dictionary.serializer-toml.plugin] TOML parsing failed: Redefinition of table package at line 104
2024-05-04 23:09:06  [Error] [omni.ext.plugin] [ext: omni.anim.curve-105.23.0+105.1.wx64.r.cp310.ub3f] Failed to parse a config. Extension failed to load. . Path: 'd:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.anim.curve-105.23.0+105.1.wx64.r.cp310.ub3f'
2024-05-04 23:09:07  [Error] [carb.dictionary.serializer-toml.plugin] TOML parsing failed: Key omni.kit.actions.core already present at line 37
2024-05-04 23:09:07  [Error] [omni.ext.plugin] [ext: omni.kit.tool.asset_importer-2.5.0] Failed to parse a config. Extension failed to load. . Path: 'd:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.kit.tool.asset_importer-2.5.0'
2024-05-04 23:09:07  [Warning] [omni.ext.plugin] Extension with the same id is already registered. ext_id: 'omni.services.transport.server.base-1.1.1', ext 1: 'd:/nvidia/librarypath/code-2022.3.3/extscache/omni.services.transport.server.base-1.1.1', ext 2: 'd:/nvidia/librarypath/audio2face-2023.2.0/extscache/omni.services.transport.server.base-1.1.1'

[W1r3d]

Trying the same in Omniverse Composer generates the following errors:

2024-05-04 23:37:18  [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2024-05-04 23:37:18  [Warning] [omni.ext.plugin] [ext: omni.graph.io-1.6.0] 'd:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph.io-1.6.0+wx64.r.cp310/bin' in '[[native.plugin]]' was not found.
2024-05-04 23:37:18  [Warning] [omni.stageupdate.plugin] Deprecated: direct use of IStageUpdate callbacks is deprecated. Use IStageUpdate::getStageUpdate instead.
2024-05-04 23:37:18  [Warning] [omni.kit.profiler.window] remove _SpanInstance.__lt__ and use insort 'key' arg instead
2024-05-04 23:37:18  [Warning] [pxr.AudioSchema] pxr.AudioSchema is deprecated - please use pxr.OmniAudioSchema instead
2024-05-04 23:37:18  [Warning] [omni.particle.system.core.scripts.extension] 
2024-05-04 23:37:18  [Warning] [omni.particle.system.core.scripts.extension]             ATTENTION!: omni.particle.system.core is currently undergoing extensive (breaking) changes.
2024-05-04 23:37:18  [Warning] [omni.particle.system.core.scripts.extension]             Please be aware that systems built with the existing extension will need to be rebuilt in
2024-05-04 23:37:18  [Warning] [omni.particle.system.core.scripts.extension]             the USD Composer 2023.2 release.
2024-05-04 23:37:21  [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated action as SPACE (On Press) with omni.kit.widget.toolbar.omni.kit.widget.toolbar::toolbar::play!
2024-05-04 23:37:21  [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] Cannot register Global [omni.kit.widget.toolbar] omni.kit.widget.toolbar::toolbar::play.SPACE, error code: Duplicated action definition
2024-05-04 23:37:23  [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated action as SPACE (On Press) with omni.kit.widget.toolbar.omni.kit.widget.toolbar::toolbar::play!
2024-05-04 23:37:23  [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] Cannot register Global [omni.kit.widget.toolbar] omni.kit.widget.toolbar::toolbar::play.SPACE, error code: Duplicated action definition
2024-05-04 23:38:02  [Warning] [carb] Client omni.ui has acquired [omni::ui::IGlyphManager v1.0] 100 times. Consider accessing this interface with carb::getCachedInterface() (Performance warning)

[markrmiller]

You would think this should be possible in Composer since 105 came out. It’s supposed to have made it easy to compose whatever extensions you want. I have not tried myself, but I assume there must be a way.

[W1r3d]

Trying to enable A2F in Composer
Loading just the extcashe from A2F did nothing. Loading both the A2F and Machinema enabled a lot of third party plugins. I also got a lot of errors when i didn’t have A2F loaded and only machinema. If i only have A2F extcache there are no plugins at all.

In Extensions > Settings

• Load extcache from both Audio 2 Face and Machinema

Without errors i can enable:

• omni.audio2face.inference_deps
• omni.audio2face.inference
• omni.audio2face.utility
• omni.audio2face.common
• omni.audio2face.player_deps
• omni.audio2face.ui_common

I get errors when trying to load

• omni.audio2face.ui.player
• omni.audio2face.player(this gets automatically loaded when i load the ui.player)

omni.audio2face.ui.player and .player errors.
I’m seeing some common themes at a cursory glance

• cygrpc from grpc._cython (ok so cygrpc is the server. Did pip install and no difference. )
• NumPy version >=1.16.5 <1.23.0 ( i have 1.26.4)
  • This is causing all sorts of problems

2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer] Failed to import python module omni.audio2face.player. Error: cannot import name 'cygrpc' from 'grpc._cython' (d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/grpc/_cython/__init__.py). Traceback:
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer] Traceback (most recent call last):
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\omni\ext\_impl\custom_importer.py", line 76, in import_module
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     return importlib.import_module(name)
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:\nvidia\librarypath\create-2023.2.5\kit\python\lib\importlib\__init__.py", line 126, in import_module
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     return _bootstrap._gcd_import(name[level:], package, level)
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/__init__.py", line 2, in <module>
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     from .scripts.extension import *
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/extension.py", line 15, in <module>
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     from .instances import (
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/instances.py", line 20, in <module>
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     from .streaming_server import StreamingServer
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/streaming_server/__init__.py", line 1, in <module>
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     from .server import *
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/streaming_server/server.py", line 5, in <module>
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     import grpc
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/grpc/__init__.py", line 23, in <module>
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer]     from grpc._cython import cygrpc as _cygrpc
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer] ImportError: cannot import name 'cygrpc' from 'grpc._cython' (d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/grpc/_cython/__init__.py)
2024-05-05 18:32:41  [Error] [omni.ext._impl.custom_importer] 
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.audio2face.player' in 'd:\nvidia\librarypath\machinima-2022.3.0\extscache\omni.audio2face.player-104.8.5+104.1.wx64' failed to load.
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin] 
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin] At:
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\omni\ext\_impl\_internal.py(213): startup
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\omni\ext\_impl\_internal.py(328): startup_extension
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   PythonExtension.cpp::startup()(2): <module>
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_commands.py(29): do
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/undo/undo.py(80): execute
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/commands/command.py(463): execute
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/common.py(288): toggle_extension
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_components.py(93): toggle
2024-05-05 18:32:41  [Error] [carb.scripting-python.plugin] 
2024-05-05 18:32:41  [Error] [omni.ext.plugin] [ext: omni.audio2face.player-104.8.5] Failed to startup python extension.
2024-05-05 18:32:41  [Error] [omni.ext.python] ImportError: cannot import name 'cygrpc' from 'grpc._cython' (d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/grpc/_cython/__init__.py)
2024-05-05 18:32:41  [Error] [omni.ext.python] 
2024-05-05 18:32:41  [Error] [omni.ext.python] At:
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/grpc/__init__.py(23): <module>
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/streaming_server/server.py(5): <module>
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/streaming_server/__init__.py(1): <module>
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/instances.py(20): <module>
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/scripts/extension.py(15): <module>
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player-104.8.5+104.1.wx64/omni/audio2face/player/__init__.py(2): <module>
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:41  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1050): _gcd_import
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:\nvidia\librarypath\create-2023.2.5\kit\python\lib\importlib\__init__.py(126): import_module
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/_registration/extension_management.py(154): extension_management_factory
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/_registration/register_python_ogn.py(70): register_python_ogn
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\carb\profiler\__init__.py(85): wrapper
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/registration.py(34): __init__
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/extension.py(120): on_after_ext_enabled
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_commands.py(29): do
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/undo/undo.py(80): execute
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/commands/command.py(463): execute
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/common.py(288): toggle_extension
2024-05-05 18:32:41  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_components.py(93): toggle
2024-05-05 18:32:41  [Error] [omni.ext.python] 
2024-05-05 18:32:43  [Error] [omni.kit.app._impl] [py stderr]: d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5
2024-05-05 18:32:43  [Error] [omni.kit.app._impl]   warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
2024-05-05 18:32:43  [Error] [omni.kit.app._impl] 
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer] Failed to import python module omni.audio2face.ui.player. Error: cannot import name '_ccallback_c' from 'scipy._lib' (d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/_lib/__init__.py). Traceback:
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer] Traceback (most recent call last):
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\omni\ext\_impl\custom_importer.py", line 76, in import_module
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     return importlib.import_module(name)
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:\nvidia\librarypath\create-2023.2.5\kit\python\lib\importlib\__init__.py", line 126, in import_module
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     return _bootstrap._gcd_import(name[level:], package, level)
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap_external>", line 883, in exec_module
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.ui.player-104.8.0+104.1.wx64/omni/audio2face/ui/player/__init__.py", line 1, in <module>
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     from .scripts.extension import *
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.ui.player-104.8.0+104.1.wx64/omni/audio2face/ui/player/scripts/extension.py", line 13, in <module>
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     from .ui import PlayerInstanceManagerWidget
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.ui.player-104.8.0+104.1.wx64/omni/audio2face/ui/player/scripts/ui.py", line 4, in <module>
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     import scipy.ndimage
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/__init__.py", line 153, in <module>
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     from scipy._lib._ccallback import LowLevelCallable
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]   File "d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/_lib/_ccallback.py", line 1, in <module>
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer]     from . import _ccallback_c
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer] ImportError: cannot import name '_ccallback_c' from 'scipy._lib' (d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/_lib/__init__.py)
2024-05-05 18:32:43  [Error] [omni.ext._impl.custom_importer] 
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin] Exception: Extension python module: 'omni.audio2face.ui.player' in 'd:\nvidia\librarypath\machinima-2022.3.0\extscache\omni.audio2face.ui.player-104.8.0+104.1.wx64' failed to load.
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin] 
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin] At:
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\omni\ext\_impl\_internal.py(213): startup
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\omni\ext\_impl\_internal.py(328): startup_extension
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   PythonExtension.cpp::startup()(2): <module>
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_commands.py(29): do
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/undo/undo.py(80): execute
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/commands/command.py(463): execute
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/common.py(288): toggle_extension
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_components.py(93): toggle
2024-05-05 18:32:43  [Error] [carb.scripting-python.plugin] 
2024-05-05 18:32:43  [Error] [omni.ext.plugin] [ext: omni.audio2face.ui.player-104.8.0] Failed to startup python extension.
2024-05-05 18:32:44  [Error] [omni.kit.app._impl] [py stderr]: d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5
2024-05-05 18:32:44  [Error] [omni.kit.app._impl]   warnings.warn(f"A NumPy version >={np_minversion} and <{np_maxversion}"
2024-05-05 18:32:44  [Error] [omni.kit.app._impl] 
2024-05-05 18:32:44  [Error] [omni.ext.python] ImportError: cannot import name '_ccallback_c' from 'scipy._lib' (d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/_lib/__init__.py)
2024-05-05 18:32:44  [Error] [omni.ext.python] 
2024-05-05 18:32:44  [Error] [omni.ext.python] At:
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/_lib/_ccallback.py(1): <module>
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/__init__.py(153): <module>
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(992): _find_and_load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.ui.player-104.8.0+104.1.wx64/omni/audio2face/ui/player/scripts/ui.py(4): <module>
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.ui.player-104.8.0+104.1.wx64/omni/audio2face/ui/player/scripts/extension.py(13): <module>
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.ui.player-104.8.0+104.1.wx64/omni/audio2face/ui/player/__init__.py(1): <module>
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(241): _call_with_frames_removed
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap_external>(883): exec_module
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(703): _load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1006): _find_and_load_unlocked
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1027): _find_and_load
2024-05-05 18:32:44  [Error] [omni.ext.python]   <frozen importlib._bootstrap>(1050): _gcd_import
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:\nvidia\librarypath\create-2023.2.5\kit\python\lib\importlib\__init__.py(126): import_module
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/_registration/extension_management.py(154): extension_management_factory
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/_registration/register_python_ogn.py(70): register_python_ogn
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:\nvidia/librarypath/create-2023.2.5/kit/kernel/py\carb\profiler\__init__.py(85): wrapper
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/registration.py(34): __init__
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.graph-1.106.1+ece658d9.wx64.r.cp310/omni/graph/core/_impl/extension.py(120): on_after_ext_enabled
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_commands.py(29): do
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/undo/undo.py(80): execute
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.commands-1.4.6+ece658d9.wx64.r.cp310/omni/kit/commands/command.py(463): execute
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/common.py(288): toggle_extension
2024-05-05 18:32:44  [Error] [omni.ext.python]   d:/nvidia/librarypath/create-2023.2.5/extscache/omni.kit.window.extensions-1.3.0+ece658d9/omni/kit/window/extensions/ext_components.py(93): toggle
2024-05-05 18:32:44  [Error] [omni.ext.python] 

[W1r3d]

Ok so not i’ve reinstalled

• Python 3.8
• Numpy 1.22.4
• SciPy 1.7.3

But i’m still getting this error in Composer

2024-05-06 15:42:08  [Error] [omni.kit.app._impl] [py stderr]: d:/nvidia/librarypath/machinima-2022.3.0/extscache/omni.audio2face.player_deps-104.7.0+104.1.wx64/deps/py/scipy/__init__.py:146: UserWarning: A NumPy version >=1.16.5 and <1.23.0 is required for this version of SciPy (detected version 1.23.5

So i’m guessing composer 2023.2.5 has a newer version of numpy as a dependency somewhere.

Ok ok ok, starting to wrap my head around this.
Is Livelink the way to stream A2F into any application?
I think i can see that working.

[W1r3d]

I’m trying to find the correct way to Proceed with this. It seems like if i solve this then i can Stream from A2F to something like composer. Right now there is only the AvatarStream app and that in itself doesn’t do any good other than showing that there is the possibility.

I get the Graph setup in A2F and i think i can replicate it in my target application. But i feel the Workflow for the character itself is missing.

• Q: How do i set up my full body character from CC4 to be compadible with this workflow?
  – I still want to be able to use my idle animation or set up a animation blueprint for the character. But have the face animated from A2F.
  – Is this what the Ferret tutorial or the Slacker tutorial goes through? (Looks like it)
• Q: How do i set up my character to be able to map it correctly to the Blenshape Solve node?
  – Q: Do i need to transfer the blendshapes onto my character mesh(like with Blender Rain)?
• Q: Which one of the following options do i use?
  

Super appriciate any insights on this as i’m trying to weed through the information :)
and thank you for your patience and hard work.
The workflow is becoming more clear for me for everyday i spend on this.

[W1r3d]

Option 1: No AnimGraph

Step 1. Load solved arkit sample in A2F

Step 2. Load the saved result file after running step 1 - step 5 above in Avatar Stream app.

Step 3. Setup the stream receive graph onto the Animation_clean skel Animation

This does not make sense to me.
I interpret these instructions like this.

• Start A2F
  – Load mark_solved_arkit

• Do step 1-5 on your character(in my case Kevin) in the Avatar Stream application

  This doesn’t make sense. Don’t i just import my Result into Avatar Stream App?
  Q: Should i have been doing the 1-5 steps in the Avatar Stream app all the way?

• Start Avatar Stream App
  – From the reference image i can see how it should be set up so that is pretty straight forward

What i did was

• Do step 1-5 in A2F
  – Save the scene as Result
• Open a new A2F Scene with mark_solved_arkit
• Open Avatar Stream App
  – Load my Result into Avatar Stream App
  – Find CC_Base_Body and from the menu Avatar > Add Blendshape Recieve Setup
• Do the RecieveLivelink setup
• Activate both instances of A2F and Avatar Stream App
• Press play in A2F
  (nothing happends in Avatar Stream App)

Solved
Ok this is working now. Still need to figure out how to get it into Composer or similar app. But yeah.

But wow what a difference there is in Quality between the A2F workflow and the ARkit one.

[W1r3d]

So i can connect between two separate computers on my network using Live link and ARkit.
One using A2F and the other Avatar Stream

I can verify this because of how the live link → burst mode behaves.
If there is no connection it says i need to connect
If i have a connection i can send a burst. Burst finishes Sending to localhost Frame rate 30 at 100%
I can connect but getting a WinError 10053 which i think means that there is a firewall interfering.

On the machine running Avatar Stream i’m getting this error

2024-05-08 20:58:59  [Warning] [omni.avatar.livelink.scripts.livelinkreceiver] Cannot decode data to JSON: b'WAVE:48000:1:32:3'

LoL i can see the mouth moving. So it works but i’m guessing there is something wrong with the connection / ports