Audio2Face 2023.1 (Open Beta) Released

Omniverse Apps Audio2Face
1

Audio2Face 2023.1 Release Highlights

Audio2Face 2023.1 comes packed with lots of new features, functionality, and workflow tutorials. We’ve expanded on our AI models further improving A2F multi-language support with the addition of a new Asian female AI model called Claire - providing users with more options for generating realistic facial animations.

A2F_2023_1_ReleaseImage_04
A2F_2023_1_ReleaseImage_041920×1080 93 KB

In addition, 2023.1 comes with Livelink and Avatar Streaming, providing the ability to stream blendshape animation from Audio2Face to other Applications like “Unreal Engines, Meta Human”. With blendshape support for tongues on both AI Models. Also, the Streaming audio player now engages Audio2Emotion providing developers and users the full performance capabilities of Audio2Face. In addition, we have improved the Rest API with more comprehensive controls. Plus, the ability to export “emotion keyframes” from Audio2Emotion for use in other applications.

New AI model Claire

Meet Claire, our new multi-language Asian Female AI model.

Introducing the new Ai Models Panel

New to this release is the AI models panel - providing more choice and a simplified workflow for getting started within the application. Choose which Ai Model you would like to use with which Audio Player. You can also quickly change between available trained Networks for each respective AI model where multiple networks are present.

Ai Model Panel Tutorial.

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1745

Live Link and Avatar Streaming

A2F can now output a Live stream of blendshape animation data to connect to external applications that drive a characters facial animation performance. See the tutorial here for setting up the Streaming Workflow.

Avatar Stream Tutorial

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1744/

BlendShape Streaming to Metahuman – Part 1

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1747

BlendShape Streaming to Metahuman – Part 2

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1748

Improved Rest API

  • Additional routes to Query/Set the Live Streaming of A2F
  • Additional routes to export the Emotion keys as JSON file
  • Additional Audio2Emotion routes
  • Additional routes for the Regular Player
  • Updated routes for the Regular Player

Improved BlendShapes with added Tongue support

Performance improvements to Blendshape Solvers and We’ve added a tongue blendshape solution for both Mark and Claire models. You can find these assets in the A2F Samples Tab.

Export Emotion keyframes

Provided in the export tab is the ability to export emotion keyframes as .json

A2F 2023.1 Sample files

All Sample files are available in the examples Tab within the A2F application or via the content browser in your localhost at NVIDIA/Assets/Audio2Face/Samples_2023.1/

Additional Tutorials.

Updated Tutorial for Setting up Custom Character.

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1746

Arkit Blendshape – part 1 - Ferret Model Overview.

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1749

Arkit Blendshape – part 2 - Ferret Character Setup.

https://www.nvidia.com/en-us/on-demand/session/omniverse2020-om1750/

Added

OM-96646 - AI model Template UI.

OM- 94489 - Separate Jaw Transform retarget out of main A2F node into its own omnigraph node.

OM-94221 - Add Mandarin Audio Samples to A2F.

OM-93156 - Claire Template for Blendshape Solve.

OM- 92260 - [A2F] Add NumPy dependency to mesh separate.

OM-92178 - [A2F] Send and receive BlendShape through burst mode.

OM-91784 - [A2F] REST API - Expose new emotion key export.

OM-89282 - Expose Blendshape transfer from custom source to charTransfer UI.

OM-88437 - Enable Samples browser in A2F.

OM-86846 - [A2F] Add USDSKel Mesh BindTransform fix python script to toolbox.

OM-86868 - [A2F] add support of receiving streaming audio into AvatarStream or Kit.

OM-86264 - [A2F] Support Burst mode protocol on Receive Livelink node or related workflow.

OM-79096 - Integrate Training for Claires tongue into A2F.

OM-77176 - [A2F] Add livestream option for blendshapes.

OM-67428 -Support for Long audio files. Improved handling when file is too long.

OM-61494 - Streaming Audio2Emotion.

OM-64642 - Ability to Export emotion keyframes from Audio2Face.

OM-63926 - Integrate Mandarin data training and test into product.

Fixed

OM-97202 - Delete setup and delete correspondence makes error in fullface generic template.

OM-94931 - [A2E] RestAPI errors with streaming player - GetEmotions.

OM-94911 - Turning on/off A2E with streaming player breaks the keyframe functions.

OM-94286 - [A2F] load preset on Float Array Tuner error - no such file or directory.

OM-94051 - Fix jaw open poses that have issues.

OM-92740 - Jaw Broken on target character when switching A2F model.

OM-91680 - Cuda memory access error when working with keyframes from the old example files.

OM-90670 - Target mesh deformation breaks after previewing Blendshape Transfer.

OM-87886 - Old scene file with keyframe cannot be loaded properly (keyframe lost).

OM-84827 - [A2F] Default scene not loading after creating key in post processing.

OM-84055 - A2F Start up Freeze. The demo scene gets stuck at 90% after the jaw/tongue verification. It doesn’t lead to a crash.

OM-83997 - A2E Better handling of model update in older scenes.

OM-83825 - [A2F] Audio2Emotion with 105-SDK crashes in both Regular and Streaming Player.

OM-97222 - Exporting USD Cache animation with Batch enabled stops at saving time on the first export.

OM-79210 - User prompt of Auto Generate On Track Change keeps showing up on every scene save.

OM-78655 - Unable to export geometry cache for a charTransferred tongue.

OM-75641 - Some menu item buttons are behaving as toggles (menu → Audio2Face).

OM-59989 - Blendshape solve toggle use button is not working correctly.

OM-56982 - Auto Generate on Track ON stops the scene from saving.

OM-56276 - A2E .TRT build taking very long time.

Improved

OM-96966 - Remove “reset Layout” on demo Scene when opened.

OM-95980 - Disable Shadow Denoiser in A2F.

OM-93726 - Remove use of GetPoints if they can be bypassed.

OM-93045 - Improve demo Scene Lighting.

OM-93054 - Pickup lighting changes for Demo Scene.

OM-87683 - Update to kit 105.

OM-86273 - Replace mark teeth/gum with the new mesh.

OM-85539 - Improve ARkit To Metahuman mapping.

OM-78699 - Update A2E network to remove neutral slider.

1 Like
2
3

Thanks for this realese more than hoped for, solve many problems, cartoon faces and animals to ue fantastics.
just downloaded, christmas in july. thanks.

1 Like
4

Where can we locate the unreal audio2face plugin?

1 Like
5

Can you find it? I have the same problem.

6

audio2face plugin?
About Epic Games Unreal Engine 5.2 Omniverse Connector is from 12 july maybe its not in there maybe needs uppdate , its in the videos i looked at first. also looking for the ferret examples? it relased on a friday so maby some will adjust this when it opens on monday.

7

The current availability of the Unreal plugin is limited to ACE users. However, we are collaborating with other teams to develop a beta version of the plugin that will be accessible to the public via the Launcher.

The Ferret model is scheduled to be published shortly after Siggraph, approximately within a month. However, it’s important to note that this is just one of the available content options, and for testing purposes, users are free to utilize any model they prefer.

We will keep the forum updated with any notifications regarding these.

8

我的系统是centos stream8，用root权限能运行omniverse-launcher-linux.AppImage，但是打不开audio2face, 点启动之后没反应。
My system is CentOS Stream 8, and I can run omniverse-launcher-linux.AppImage with root privileges, but I cannot open Audio2Face. There’s no response after I click ‘start’.

9

Can you please send your latest Audio2Face log file which is located inside C:\Users\<USERNAME>\.nvidia-omniverse\logs\Kit\Audio2Face on Windows and ~/.nvidia-omniverse/logs/Kit/Audio2Face on Linux?

10

logs.zip (548.2 KB)
[root@centos8Stream logs]# ls
Cache cache-setup ‘cache-setup\cache-setup.log’ ‘Discovery Service’ launcher.log omni.processlifetime.log ‘System Monitor’
[root@centos8Stream logs]# pwd
/root/.nvidia-omniverse/logs

I can’t find the Audio2Face

11

Unfortunately Audio2Face doesn’t support CentOS.
Here’s the requirements page: Audio2Face Overview — audio2face latest documentation (nvidia.com)

12

Hi there, I am trying to use the REST API to export the ARKIT blendshapes of given track. I am following the tutorial given by official: Rest API — audio2face latest documentation. Basically I did the following steps: Load USD → SetRootPath → setTrack → GetBlendShapeSolvers → ExportBlendshapes. Everything is fine and getting 200 response except last step. It throws 500 internal error and looks very confusing to me. I am using the USD in the local server: omniverse://localhost/NVIDIA/Assets/Audio2Face/Samples_2023.1/blendshape_solve/mark_solved_arkit.usd. I am using this USD because in GUI I can simply export the bs json file by clicking one export button. Here is the log, and in console of A2F headless server it also keeps saying cannot find a prim node on a certain model.
kit_20230824_150519.log (316.9 KB)

13

Hi @wanzhang Could you kindly create a new thread. That way more people will see your question and you have a higher chance of receiving a relevant answer. Thanks

14

yes, I created one : A2F REST API ExportBlendshapes does not work