Audio2Face 2023.1 comes packed with lots of new features, functionality, and workflow tutorials. We’ve expanded on our AI models further improving A2F multi-language support with the addition of a new Asian female AI model called Claire - providing users with more options for generating realistic facial animations.
In addition, 2023.1 comes with Livelink and Avatar Streaming, providing the ability to stream blendshape animation from Audio2Face to other Applications like “Unreal Engines, Meta Human”. With blendshape support for tongues on both AI Models. Also, the Streaming audio player now engages Audio2Emotion providing developers and users the full performance capabilities of Audio2Face. In addition, we have improved the Rest API with more comprehensive controls. Plus, the ability to export “emotion keyframes” from Audio2Emotion for use in other applications.
New AI model Claire
Meet Claire, our new multi-language Asian Female AI model.
Introducing the new Ai Models Panel
New to this release is the AI models panel - providing more choice and a simplified workflow for getting started within the application. Choose which Ai Model you would like to use with which Audio Player. You can also quickly change between available trained Networks for each respective AI model where multiple networks are present.
Ai Model Panel Tutorial.
Live Link and Avatar Streaming
A2F can now output a Live stream of blendshape animation data to connect to external applications that drive a characters facial animation performance. See the tutorial here for setting up the Streaming Workflow.
Avatar Stream Tutorial
BlendShape Streaming to Metahuman – Part 1
BlendShape Streaming to Metahuman – Part 2
Improved Rest API
- Additional routes to Query/Set the Live Streaming of A2F
- Additional routes to export the Emotion keys as JSON file
- Additional Audio2Emotion routes
- Additional routes for the Regular Player
- Updated routes for the Regular Player
Improved BlendShapes with added Tongue support
Performance improvements to Blendshape Solvers and We’ve added a tongue blendshape solution for both Mark and Claire models. You can find these assets in the A2F Samples Tab.
Export Emotion keyframes
Provided in the export tab is the ability to export emotion keyframes as .json
A2F 2023.1 Sample files
All Sample files are available in the examples Tab within the A2F application or via the content browser in your localhost at NVIDIA/Assets/Audio2Face/Samples_2023.1/
Updated Tutorial for Setting up Custom Character.
Arkit Blendshape – part 1 - Ferret Model Overview.
Arkit Blendshape – part 2 - Ferret Character Setup.
OM-96646 - AI model Template UI.
OM- 94489 - Separate Jaw Transform retarget out of main A2F node into its own omnigraph node.
OM-94221 - Add Mandarin Audio Samples to A2F.
OM-93156 - Claire Template for Blendshape Solve.
OM- 92260 - [A2F] Add NumPy dependency to mesh separate.
OM-92178 - [A2F] Send and receive BlendShape through burst mode.
OM-91784 - [A2F] REST API - Expose new emotion key export.
OM-89282 - Expose Blendshape transfer from custom source to charTransfer UI.
OM-88437 - Enable Samples browser in A2F.
OM-86846 - [A2F] Add USDSKel Mesh BindTransform fix python script to toolbox.
OM-86868 - [A2F] add support of receiving streaming audio into AvatarStream or Kit.
OM-86264 - [A2F] Support Burst mode protocol on Receive Livelink node or related workflow.
OM-79096 - Integrate Training for Claires tongue into A2F.
OM-77176 - [A2F] Add livestream option for blendshapes.
OM-67428 -Support for Long audio files. Improved handling when file is too long.
OM-61494 - Streaming Audio2Emotion.
OM-64642 - Ability to Export emotion keyframes from Audio2Face.
OM-63926 - Integrate Mandarin data training and test into product.
OM-97202 - Delete setup and delete correspondence makes error in fullface generic template.
OM-94931 - [A2E] RestAPI errors with streaming player - GetEmotions.
OM-94911 - Turning on/off A2E with streaming player breaks the keyframe functions.
OM-94286 - [A2F] load preset on Float Array Tuner error - no such file or directory.
OM-94051 - Fix jaw open poses that have issues.
OM-92740 - Jaw Broken on target character when switching A2F model.
OM-91680 - Cuda memory access error when working with keyframes from the old example files.
OM-90670 - Target mesh deformation breaks after previewing Blendshape Transfer.
OM-87886 - Old scene file with keyframe cannot be loaded properly (keyframe lost).
OM-84827 - [A2F] Default scene not loading after creating key in post processing.
OM-84055 - A2F Start up Freeze. The demo scene gets stuck at 90% after the jaw/tongue verification. It doesn’t lead to a crash.
OM-83997 - A2E Better handling of model update in older scenes.
OM-83825 - [A2F] Audio2Emotion with 105-SDK crashes in both Regular and Streaming Player.
OM-97222 - Exporting USD Cache animation with Batch enabled stops at saving time on the first export.
OM-79210 - User prompt of Auto Generate On Track Change keeps showing up on every scene save.
OM-78655 - Unable to export geometry cache for a charTransferred tongue.
OM-75641 - Some menu item buttons are behaving as toggles (menu → Audio2Face).
OM-59989 - Blendshape solve toggle use button is not working correctly.
OM-56982 - Auto Generate on Track ON stops the scene from saving.
OM-56276 - A2E .TRT build taking very long time.
OM-96966 - Remove “reset Layout” on demo Scene when opened.
OM-95980 - Disable Shadow Denoiser in A2F.
OM-93726 - Remove use of GetPoints if they can be bypassed.
OM-93045 - Improve demo Scene Lighting.
OM-93054 - Pickup lighting changes for Demo Scene.
OM-87683 - Update to kit 105.
OM-86273 - Replace mark teeth/gum with the new mesh.
OM-85539 - Improve ARkit To Metahuman mapping.
OM-78699 - Update A2E network to remove neutral slider.