CC4.41 to A2F 2023.2.0

With this thread i’m going to document each and every step that i’am taking when going through the setup for Audio 2 Face. Feel free to Comment or if you see obvious errors then point them out.

  • Result so far is the up until + CREATE A2F PIPELINE everything seems to be working.
  • After that point i get unpredictable results as described.
  • I sometimes also get different results if i reload the scene or re-open the scene
  • Short summary would be that the Mark AI Model moves up dislocating the teeth and eyes, the eyes of the CC4 Kevin Character moves around outside of the head, the eyes and lower dentures does not animate. The Tech Fix described in Nvidia Tutorials does not fix this.
  • Next step is to redo everything from + CREATEA A2F PIPELINE but choose YES, ATTACH instead of NO

Preparing the Mesh in CC4 and exporting

  • I am using Character Creator 4.41
  • I’m using the Free resource Model Kevin for Character Creator 4.41
  • I’m using the Male_Idle_01 animation(it is the one at the top of the list)
  • I’m converting all materials to PBR before export

Export USD (Omniverse) Options
Export: Character
Material > Render Mode: RTX Real-Time
Include motion: Yes(Checked)
Motion FPS: 60
Current Animation: Radio box checked
Include Omniverse Audio2FaceMesh: Yes(Checked)
Delete Hidden Faces: Yes(Checked)
Include Lights and IBL: No(Unchecked)


We will Call this the Original Workflow

Audio 2 Face Setup

  • I am using Audio 2 Face 2023.2.0
  • I have installed the Reallusion Character Creator 4.0.0 Connector

Preparing the Scene

  • In an empty scene i reference the Full Body USD file found in the root of your export folder by dragging it into my scene under World
  • In the same scene i reference the A2F USD file found in the A2F folder under your root folder in the root folder for your export

To better view my results i switch on the viewport option RTX- Real-Time > camera light

In the Character setup tab under CHARACTER SETUP
Full Face: Checked
Import Fitting Templates: Checked
Character Orientation: Forward= -x, Up= y
Skin: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Body/CC_Base_Body

FULL FACE MESHES
Gum / Teeth: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Teeth_Upper/CC_Base_Teeth_Upper
Tongue: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Tongue/CC_Base_Tongue
EYES:
LeftEye: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Eye_L/CC_Base_Eye_L
RightEye:/World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Eye_R/CC_Base_Eye_R
Pivot: Select from drop down menu > Compute From Selected Mesh
Then Select by pressing with the LMB the prim path in the LeftEye and RightEye input boxes

ADDITIONAL MESHES
Static:
From A2F reference
/World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Teeth_Upper/CC_Base_Teeth_Upper
From Full Body Reference
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Classic_short/Classic_short/Classic_short

Dynamic
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Eye/CC_Base_Eye/CC_Base_Eye
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Teeth/CC_Base_Teeth/CC_Base_Teeth
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Sideburns_Stubble/Sideburns_Stubble/Sideburns_Stubble
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Circle_Sparse/Circle_Sparse/Circle_Sparse
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Mustache_Sparse/Mustache_Sparse/Mustache_Sparse
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Soul_Patch_Sparse/Soul_Patch_Sparse/Soul_Patch_Sparse
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Tongue/CC_Base_Tongue/CC_Base_Tongue
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Body/CC_Base_Body/CC_Base_Body

Save File: Step1_CharacterSetup_Setup

Click SETUP CHARACTER
New items created

  • Xform: transfer_character
  • Scope: a2fTemplates
  • Scope: Looks
  • Omnigraph: PushGraph

Save File as: Step1_CharacterSetup_Result

In the Character Transfer Tab under SKIN MESHES
Driver A2F Mesh: /World/a2fTemplates/gen_xform_grp/gen
Target Mesh: /World/transfer_character/CC_Base_Body
OpenMouth A2F Mesh: /World/a2fTemplates/gen_openMouth

SKIN MESH FITTING
Correspondance visibility: Checked
Numbers visibility: Checked
Preset: Reallusion CC4
(Popup) Preset Load > Choose option Yes

POST WRAP
Fitted Mesh: /World/transfer_character/CC_Base_Body_fitted
Click Begin Post Wrap (after clicked the option should dissapear after completion and the Fitted Mesh drop down menu should now say just Select Mesh)

LOWER DENTURE
Target Jaw Xform: /World/transfer_character/lower_denture_grp
Click: JAW MOTION PREP
(Popup) Click OK

Click: TONGUE MESH FITTING
(Popup) Click OK

Click: TONGUE MESH POST WRAP
(Popup) Click OK

Save File: Step2_CharacterMeshFitting_Result

Switch to the Audio2Face Tool tab

In the Viewport or throught the Stage Tab
Select Mesh: /World/a2fTemplates/gen_xform_grp/gen

In the Audio2Face Tool Tab
Click: + A2F PIPELINE
(Popup) CREATE A2F PIPELINE
Select Audio Player type: Regular Audio Player
Select AI Model type: Mark
Auto-attach full face components: Yes(Checked)

Do you want to attach newly created A2F Core to this mesh?
Note: According to online tutorials from NVIDIA. Attaching the meshes will create transforms on all the meshes that will affect the following steps negatively. We will because of this choose the option: NO, JUST CREATE
Note: After clicking NO, JUST CREATE I think there is an error. The AI Model has not switched from Gen to Mark at this point.

Save File as: Step3_A2FPIPELINE_Result
Save File as: Step4_A2FTool_Setup

In the top menu click Animation > Deformer > Create Prox Deformer…
(Popup)PROX UI
Use Max Distance: Yes(Checked)
Max Distance: 0.5
Mode: translate and rotate
Rigid: No(Unchecked)

!! Note select first the driver and then what you want driven !!

Select driver: /World/transfer_character/eye_root/l_eye_grp/CC_Base_Eye_L
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.

Select driver: /World/transfer_character/eye_root/r_eye_grp/CC_Base_Eye_R
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.

Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Tongue_result
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Tongue
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Male_Brow_2

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Soul_Patch_Sparse

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Sideburns_Stubble

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Circle_Sparse

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Mustache_Sparse

In Popup PROX UI
Change setting Max distance to 0.0001
Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Teeth_Lower
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Teeth
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.

Note: When i’m going through this process i am getting this warning in the console: 2024-05-23 14:19:53 [Warning] [omni.deform.shared.scripts.utils] Could not find the output point attribute on Node(“/World/LazyGraph/CC_Base_Eye_WritePrimsV2”).

Save File: Step4_A2FTool_Setup
Save File as: Step5_ATTACHEDPRIMS_Setup

In the Audio2Face Tool Tab
Unfold the ATTACHED PRIMS section
Select /World/a2fTemplates/gen_xform_grp/gen in Stage or viewport
Press + on the right of Face Skin Mesh dropdown
Result = Face Skin Mesh: /World/a2fTemplates/gen_xform_grp/gen
Note: This seems to produce a error. It now switches to the Mark AI Model. But moves it up like 8 units. So the eyes and the dentures for the model is a missmatch now. My assumption is that this will affect other meshes like the tongue.

Select: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi/c_tongue_hi in Stage of Viewport
Press + to the right of the Tongue Mesh Dropdown
Result = Tongue Mesh: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi/c_tongue_hi
Note: This moves my tongue to the forehead of the CC4 Kevin Character. It is in the correct place for the Mark Character. The jaw is still missplaced as well as the eyes.

Note: There is some confusion in the next step. Because if i look at the automatic population of the list the Left Eye Xform should look like this: /World/transfer_character/eye_root/l_eye_grp, /World/a2fTemplates/gen_xform_grp/l_eye_grp_hi. But there is no way that i can accomplish that.
If i go back and choose the option Yes, Attach when creating the + A2F PIPELINE. Then the Left and right eye points to the /World/transfer_character/eye_root/l_eye_grp and r_eye_grp. So i will choose these as my groups.
Select: /World/transfer_character/eye_root/l_eye_grp in Stage or viewport
Press + to the right of the Left Eye Xform:
Result = Left Eye Xform: /World/transfer_character/eye_root/l_eye_grp

Select: /World/transfer_character/eye_root/r_eye_grp in Stage or viewport
Press + to the right of the Right Eye Xform:
Result = Right Eye Xform: /World/transfer_character/eye_root/r_eye_grp
Note: This moves the Left eye for mesh /World/transfer_character/extra/dynamic_grp/CC_Base_Eye about 4 units in the positive Z axis. Well outside of the CC4 Kevin Mesh

Note: Same thing here. The CREATE A2F PIPELINE points to two different groups. I have selected the one that
Select: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi
Press + to the right of the Right of Lower Denture Xform:
Result = Lower Denture Xform: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi


!! Summary of result up to this point !!

  • Scrubbing the AUDIO PLAYER track will animate the CC4 Kevin AND Mark AI Model
  • CC4 Character Tongue is in it’s forhead(Moving it down -8 z in and +2 in y will kind of fix that). I have choosen this as my solution for now and hidden the tongue_result
  • The eyes remain a mystery to me.
    • Sometimes they keep jumping around to the wrong places. I re-did this step a few times and i keep getting different results. Sometimes it is just the one eye that is moved. Sometimes it is both.
    • Currently the eyes are now in the right place. I have choosen to accept this before continuing
    • Currently the CC4 Kevin eyes are NOT moving. I think the AI Model Mark eyes are moving but they are inside his mouth o i cannot see.
  • Tongue is animated but in the wrong place for CC4 Kevin but it is in the right place for AI Model Mark
  • Lower denture is NOT animated on CC4 Kevin but it IS animated on AI Model Mark. The AI Model Mark dentures are not in the right place. So they are about -8 below the AI Model Mark Face

Save File as: Step6_TechFIX

Note: We will now try to fix the Lower denture and the eyes. In online tutorials from NVIDIA it is described that we need to edit the LazyGraph and connect the AUDIO PLAYER > Time output to the CC_Base_Teeth_Prox and the CC_Base_Eye_Prox we created.
In the Stage Tab Select LazyGraph
Right Click and choose: Open Graph from the menu
Note: The Generic Graph should now load the LazyGraph nodes

In the Stage Tab Select CC_Base_Teeth_Prox
In the Generic Graph. Locate the CC_Base_Teeth_Prox
Note: It will have a light blue highlight because it is selected

In the Generic Graph. Locate the AUDIO PLAYER
Drag the AUDIO PLAYER Node to the CC_Base_Teeth_Prox Node

Locate the import USD prim node connected to the CC_Base_Teeth Prox Node Mesh Bundle input
Connect the AUDIO PLAYER > Time(output) to the import USD prim data > Usd Timecode(input)
Note: According to online tutorials by NVIDIA this should not enable the lower dentures to update not just once but all the time. Test this by scrubbing the AUDIO PLAYER’s audio track
!! Stopping here as this did not work for me !!

1 Like

I am continuing this from Step 2.

  • I will choose YES, ATTACH instead of NO
  • I will do everything else in the same way

We will Call this the Alternative Workflow

Save File: Step2_CharacterMeshFitting_Result

Switch to the Audio2Face Tool tab

In the Viewport or throught the Stage Tab
Select Mesh: /World/a2fTemplates/gen_xform_grp/gen

In the Audio2Face Tool Tab
Click: + A2F PIPELINE
(Popup) CREATE A2F PIPELINE
Select Audio Player type: Regular Audio Player
Select AI Model type: Mark
Auto-attach full face components: Yes(Checked)

Do you want to attach newly created A2F Core to this mesh?
Note: Because we are exploring the alternative way we are going to select YES, ATTACH. According to online tutorials from Nvidia this will create transformations on the meshes and we will not be able to accurately use PROX UI to map the points onto our dynamic meshes. Which we need in order to drive the full body.
Note: When we do this we can note that the AI Model Mark has still been moved up a bit. But so has his teeth, eyes, tongue. We can also note that the AI Model has switched from Gen to Mark. Even though all prims are still pointing to a mesh or xform called gen.

In the Audio2Face Tool tab. If we srub the AUDIO PLAYER’s loaded audio track.
We can see that CC4 Kevin is being animated by the AI Model Mark.
We can confirm that
/World/transfer_character/lower_denture_grp
/World/transfer_character/lower_denture_grp/CC_Base_Tongue_result
/World/transfer_character/eye_root/l_eye_grp
/World/transfer_character/eye_root/r_eye_grp
is working as intended and is animated in the correct place.

Save File as: Step3_A2FPIPELINE_Result
Save File as: Step4_A2FTool_Setup

In the top menu click Animation > Deformer > Create Prox Deformer…
(Popup)PROX UI
Use Max Distance: Yes(Checked)
Max Distance: 0.5
Mode: translate and rotate
Rigid: No(Unchecked)

Note select first the driver and then what you want driven

Select driver: /World/transfer_character/eye_root/l_eye_grp/CC_Base_Eye_L
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: The left eye of /World/transfer_character/extra/dynamic_grp/CC_Base_Eye has now moved outside of CC4 Kevins Head

Select driver: /World/transfer_character/eye_root/r_eye_grp/CC_Base_Eye_R
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: The right eye of /World/transfer_character/extra/dynamic_grp/CC_Base_Eye has now moved outside of CC4 Kevins Head.
Note: I’m also getting this Warning: 2024-05-23 20:32:54 [Warning] [omni.deform.shared.scripts.utils] Could not find the output point attribute on Node(“/World/LazyGraph/CC_Base_Eye_WritePrimsV2”).

Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Tongue_result
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Tongue
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: The tongue for Kevin is now in his cheek. Also CC4 Kevin’s left eye moved back into place

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Male_Brow_2

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Soul_Patch_Sparse

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Sideburns_Stubble

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Circle_Sparse

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Mustache_Sparse

In Popup PROX UI
Change setting Max distance to 0.0001
Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Teeth_Lower
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Teeth
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: CC4 Kevin’s lower teeth are now intersecting his cheek and is in the same place as the tongue

Note: When i’m going through this process i am getting this warning in the console: 2024-05-23 14:19:53 [Warning] [omni.deform.shared.scripts.utils] Could not find the output point attribute on Node(“/World/LazyGraph/CC_Base_Eye_WritePrimsV2”). I’m getting this warning for all meshes.


!! Summary of result up to this point !!

  • Scrubbing the AUDIO PLAYER track will animate the CC4 Kevin AND Mark AI Model

  • The lower_denture group, tongue, eyes that are mapped to the AI Model all behave as expected.

  • The PROX UI meshes driven by /World/transfer_character/CC_Base_Body_result:
    /World/transfer_character/extra/dynamic_grp/CC_Base_Body
    /World/transfer_character/extra/dynamic_grp/Mustache_Sparse
    /World/transfer_character/extra/dynamic_grp/Circle_Sparse
    /World/transfer_character/extra/dynamic_grp/Sideburns_Stubble
    /World/transfer_character/extra/dynamic_grp/Male_Brow_2
    /World/transfer_character/extra/dynamic_grp/Soul_Patch_Sparse
    are all behaving as expected

  • !! CC4 Kevins meshes that will drive the full body Tongue, Lower teeth and right eye are all in the wrong place. These are the meshes that are driven by the lower_denture group, tongue_result as well as left and right eyes.

  • !! The eyes remain a mystery to me.

  • !! Applying the TechFIX to the eyes and teeth at this point does nothing to improve the situation. The eyes are not moving and the bottom teeth are rotating around like their pivot is offset.

!! There is no point in continuing beyond this point !!

I am continuing this from Step 2.

  • I will choose YES, ATTACH instead of NO
  • This will be a hybrid workflow. Where i combine the first way of working with this one. My assumption is that i can remove any transforms from the eyes etc by removing them from ATTACHED PRIMS after i have pressed YES, ATTACH.

We will call this the Hybrid Workflow

Save File: Step2_CharacterMeshFitting_Result

Switch to the Audio2Face Tool tab

In the Viewport or throught the Stage Tab
Select Mesh: /World/a2fTemplates/gen_xform_grp/gen

In the Audio2Face Tool Tab
Click: + A2F PIPELINE
(Popup) CREATE A2F PIPELINE
Select Audio Player type: Regular Audio Player
Select AI Model type: Mark
Auto-attach full face components: Yes(Checked)

Do you want to attach newly created A2F Core to this mesh?
Note: Because we are exploring the alternative way we are going to select YES, ATTACH. According to online tutorials from Nvidia this will create transformations on the meshes and we will not be able to accurately use PROX UI to map the points onto our dynamic meshes. Which we need in order to drive the full body.
Note: When we do this we can note that the AI Model Mark has still been moved up a bit. But so has his teeth, eyes, tongue. We can also note that the AI Model has switched from Gen to Mark. Even though all prims are still pointing to a mesh or xform called gen.

In the Audio2Face Tool tab. If we srub the AUDIO PLAYER’s loaded audio track.
We can see that CC4 Kevin is being animated by the AI Model Mark.
We can confirm that
/World/transfer_character/lower_denture_grp
/World/transfer_character/lower_denture_grp/CC_Base_Tongue_result
/World/transfer_character/eye_root/l_eye_grp
/World/transfer_character/eye_root/r_eye_grp
is working as intended and is animated in the correct place.

Save File as: Step3_A2FPIPELINE_Result
Save File as: Step4_A2FTool_Setup

In the top menu click Animation > Deformer > Create Prox Deformer…
(Popup)PROX UI
Use Max Distance: Yes(Checked)
Max Distance: 0.5
Mode: translate and rotate
Rigid: No(Unchecked)

Note select first the driver and then what you want driven

!! Before starting. remove all ATTACHED PRIMS. EXCEPT for the Face Skin Mesh !!
Note: When i did this the /World/transfer_character/lower_denture_grp/CC_Base_Tongue_result jumped about -8 units down for CC4 Kevin. Placing it near his adams apple

Select driver: /World/transfer_character/eye_root/l_eye_grp/CC_Base_Eye_L
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: Left Eye is in correct place

Select driver: /World/transfer_character/eye_root/r_eye_grp/CC_Base_Eye_R
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: Right Eye is in correct place

Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Tongue_result
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Tongue
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: The /World/transfer_character/lower_denture_grp/CC_Base_Tongue_result for CC4 Kevin is at his adams apple still. This just moved /World/transfer_character/extra/dynamic_grp/CC_Base_Tongue to the same place

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Male_Brow_2

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Soul_Patch_Sparse

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Sideburns_Stubble

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Circle_Sparse

Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Mustache_Sparse

In Popup PROX UI
Change setting Max distance to 0.0001
**Select driver: /**World/transfer_character/lower_denture_grp/CC_Base_Teeth_Lower
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Teeth
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: This produces the correct result

Note: I am still getting this warning in the console: 2024-05-23 14:19:53 [Warning] [omni.deform.shared.scripts.utils] Could not find the output point attribute on Node(“/World/LazyGraph/CC_Base_Eye_WritePrimsV2”). This goes for all meshes.

Note: We are now going to re-attach the prims and see what happends
In the viewport or Stage select /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi/c_tongue_hi
In Audio2Face Tool > ATTACHED PRIMS > Tongue Mesh. Press the + button to the right while having c_tongue_hi selected
Note: This moved the CC4 Kevin Tongue and Tongue Result to the right place

In the viewport or Stage select /World/transfer_character/eye_root/l_eye_grp
In Audio2Face Tool > ATTACHED PRIMS > Left Eye Xform. Press the + button to the right while having l_eye_grp selected

In the viewport or Stage select /World/transfer_character/eye_root/r_eye_grp
In Audio2Face Tool > ATTACHED PRIMS > Right Eye Xform. Press the + button to the right while having _eye_grp selected
Note: Doing this moved the right eye for CC_Base_Eye out of CC4 Kevins head about 8 units in z-axis(world)

In the viewport or Stage select /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi
In Audio2Face Tool > ATTACHED PRIMS > Lower Denture Xform. Press the + button to the right while having c_bottomDenture_grp_hi selected

Note: We can notice while scrubbing that some meshes are not moving at all at this point. Like the lower dentures for CC4 Kevin. the CC_Base_Eye mesh is also not moving. We will try to fix this with the Tech fix described before.

In the Stage tab find and select LazyGraph
Right click on LazyGraph and choose Open Graph from the menu
This will open the Generic Graph tab and show you the nodes for LazyGraph
In the stage tab expand the LazyGraph and find and select CC_Base_Teeth_Prox

With CC_Base_Teeth_Prox selected. In the generic graph locate the node. It will be highlighted using a light blue rim.
In the Generic Graph tab find and drag the AUDIO PLAYER node to the same location as the CC_Base_Teeth_Prox

To the Mesh Bundle Input for CC_Base_Teeth_Prox there is a import USD prim data node connected
Connect the AUDIO PLAYER > Time(output) to the import USD prim data > Usd Timecode(input)

With CC_Base_Eye_Prox selected. In the generic graph locate the node. It will be highlighted using a light blue rim.
In the Generic Graph tab find and drag the AUDIO PLAYER node to the same location as the CC_Base_Eye_Prox

To the Mesh Bundle Input for CC_Base_Eye_Prox there is a import USD prim data node connected
Connect the AUDIO PLAYER > Time(output) to the import USD prim data > Usd Timecode(input)

In the Audio2Face Tool > AUDIO PLAYER scrub the audio track and see if the CC_Base_Eye and CC_Base_Teeth are moving
Note: For me they are NOT MOVING


!! Summary of result up to this point !!

  • Scrubbing the AUDIO PLAYER track will animate some parts of CC4 Kevin AND all parts of Mark AI Model

  • For CC4 Kevin the left and right eye in eye_root are moving but NOT the CC_Base_Eye

  • For CC4 Kevin NO part of the CC_Base_Teeth or the /World/transfer_character/lower_denture_grp moves

  • For CC4 Kevin the tongue result and tongue IS moving

  • The PROX UI meshes driven by /World/transfer_character/CC_Base_Body_result:
    /World/transfer_character/extra/dynamic_grp/CC_Base_Body
    /World/transfer_character/extra/dynamic_grp/Mustache_Sparse
    /World/transfer_character/extra/dynamic_grp/Circle_Sparse
    /World/transfer_character/extra/dynamic_grp/Sideburns_Stubble
    /World/transfer_character/extra/dynamic_grp/Male_Brow_2
    /World/transfer_character/extra/dynamic_grp/Soul_Patch_Sparse
    are all behaving as expected. No change here

  • !! The Righe eye for CC_Base_Eye is still outside of CC4 Kevins Head

  • !! Applying the TechFIX to the eyes and teeth at this point does nothing to improve the situation. Neither the eyes or the bottom teeth are moving

!! There is no point in continuing beyond this point !!

Notes this far

  • Tutorials like the CC4 Camilla are great but highlight that there are a lot of customization that is needed for the Eyelashes and Eyes in particular.
    – I have not yet done this in any om my setups. I can see that there are some errors with the eyelashes that i have and that this would fix. But i also see these meshes in the CC4 Character and not really sure why we need to do the fixes. Needs more exploration on my part.
    – The tutorial also refers to the A2F mesh group for the tongue, teeth etc as the Cavity. This is not consistent with the CC4 export
    – I’m not sure
  • This Warning troubles me: 2024-05-23 14:19:53 [Warning] [omni.deform.shared.scripts.utils] Could not find the output point attribute on Node(“/World/LazyGraph/CC_Base_Eye_WritePrimsV2”)
  • I feel like the Hybrid Workflow is the one that is working the best but it’s still not working in the end.
  • I realize that it is still in Beta but is a bit bothersome in comparison to how easy the ARKit workflow is.
    – I also feel like i might be able to troubleshoot the ARKit workflow much easier.

This part of the Setup gets me a bit confused
When clicking on the Gen AI Models head and then + A2F PIPELINE
i get this popup:

Worth noting is the meshes it wants to attach and the paths. Versus when it is completed.
I think i can figure out what is happening but not entierly sure.

I have some Questions

  • Is the A2F pipeline exclusively for pre-rendered or is this meant for rea-time as well?
  • How can we improve on the ARKit blendshapes and increase the fidelity of the facial animations?
  • Is Live Link only meant to stream Locally or can it be adapted to stream over the webb?
  • How can i train my very own AI Model for A2F to replace Mark, Claire and Gen?
  • A2F Headless still require a AI Model Head, A transfer Head and then if i want a full body a full body head to exist in the same scene?