With this thread i’m going to document each and every step that i’am taking when going through the setup for Audio 2 Face. Feel free to Comment or if you see obvious errors then point them out.
- Result so far is the up until + CREATE A2F PIPELINE everything seems to be working.
- After that point i get unpredictable results as described.
- I sometimes also get different results if i reload the scene or re-open the scene
- Short summary would be that the Mark AI Model moves up dislocating the teeth and eyes, the eyes of the CC4 Kevin Character moves around outside of the head, the eyes and lower dentures does not animate. The Tech Fix described in Nvidia Tutorials does not fix this.
- Next step is to redo everything from + CREATEA A2F PIPELINE but choose YES, ATTACH instead of NO
Preparing the Mesh in CC4 and exporting
- I am using Character Creator 4.41
- I’m using the Free resource Model Kevin for Character Creator 4.41
- I’m using the Male_Idle_01 animation(it is the one at the top of the list)
- I’m converting all materials to PBR before export
Export USD (Omniverse) Options
Export: Character
Material > Render Mode: RTX Real-Time
Include motion: Yes(Checked)
Motion FPS: 60
Current Animation: Radio box checked
Include Omniverse Audio2FaceMesh: Yes(Checked)
Delete Hidden Faces: Yes(Checked)
Include Lights and IBL: No(Unchecked)
We will Call this the Original Workflow
Audio 2 Face Setup
- I am using Audio 2 Face 2023.2.0
- I have installed the Reallusion Character Creator 4.0.0 Connector
Preparing the Scene
- In an empty scene i reference the Full Body USD file found in the root of your export folder by dragging it into my scene under World
- In the same scene i reference the A2F USD file found in the A2F folder under your root folder in the root folder for your export
To better view my results i switch on the viewport option RTX- Real-Time > camera light
In the Character setup tab under CHARACTER SETUP
Full Face: Checked
Import Fitting Templates: Checked
Character Orientation: Forward= -x, Up= y
Skin: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Body/CC_Base_Body
FULL FACE MESHES
Gum / Teeth: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Teeth_Upper/CC_Base_Teeth_Upper
Tongue: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Tongue/CC_Base_Tongue
EYES:
LeftEye: /World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Eye_L/CC_Base_Eye_L
RightEye:/World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Eye_R/CC_Base_Eye_R
Pivot: Select from drop down menu > Compute From Selected Mesh
Then Select by pressing with the LMB the prim path in the LeftEye and RightEye input boxes
ADDITIONAL MESHES
Static:
From A2F reference
/World/Kevin_A2F/CharRoot/Kevin_A2F/Kevin_A2F/CC_Base_Teeth_Upper/CC_Base_Teeth_Upper
From Full Body Reference
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Classic_short/Classic_short/Classic_short
Dynamic
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Eye/CC_Base_Eye/CC_Base_Eye
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Teeth/CC_Base_Teeth/CC_Base_Teeth
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Sideburns_Stubble/Sideburns_Stubble/Sideburns_Stubble
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Circle_Sparse/Circle_Sparse/Circle_Sparse
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Mustache_Sparse/Mustache_Sparse/Mustache_Sparse
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/Soul_Patch_Sparse/Soul_Patch_Sparse/Soul_Patch_Sparse
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Tongue/CC_Base_Tongue/CC_Base_Tongue
/World/CC4_Kevin_PBR/CharRoot/Kevin/Kevin/CC_Base_Body/CC_Base_Body/CC_Base_Body
Save File: Step1_CharacterSetup_Setup
Click SETUP CHARACTER
New items created
- Xform: transfer_character
- Scope: a2fTemplates
- Scope: Looks
- Omnigraph: PushGraph
Save File as: Step1_CharacterSetup_Result
In the Character Transfer Tab under SKIN MESHES
Driver A2F Mesh: /World/a2fTemplates/gen_xform_grp/gen
Target Mesh: /World/transfer_character/CC_Base_Body
OpenMouth A2F Mesh: /World/a2fTemplates/gen_openMouth
SKIN MESH FITTING
Correspondance visibility: Checked
Numbers visibility: Checked
Preset: Reallusion CC4
(Popup) Preset Load > Choose option Yes
POST WRAP
Fitted Mesh: /World/transfer_character/CC_Base_Body_fitted
Click Begin Post Wrap (after clicked the option should dissapear after completion and the Fitted Mesh drop down menu should now say just Select Mesh)
LOWER DENTURE
Target Jaw Xform: /World/transfer_character/lower_denture_grp
Click: JAW MOTION PREP
(Popup) Click OK
Click: TONGUE MESH FITTING
(Popup) Click OK
Click: TONGUE MESH POST WRAP
(Popup) Click OK
Save File: Step2_CharacterMeshFitting_Result
Switch to the Audio2Face Tool tab
In the Viewport or throught the Stage Tab
Select Mesh: /World/a2fTemplates/gen_xform_grp/gen
In the Audio2Face Tool Tab
Click: + A2F PIPELINE
(Popup) CREATE A2F PIPELINE
Select Audio Player type: Regular Audio Player
Select AI Model type: Mark
Auto-attach full face components: Yes(Checked)
Do you want to attach newly created A2F Core to this mesh?
Note: According to online tutorials from NVIDIA. Attaching the meshes will create transforms on all the meshes that will affect the following steps negatively. We will because of this choose the option: NO, JUST CREATE
Note: After clicking NO, JUST CREATE I think there is an error. The AI Model has not switched from Gen to Mark at this point.
Save File as: Step3_A2FPIPELINE_Result
Save File as: Step4_A2FTool_Setup
In the top menu click Animation > Deformer > Create Prox Deformer…
(Popup)PROX UI
Use Max Distance: Yes(Checked)
Max Distance: 0.5
Mode: translate and rotate
Rigid: No(Unchecked)
!! Note select first the driver and then what you want driven !!
Select driver: /World/transfer_character/eye_root/l_eye_grp/CC_Base_Eye_L
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Select driver: /World/transfer_character/eye_root/r_eye_grp/CC_Base_Eye_R
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Eye
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Tongue_result
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Tongue
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Male_Brow_2
Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Soul_Patch_Sparse
Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Sideburns_Stubble
Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Circle_Sparse
Select driver: /World/transfer_character/CC_Base_Body_result
Select driven: /World/transfer_character/extra/dynamic_grp/Mustache_Sparse
In Popup PROX UI
Change setting Max distance to 0.0001
Select driver: /World/transfer_character/lower_denture_grp/CC_Base_Teeth_Lower
Select driven: /World/transfer_character/extra/dynamic_grp/CC_Base_Teeth
In PROX UI Popup click: Apply
Yellow message should appear: Success!! Prox deformer applied.
Note: When i’m going through this process i am getting this warning in the console: 2024-05-23 14:19:53 [Warning] [omni.deform.shared.scripts.utils] Could not find the output point attribute on Node(“/World/LazyGraph/CC_Base_Eye_WritePrimsV2”).
Save File: Step4_A2FTool_Setup
Save File as: Step5_ATTACHEDPRIMS_Setup
In the Audio2Face Tool Tab
Unfold the ATTACHED PRIMS section
Select /World/a2fTemplates/gen_xform_grp/gen in Stage or viewport
Press + on the right of Face Skin Mesh dropdown
Result = Face Skin Mesh: /World/a2fTemplates/gen_xform_grp/gen
Note: This seems to produce a error. It now switches to the Mark AI Model. But moves it up like 8 units. So the eyes and the dentures for the model is a missmatch now. My assumption is that this will affect other meshes like the tongue.
Select: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi/c_tongue_hi in Stage of Viewport
Press + to the right of the Tongue Mesh Dropdown
Result = Tongue Mesh: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi/c_tongue_hi
Note: This moves my tongue to the forehead of the CC4 Kevin Character. It is in the correct place for the Mark Character. The jaw is still missplaced as well as the eyes.
Note: There is some confusion in the next step. Because if i look at the automatic population of the list the Left Eye Xform should look like this: /World/transfer_character/eye_root/l_eye_grp, /World/a2fTemplates/gen_xform_grp/l_eye_grp_hi. But there is no way that i can accomplish that.
If i go back and choose the option Yes, Attach when creating the + A2F PIPELINE. Then the Left and right eye points to the /World/transfer_character/eye_root/l_eye_grp and r_eye_grp. So i will choose these as my groups.
Select: /World/transfer_character/eye_root/l_eye_grp in Stage or viewport
Press + to the right of the Left Eye Xform:
Result = Left Eye Xform: /World/transfer_character/eye_root/l_eye_grp
Select: /World/transfer_character/eye_root/r_eye_grp in Stage or viewport
Press + to the right of the Right Eye Xform:
Result = Right Eye Xform: /World/transfer_character/eye_root/r_eye_grp
Note: This moves the Left eye for mesh /World/transfer_character/extra/dynamic_grp/CC_Base_Eye about 4 units in the positive Z axis. Well outside of the CC4 Kevin Mesh
Note: Same thing here. The CREATE A2F PIPELINE points to two different groups. I have selected the one that
Select: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi
Press + to the right of the Right of Lower Denture Xform:
Result = Lower Denture Xform: /World/a2fTemplates/gen_xform_grp/c_bottomDenture_grp_hi
!! Summary of result up to this point !!
- Scrubbing the AUDIO PLAYER track will animate the CC4 Kevin AND Mark AI Model
- CC4 Character Tongue is in it’s forhead(Moving it down -8 z in and +2 in y will kind of fix that). I have choosen this as my solution for now and hidden the tongue_result
- The eyes remain a mystery to me.
- Sometimes they keep jumping around to the wrong places. I re-did this step a few times and i keep getting different results. Sometimes it is just the one eye that is moved. Sometimes it is both.
- Currently the eyes are now in the right place. I have choosen to accept this before continuing
- Currently the CC4 Kevin eyes are NOT moving. I think the AI Model Mark eyes are moving but they are inside his mouth o i cannot see.
- Tongue is animated but in the wrong place for CC4 Kevin but it is in the right place for AI Model Mark
- Lower denture is NOT animated on CC4 Kevin but it IS animated on AI Model Mark. The AI Model Mark dentures are not in the right place. So they are about -8 below the AI Model Mark Face
Save File as: Step6_TechFIX
Note: We will now try to fix the Lower denture and the eyes. In online tutorials from NVIDIA it is described that we need to edit the LazyGraph and connect the AUDIO PLAYER > Time output to the CC_Base_Teeth_Prox and the CC_Base_Eye_Prox we created.
In the Stage Tab Select LazyGraph
Right Click and choose: Open Graph from the menu
Note: The Generic Graph should now load the LazyGraph nodes
In the Stage Tab Select CC_Base_Teeth_Prox
In the Generic Graph. Locate the CC_Base_Teeth_Prox
Note: It will have a light blue highlight because it is selected
In the Generic Graph. Locate the AUDIO PLAYER
Drag the AUDIO PLAYER Node to the CC_Base_Teeth_Prox Node
Locate the import USD prim node connected to the CC_Base_Teeth Prox Node Mesh Bundle input
Connect the AUDIO PLAYER > Time(output) to the import USD prim data > Usd Timecode(input)
Note: According to online tutorials by NVIDIA this should not enable the lower dentures to update not just once but all the time. Test this by scrubbing the AUDIO PLAYER’s audio track
!! Stopping here as this did not work for me !!