Hello,
I am new to animation in general including blender. But I wanted to make a sort of live animation project using the rain v2 blender model so that she can say what I am saying in near real time.
At first I was looking for a way to export the audio2face animation as a video file of some sort (like an mp4) that I could play immediately afterward. As far as I can tell this is not possible to do so quickly as the general approach is to export the blendshapes of the animation to use elsewhere like blender or machinima, where it can be exported as a video there.
I eventually found out about the audio player streaming option in audio2face and got that working even better than the original option with my audio streaming directly into the program. But the issue I am now facing is none of the rain v2 textures/colors are being imported into audio2face. I am using the blender 3.6 alpha usd branch inside of omniverse and the audio2face usd exporter, but when importing to audio2face all textures are lost, leaving me with a blank model.
I recognize this makes sense for the regular approach which is exporting blendshapes elsewhere, but is there any way to import textures into audio2face that can be used on my rain model as she copies what I am saying in real time? I have included an image of my project below to show you what I mean.
I understand this is not the regular use-case and have seen most tutorials online also have blank models, but some tutorials using character creator characters from reallusion seem to include color so it seems possible to have color inside the audio2face app. I have also found that you can assign materials inside of audio2face. But I cannot figure out how to get the rain v2 colors out of blender into audio2face to assign them to my model.
Any guidance would be greatly appreciated!
The materials might be already in your scene, but their texture paths might be wrong.
Take a look under “Looks” folder in stage view and see if materials are imported.
You should be able to update the texture paths manually in your scene file.
Do you mean in here?
I only have the grey and green omni materials there. Might there be a way to import materials to audio2face separately? I am not sure if they are regularly included in usda files (i think this is the blender project file?), but as shown in this video:
at 27:58, the regular usd file seems suboptimal compared to the usda file. Could using the usda file be what is causing this issue? And if so might there be some way to manually export materials from blender/import them to audio2face?
It seems that if I right click on the looks folder in stage, and go to create > material and look under usd materials there is an “add mdl file” option. Is there some way that could help out here?
Here is a screenshot of my blender export settings if it helps in any way:
I have been stuck on this for quite a while so thank you for taking the time to reply!
I just exported Rain rig from Blender as usd and it exported all materials and textures properly.
Here are the steps I took:
- Install
Prod Blender 3.6 Alpha USD Branch from Launcher
- Launch it and open Rain rig
- Use Omniverse menu to export Rain, make sure to specify the Texture Directory in Project section
- Open the exported file inside Audio2Face. All textures should already be applied. I had to hide the eye glass layer as glass material export didn’t take care of transparency. (You can create a glass shader and apply it manually)
Initially I was following that video I had linked and exported through the audio2face section of the omniverse tab in blender and after doing the proper static and dynamic mesh labeling, i exported as “face usd”.
I followed your steps and exported usd below the nucleus section in the omniverse tab and got seem to have gotten much closer as the textures now all seem to have been added however everything seems to be a shiny metallic texture now.
Any idea why that might be?
At first it was just almost pitch black, but after adding a distant light and angling it correctly I was able to make the shine visible
It looks like it can’t find the texture folder. Can you check and see if you have the Maps folder right beside the exporte Rain.usda file?
Should it be named maps? When I export usd it automatically adds a “texture” folder next to the usd file.
as shown above it seems that the materials are added to the stage, as well as applied to the mesh.
And the texture folder is right next to the usd file.
Although I just checked inside the texture folder and I don’t see any files there at all. Not even hidden files. That would explain why it can’t find them like you said. If there is nothing in the texture folder how would audio2face be recognizing the material names properly?
Here is the export view from blender. As far as I can tell I have the same settings you did?
You’re right, that folder is called textures. Did you make sure to set the texture folder before exporting from Blender, like it’s highlighted in this image:
Oh. Should the textures directory be the place I want them to be exported to, or where I want to export them from?
If it should be the location I want to export from, I’m not sure where I would find that in blender, because isn’t it all just in the .blend file?
If it is supposed to be the output location, that is what I have been trying to do.
The selected textures directory is this textures folder right next to the usd export. But after clicking ‘OK’ it remains empty
interesting, when I extract the Rain rig, it has the maps folder right beside the blender file. this maps folder is the one you need to point to before exporting, at least that’s what I did
Oh I see. That was the misunderstanding I was having. I don’t remember seeing any maps folder so I redownloaded the model and after extracting this time I do have a maps folder. I’m not sure what happened last time.
I exported that with the textures directory pointing to maps and after importing to A2F it finally works!!
Thanks for all of the help!
I’m not sure how familiar you are with the inner workings of A2F and this blender alpha branch, but in that tutorial I was following they had me go into this Audio2Face tab under omniverse in blender:
Inside of that I separated the left and right eye meshes, and exported the relevant static and dynamic meshes, along with the face mesh, and clicked I think “export face usd” with the project file for an easy import and character transfer to the audio2face app.
I think a potential problem with that method is that there is no way to manually select a material directory (where we select the maps folder as you instructed).
Do you know of any way to have both that face export and the texture export working at the same time?
If not I’m thinking it might be best to either:
- manually do the character transfer with the textured model, adding the body back in when the audio2face processing is finished?
- Alternatively maybe import both the rain usd and the face usd into the same audio2face instance and manually apply the textures to the face usd after the character transfer steps?
I’m not entirely sure how the software is working behind the scenes. Now that you helped out the texture issue I can probably figure it out eventually on my own, but if you have a strong understanding of how it works together I’d really appreciate if you could help point me in the right direction in reference to the bold points above.
