Unable to generate blendshapes for character

AI & Data Science Digital Humans (closed) Audio2Face (closed)
1

Not sure if my desktop is running out of memory or what (I have created blendshapes for other characters before), but when I do it for this character I get a rapidly scrolling error messages in the console looping apparently forever. Sharing in case anyone has a suggestion of what I need to change. It is most likely that I am doing something wrong (I generate the testing.usda file from a GLB file), but not sure what it is. (E.g. do I have to delete all existing blendshapes first?)

Note: I am only after the blendshapes in this particular case. I can do all the correspondence, I have Xforms above the eyes, got all the meshes connected to stop those error messages etc.

I am using a free tool called VRoid Studio to create models. I believe all models have one of two or three mesh structures (e.g. male and female), so I should be able to transfer all the blendshapes across once I have one good set.

Any suggestions appreciated!

2

A little more information. I gave it another go this morning. Tried a few characters, demo scenes, etc. I get a2f crashes etc - for some reason I cannot get a2f to generate even a simple blendshape, but its hard to work out what is wrong with my character model. I get different things go wrong, like the face mesh gets copied to the transfer character, but is then invisible. Another time a2f crashes (no errors in console).

Some errors from a2f log file in case any are relevant.

  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/extscore/omni.kit.async_engine/omni/kit/async_engine/async_engine.py", line 154, in _on_update
    self._loop.run_forever()
  File "d:\omniverse\pkg\audio2face-2023.1.1\kit\python\lib\asyncio\windows_events.py", line 321, in run_forever
    super().run_forever()
  File "d:\omniverse\pkg\audio2face-2023.1.1\kit\python\lib\asyncio\base_events.py", line 603, in run_forever
    self._run_once()
  File "d:\omniverse\pkg\audio2face-2023.1.1\kit\python\lib\asyncio\base_events.py", line 1909, in _run_once
    handle._run()
  File "d:\omniverse\pkg\audio2face-2023.1.1\kit\python\lib\asyncio\events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.hotkeys.core/omni/kit/hotkeys/core/trigger.py", line 76, in _run
    self.__trigger(key_combination, pos_x, pos_y)
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.hotkeys.core/omni/kit/hotkeys/core/trigger.py", line 71, in __trigger
    hotkey.execute()
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.hotkeys.core/omni/kit/hotkeys/core/hotkey.py", line 76, in execute
    self.action.execute()
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.menu.edit/omni/kit/menu/edit/scripts/edit_actions.py", line 204, in <lambda>
    lambda: cls.delete_prim(False),
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.menu.edit/omni/kit/menu/edit/scripts/edit.py", line 606, in delete_prim
    omni.kit.commands.execute("DeletePrims", paths=paths, destructive=destructive)
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.commands/omni/kit/commands/command.py", line 463, in execute
    result = omni.kit.undo.execute(command, name, kwargs)
[...skipped...]
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.commands/omni/kit/undo/undo.py", line 77, in execute
    result = _execute(command, name, level, history_key)
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.commands/omni/kit/undo/undo.py", line 458, in _execute
    raise error
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.kit.commands/omni/kit/undo/undo.py", line 419, in _execute
    result = command.do()
  File "d:/omniverse/pkg/audio2face-2023.1.1/kit/exts/omni.usd/omni/usd/commands/usd_commands.py", line 1167, in do
    with Sdf.ChangeBlock():

 <class 'pxr.Tf.ErrorException'> 
	Error in 'pxrInternal_v0_22__pxrReserved__::UsdStage::_HandleLayersDidChange' at line 3943 in file W:\ac88d7d902b57417\USD\pxr\usd\usd\stage.cpp : 'Detected usd threading violation.  Concurrent changes to layer(s) composed in stage 000001CF9898C170 rooted at @D:/omniverse/pkg/audio2face-2023.1.1/exts/omni.audio2face.wizard/assets/demo_fullface_mark.usda@.  (serial=2927, lastSerial=2952).'
2023-10-02 16:22:34 [593,591ms] [Info] [omni.usd] [Watcher] MATERIAL STARTING to create a mdl entity named 'gltf_material' in module 'D:\omniverse\pkg\audio2face-2023.1.1\kit\mdl\core\mdl\gltf\pbr.mdl' from prim '/World/testing/Looks/N00_000_00_FaceMouth_00_FACE__Instance_/N00_000_00_FaceMouth_00_FACE__Instance_'
2023-10-02 16:22:34 [593,591ms] [Info] [omni.usd] [Watcher] MATERIAL STARTING to create a mdl entity named 'gltf_texture_lookup' in module 'D:\omniverse\pkg\audio2face-2023.1.1\kit\mdl\core\mdl\gltf\pbr.mdl' from prim '/World/testing/Looks/N00_000_00_FaceMouth_00_FACE__Instance_/baseColorTex'
2023-10-02 16:22:34 [593,591ms] [Info] [omni.usd] [Watcher] MATERIAL STARTING to create a mdl entity named 'gltf_normal_texture_lookup' in module 'D:\omniverse\pkg\audio2face-2023.1.1\kit\mdl\core\mdl\gltf\pbr.mdl' from prim '/World/testing/Looks/N00_000_00_FaceMouth_00_FACE__Instance_/normalTex'
...
2023-10-02 16:22:34 [593,645ms] [Warning] [omni.usd] Coding Error (secondary thread): in _Get at line 2998 of W:\ac88d7d902b57417\USD\pxr\usdImaging\usdImaging\delegate.cpp -- Failed verification: ' prim '
...
2023-10-02 16:26:58 [857,330ms] [Warning] [omni.usd] Coding Error (secondary thread): in Get at line 365 of C:\b\w\kit\kit\source\extensions\omni.hydra.scene_delegate\plugins\omni.hydra.scene_delegate.plugin\OmniHydraDelegate.cpp -- Failed verification: ' cacheItem ' -- </__Prototype_1/defaultLight_02> (intensity)
...
2023-10-02 16:26:58 [857,545ms] [Warning] [omni.kit.window.filepicker.model] Failed to find item at 'http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Audio2Face/Samples_2023.1/char_transfer'
2023-10-02 16:26:58 [857,545ms] [Warning] [omni.kit.window.filepicker.api] Uh-oh! item at 'http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Audio2Face/Samples_2023.1/char_transfer' not found.
2023-10-02 16:26:58 [857,545ms] [Warning] [omni.kit.window.filepicker.model] Failed to find item at 'http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Audio2Face/Samples_2023.1'
2023-10-02 16:26:58 [857,546ms] [Warning] [omni.kit.window.filepicker.api] Uh-oh! item at 'http://omniverse-content-production.s3-us-west-2.amazonaws.com/Assets/Audio2Face/Samples_2023.1' not found.
2023-10-02 16:26:58 [857,547ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_omni_kit_widget_viewport_ViewportTexture_0) assigned to device 0
2023-10-02 16:26:58 [857,547ms] [Info] [rtx.multigpumanager.plugin] View 0 (/Render/RenderProduct_omni_kit_widget_viewport_ViewportTexture_0) switching to 1x1 layout
2023-10-02 16:26:58 [857,573ms] [Warning] [omni.kit.hotkeys.core.registry] [Hotkey] duplicated key:'P (On Press)'
...
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_000_Hair_00_HAIR_03__Instance_/normalTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_000_Hair_00_HAIR_03__Instance_/emissiveTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_000_Hair_00_HAIR_04__Instance_/baseColorTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_000_Hair_00_HAIR_04__Instance_/emissiveTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_000_Hair_00_HAIR_04__Instance_/normalTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_000_Hair_00_HAIR_04__Instance_/N00_000_Hair_00_HAIR_04__Instance_ (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_01_Shoes_01_CLOTH__Instance_/baseColorTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_01_Shoes_01_CLOTH__Instance_/emissiveTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_01_Shoes_01_CLOTH__Instance_/normalTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_01_Shoes_01_CLOTH__Instance_/N00_001_01_Shoes_01_CLOTH__Instance_ (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_02_Bottoms_01_CLOTH__Instance_/baseColorTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_02_Bottoms_01_CLOTH__Instance_/N00_001_02_Bottoms_01_CLOTH__Instance_ (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_02_Bottoms_01_CLOTH__Instance_/emissiveTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_001_02_Bottoms_01_CLOTH__Instance_/normalTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_005_01_Tops_01_CLOTH__Instance_/baseColorTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_005_01_Tops_01_CLOTH__Instance_/emissiveTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_005_01_Tops_01_CLOTH__Instance_/normalTex (rtx)
2023-10-02 16:33:03 [1,222,340ms] [Info] [omni.hydra] Destroying MdlShadeNode for prim: /World/testing/Looks/N00_005_01_Tops_01_CLOTH__Instance_/N00_005_01_Tops_01_CLOTH__Instance_ (rtx)

(crash)

No idea what the problem is or how to work around it. I have got blendshapes generated in the past - not sure what is different about this model.

3

This is 2023.1.1 by the way. I stripped the character back to only a face mesh, nothing else, and it still failed. Clicking character setup makes the mesh disappear (it flips over to the transfer character, but that mesh is not visible).

However! I got a new error message in the log, so sharing that
2023-10-02 21:03:12 [Error] [gpu.foundation.plugin] Can't find matching files for UDIM D:\omniverse\pkg\audio2face-2023.1.1\exts\omni.kit.charTransfer\assets\mark_skin_c_zb_normals_tangent.<UDIM>.exr

Next I am going to install 2022 and give the older version a go. Maybe some cloud hosted files are missing, causing problems.

4

Ah - 2022.1.1 works for me, 2023 does not. Since I am only generating blendshapes, I will go back to the older version for now. However when I export the blendshapes, it only seems to export faceSkin (no blendshapes for other meshes).

5

There’s a bug in 2023.1.1 where generating blendShapes does not work after applying A2F Pipeline. This will be resolved in the next release. But for now, the workaround is to generate blendShapes before applying the A2F pipeline.

6

In my case just created blendshapes - I was not using the a2f pipeline. This video has the steps I was using in 2022.3 (I tried to do the same flow - taking UI changes into account - in 2023.1 but it did not create working blendshapes. After a few hours of trying different combinations and sequences I went back to 2022.3 and it worked first time, so I moved on.)

It could be user error, but it beats me what I was doing wrong! No error messages, just no working blendshapes.

(One extension that would be really useful is one that shows you all the blendshape weights and lets you play with them live - makes testing easier. I might give it a go one day. I assume the right approach is to create a new SkelAnimation and assign that to the model, then give the user draggers for all the available blendshape names. Or does it already exist somewhere?)

1 Like
7

We have a ticket for a blendShape editor. Just need to find some time to implement it. In the meantime you might be able to find similar extensions other users might’ve created. I think I’ve seen it somewhere on Github.

Great video! btw, the pumpkin head moves back to the right because the joint has animation on it. You can move the group above it instead.

1 Like
8

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.