Failed to build TensorRT engine

[zanarevshatyan]

Hi, I can’t even start using the audio2face program, the program tries to build TensorRT but at approximately 50% ic comes up with an error saying “Failed to build TensorRT engine”

Here is the log with errors

2022-11-22 22:34:09 [412,550ms] [Error] [omni.audio2face.common.scripts.utils] Unable to build TensorRT model for: c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.emotion_deps\deps\audio2emotion-nets/nets\a2e_v1.3\model.onnx
2022-11-22 22:34:09 [412,570ms] [Error] [omni.audio2face.common.scripts.utils] Async Task Failed: <Task finished coro=<A2FCoreFullfaceInstance._init_a2e_processor_async() done, defined at c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.core\omni\audio2face\core\scripts\instance_fullface.py:463> exception=RuntimeError(“Error during running command: [‘c:\\users\\vladi\\appdata\\local\\ov\\pkg\\audio2face-2022.1.2\\exts\\omni.audio2face.inference_deps\\deps\\audio2face-inference-deps\\bin\\trtexec.exe’, ‘–onnx=c:\\users\\vladi\\appdata\\local\\ov\\pkg\\audio2face-2022.1.2\\exts\\omni.audio2face.emotion_deps\\deps\\audio2emotion-nets/nets\\a2e_v1.3\\model.onnx’, ‘–saveEngine=C:\\Users\\vladi\\AppData\\Local\\ov\\cache\\Kit\\103.1\\641148c1\\audio2emotion-nets\\a2e_v1.3\\model_device0_fp32.trt’, ‘–device=0’, ‘–explicitBatch’]”)>
2022-11-22 22:34:09 [412,571ms] [Error] [omni.audio2face.common.scripts.utils] Error: Error during running command: [‘c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.inference_deps\deps\audio2face-inference-deps\bin\trtexec.exe’, ‘–onnx=c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.emotion_deps\deps\audio2emotion-nets/nets\a2e_v1.3\model.onnx’, ‘–saveEngine=C:\Users\vladi\AppData\Local\ov\cache\Kit\103.1\641148c1\audio2emotion-nets\a2e_v1.3\model_device0_fp32.trt’, ‘–device=0’, ‘–explicitBatch’]
2022-11-22 22:34:09 [412,575ms] [Error] [omni.audio2face.common.scripts.utils] Traceback (most recent call last):
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\utils.py”, line 129, in _done_callback
task.result()
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.core\omni\audio2face\core\scripts\instance_fullface.py”, line 470, in _init_a2e_processor_async
await self._a2e_processor.set_network(network_fpath, network_name)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.emotion\omni\audio2face\emotion\scripts\processor.py”, line 141, in set_network
await self._build_and_load_network(network_path, network_name, self._device_id, self._trt_info)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.emotion\omni\audio2face\emotion\scripts\processor.py”, line 331, in _build_and_load_network
estimated_trt_builder_time,
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\utils.py”, line 50, in wrapper
return await func(*args, **kwargs)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.inference\omni\audio2face\inference\scripts\trt_builder.py”, line 59, in prepare_and_load_trt_model
load_model_fn, input_model_fpath, trt_model_fpath, optimizer_params
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\utils.py”, line 50, in wrapper
return await func(*args, **kwargs)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.inference\omni\audio2face\inference\scripts\trt_builder.py”, line 81, in _build_or_rebuild_and_load_trt_model
await self._build_and_load_trt_model(load_model_fn, input_model_fpath, trt_model_fpath, optimizer_params)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\utils.py”, line 50, in wrapper
return await func(*args, **kwargs)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.inference\omni\audio2face\inference\scripts\trt_builder.py”, line 87, in _build_and_load_trt_model
await self._build_trt_model(input_model_fpath, trt_model_fpath, optimizer_params)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\utils.py”, line 50, in wrapper
return await func(*args, **kwargs)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.inference\omni\audio2face\inference\scripts\trt_builder.py”, line 116, in _build_trt_model
await run_cmd(cmd, cmd_name=“TRTEXEC”, progress_fn=self._build_trt_model_progress)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\utils.py”, line 50, in wrapper
return await func(*args, **kwargs)
File “c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.common\omni\audio2face\common\scripts\cmd_runner.py”, line 63, in run_cmd
raise RuntimeError(“Error during running command: {}”.format(cmd))
RuntimeError: Error during running command: [‘c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.inference_deps\deps\audio2face-inference-deps\bin\trtexec.exe’, ‘–onnx=c:\users\vladi\appdata\local\ov\pkg\audio2face-2022.1.2\exts\omni.audio2face.emotion_deps\deps\audio2emotion-nets/nets\a2e_v1.3\model.onnx’, ‘–saveEngine=C:\Users\vladi\AppData\Local\ov\cache\Kit\103.1\641148c1\audio2emotion-nets\a2e_v1.3\model_device0_fp32.trt’, ‘–device=0’, ‘–explicitBatch’]

I’ve also tried to run the Omniverse with Administrator rights, didn’t work, run with antivirus off, also the same.
Please help :(

[zanarevshatyan]

I’ve tested a bit and managed to make it work but only with regular a2f core connecting to regular head template without eyes, in this case building TensorRT engine went succesfull, but the problem remains when I try to add full-face core :(

This error with TensorRT came with Audio2Face 2022.1.2 version, I’ve also tried uninstalling it, installing older versions such as 2022.1.1 and 2022.1.0 in those two cases I had succes on TensorRT procces for the emotion tab in full-face demo template but unfortunately I get the same TensorRT error with the log file that I’ve attached when the program is trying to build TensorRT engine for auto-emotion tab and I can’t control the character’s emotions, shortly saying I’m not in a such a big need of that auto emotion tab, maybe there is a way to use full-face feautures on with succeded “Emotion” tab?

And also here are the specs

Device name	LAPTOP-AGD9AUN9
Processor	11th Gen Intel(R) Core™ i5-11400H @ 2.70GHz 2.69 GHz
Installed RAM	8,00 GB (7,74 GB usable)
Device ID	4C634148-3598-4D42-9F65-E0F479FFFA44
Product ID	00342-20798-19815-AAOEM
System type	64-bit operating system, x64-based processor
Pen and touch	No pen or touch input is available for this display

Running this beauty on Windows 11 Home

Also uploading a copy of my logs from here: `C:\Users<USERNAME>.nvidia-omniverse\logs\Kit\Audio2Face

2022.1.rar (493.8 KB)

For Gpu drivers I’m running the program on dual graphic carded laptop , here is their info

1. Intel(R) UHD Graphics
   Driver Date - 19.08.2022
   Driver Version - 31.0.101.3413

2. NVIDIA GeForce RTX 3050 Laptop GPU
   Driver Date - 08.11.2022
   Driver Version - 31.0.15.2686

Also the detailed info for my Windows version

[zanarevshatyan]

I’ve also asked my brother to check the program on his end using his laptop with a Nvidia graphic card, he has the same error, he said he googled and seems that this new version with full-face mechanics don’t work on laptops :((((

It is extra sad, cus the older version with no eyes workes on my laptop just fine.

Can I hope that you will add an ability to use audio2face new full-face mechanics on laptops in the next version of the program? I will kindly be the happy tester with a rtx laptop :D

I’ve also read that one of your developers asked for an info from cmd, and I’ve thought it will be better if I also send the screen from my cmd, I have only one local disk C:

Failed to Load TensorRT Engine

Hello @user133974! Welcome to the Community! Thanks for attaching your logs and computer specs. I am checking in with the Audio2Face team. I will post back when I have more information!

By the way I don’t have such a path like omniverse in my local disk C: , this is very strange Watson :/

[zanarevshatyan]

Guys my opinion on this issue is that I’m running out of gpu memory, but however I’ve manages to succesfully build the fisrt emotion tab with tensorRT and it is all I need, I don’t need the second tab with auto-emotion that gives me an error and I can’t make it work for the whole full-face pipeline, maybe there is a way to work with the full-face mimics without building auto-emotion-tab? Or if there is no way, maybe you can add the possibility to work with full-face pipeline only using emotion tab in the next version of the software? It will be awesome for the people that don’t have much gpu memory.

[SimonOuellet]

Hi, that is a great suggestion, we will strongly consider it. I think it is important to have nice fallback strategy, what you propose is in line with that.

[zanarevshatyan]

Oh, thank you, it is not a great suggestion, it is just a thought.

And also I have strange but good news)))

I’ve managed somehow to build tensorRT for the auto-emotion tab also and now audio2face’s full-face pipeline works just fine, before it I was trying to build tensorRT for auto-emotion tab for hundred times and hundred times I was getting that error, somehow at a great night I’ve managed to pass that build, I don’t know how :D

Guys can try to maybe reduce brightness of the display or also try to update their windows if it needs so, those are all the major changes after which tensorRT successfuly built.

Thank you guys, you are breathtaking.