NIM Audio2Face-3D: ManifestProfileSelector fails on A100 after restart when /tmp/a2x TRT cache is persisted in Azure Container Apps

AI & Data Science NVIDIA NIM Access/Accounts
, , ,
1

Hi everyone,

We are deploying the NIM Audio2Face-3D Microservice on Azure Container Apps with an A100 80GB PCIe, and we are hitting a reproducible issue related to TensorRT engine cache invalidation across container restarts.

Environment

  • GPU: NVIDIA A100 80GB PCIe (Azure Container Apps)
  • Service: NIM Audio2Face-3D Microservice
  • NIM flag: NIM_DISABLE_MODEL_DOWNLOAD=true (required per official docs for GPUs without pre-generated profiles)
  • Cache path: /tmp/a2x mounted as persistent volume in Azure Container Apps

Reference: Troubleshooting — Audio2Face-3D

What works on first start

GPU is correctly detected:

inference:detect_gpu:340 - Detected GPU via nvidia-smi: NVIDIA A100 80GB PCIe

TRT engines are found in cache:

inference:find_cached_trt_engines:421 - Found cached TRT in /tmp/a2x: a2e.trt
inference:find_cached_trt_engines:421 - Found cached TRT in /tmp/a2x: james_v2.3.1.trt

First deployment completes successfully.

What fails on restart (same image, same GPU)

After restarting the Container App — with no changes to image, GPU or driver — startup fails with:

inference:match_profile_to_cached_engines:511 - X Cannot match profile: GPU detection failed
inference.py:245 Error using FallbackProfileSelector: GPU detection required for profile matching
profiles.py:227 TagsBasedProfileSelector not able to find the profile: No matching profiles were found, 0 profiles remaining
profiles.py:146 ManifestProfileSelector found no compatible profiles
nimlib.exceptions.NIMProfileIDNotFound: Could not match a profile in manifest at /opt/nim/etc/default/model_manifest.yaml

Full traceback:

image
image1530×432 28.3 KB

Any guidance on the intended lifecycle of the /tmp/a2x cache would be very helpful.

Thanks,
Joan.