[Audio2Face / Digital Human] SDK v3.0 Diffusion Model — Frame gap during streaming inference (posting here as Digital Human board is closed)

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Note: The Digital Human board appears to be closed, so I’m posting here instead. Apologies if this isn’t the ideal category — happy to move it if there’s a better place.

Hi all,

Background

The official Audio2Face-3D-SDK tutorials demonstrate batch-mode inference using Maya ACE, which works well for offline performance testing. However, our goal is real-time streaming — we need to stream facial expressions to an Unreal Engine MetaHuman via the ACE Plugin over gRPC, so the user sees lip-sync animation as the audio plays, not after it finishes.

To achieve this, we’re building a self-hosted gRPC server using the Audio2Face-3D-SDK v3.0 Diffusion model (ARKit 52 blendshapes @ 60FPS), intended as a replacement for the Audio2Face NIM service which uses the older v2.3 Regression model.

Batch mode works fine — the issue is specific to streaming.

Problem

When streaming audio chunks to the SDK incrementally, the Diffusion model produces frames in bursts of ~28 frames every ~470ms, due to its 1-second sliding window with 0.5-second stride. This is fundamentally different from the Regression model (NIM), which outputs 1 frame per audio chunk continuously.

The burst pattern itself isn’t a problem in steady state — the burst size (~467ms of content) closely matches the burst interval (~470ms), so playback stays smooth. The problem occurs during the warmup-to-steady-state transition:

Time(ms)  Event                          Frames
────────────────────────────────────────────────
+0        First audio chunk arrives       0
+138      Warmup burst                    15 frames (250ms of content)
+180~448  Audio accumulating...           0 frames (gap!)
+558      First normal burst              28 frames (467ms of content)
+1028     Second normal burst             28 frames (steady state)

The warmup burst only produces ~15 frames (250ms), and the next burst doesn’t arrive until ~560ms. This creates a ~170ms playback gap around the second syllable of speech, where both animation and audio stutter simultaneously (since our server sends audio buffers alongside frames).

Our Workaround — Server-side Frame Pacing

We implemented a frame pacing layer that buffers the burst frames and releases them to the client at a constant 60FPS interval, which eliminates the stutter completely.

The Tradeoff

Frame pacing requires pre-buffering ~30 frames before starting delivery, which adds ~500ms of initial latency — a significant penalty for real-time conversational use cases. The Regression model (NIM) doesn’t have this constraint since frames are produced continuously.

Questions

  1. Is there a recommended approach for streaming the Diffusion model output without significant buffering latency?

  2. Are there any SDK-level parameters to control the stride length or warmup behavior?

  3. Is this a known limitation of the Diffusion architecture, or are there plans to address streaming smoothness in future SDK versions?

Any guidance would be greatly appreciated. Happy to share server logs or frame timing data if helpful.

Environment:

  • Audio2Face-3D-SDK v3.0 (multi-diffusion model)

  • CUDA 12.9, TensorRT 10.14

  • Python gRPC server with ctypes bindings to the SDK C API

  • Client: Unreal Engine 5 MetaHuman + ACE Plugin (gRPC)