ACE Unreal plugin (UE 5.6): Audio2Face lip sync stops working after the first clip — stale GetPlaybackPercent from pooled mixer sources wipes the blen

Gaming and Visualization Technologies Engines Unreal
1

Setup: UE 5.6.0, Win64, MSVC 14.44, ACE_TTS + ACE_ASR v5.6 1.2 (beta), ACE_LLM 5.6 v0.5, NV_ACE_Reference UE 5.6.
Bug: Audio2Face lip sync only works for the first clip per PIE session; later clips play audio but the face never moves.
Root cause: UACEAudioCurveSourceComponent creates a new UAudioComponent per stream, but UE pools mixer sources, and FMixerSource::GetPlaybackPercent() returns the previous clip’s position for the first frames (the engine bug already noted in comments in HandlePlaybackFraction). That stale value is stored into AudioPlaybackTimeEstimate unconditionally, so the animation clock for clip 2 jumps to near the end of clip 1. GetCurveOutputsInterp() then finds no sample at that time (bIsPlayed == false) and calls ResetAnimSamples(), destroying the whole clip’s blendshape buffer while chunks are still streaming in.
Fix that works for me: (a) in HandlePlaybackFraction, reject estimates greater than TotalReceivedAudioTime (playback can’t be ahead of received audio, so such values can only be stale); (b) in GetCurveOutputsInterp, only ResetAnimSamples()/transition to ENDING when bAnimationAllFramesRecived is true, instead of on any overshoot mid-stream.

2

diff --git a/Plugins/NV_ACE_Reference/Source/ACERuntime/Private/ACEAudioCurveSourceComponent.cpp b/Plugins/NV_ACE_Reference/Source/ACERuntime/Private/ACEAudioCurveSourceComponent.cpp
index fa8965c..001fc36 100644
— a/Plugins/NV_ACE_Reference/Source/ACERuntime/Private/ACEAudioCurveSourceComponent.cpp
+++ b/Plugins/NV_ACE_Reference/Source/ACERuntime/Private/ACEAudioCurveSourceComponent.cpp
@@ -579,7 +579,18 @@ void UACEAudioCurveSourceComponent::HandlePlaybackFraction(const UAudioComponent

FScopeLock Lock(&AudioCompCS);
// note: the engine calls this "percentage" but it's actually a fraction, so we renamed the variable for clarity. No need to multiply by 0.01f

  • AudioPlaybackTimeEstimate = InSoundWave->GetDuration() * InPlaybackFraction;

  • // Reject values reporting playback beyond the audio received so far: pooled mixer sources report the

  • // PREVIOUS clip’s position for the first frames of a new clip (the engine bug described above), and

  • // accepting one desyncs the animation clock so far ahead that the sample buffer gets discarded.

  • const float NewPlaybackTimeEstimate = InSoundWave->GetDuration() * InPlaybackFraction;

  • if (NewPlaybackTimeEstimate <= TotalReceivedAudioTime)

  • {

  •   AudioPlaybackTimeEstimate = NewPlaybackTimeEstimate;

  • }

  • else

  • {

  •   UE_LOG(LogACERuntime, Verbose, TEXT("Rejecting implausible playback time %f (only %f s of audio received)"), NewPlaybackTimeEstimate, TotalReceivedAudioTime);

  • }
    UE_LOG(LogACERuntime, VeryVerbose, TEXT(“UACEAudioCurveSourceComponent::HandlePlaybackFraction Current Tick %d”), FDateTime::Now().GetTicks());
    UE_LOG(LogACERuntime, VeryVerbose, TEXT(“UACEAudioCurveSourceComponent::HandlePlaybackFraction AudioPlaybackTimeEstimate %f”), AudioPlaybackTimeEstimate);

@@ -877,11 +888,17 @@ void UACEAudioCurveSourceComponent::GetCurveOutputsInterp(TArray& OutWeig
// animation buffer ran out - nothing to play new, return default pose from last frame
if (!bIsPlayed)
{

  •   	// clear the buffer, it played

  •   	ResetAnimSamples();

  •   	if ((AnimState == EAnimState::IN_PROGRESS) && bAnimationAllFramesRecived)

  •   	// Only treat this as the end of the clip when every frame has actually arrived.

  •   	// Mid-stream, a transient bogus playback time (see HandlePlaybackFraction) can

  •   	// overshoot the received samples; wiping the buffer then destroys the whole clip.

  •   	if (bAnimationAllFramesRecived)
  	{

  •   		AnimState = EAnimState::ENDING;

  •   		// clear the buffer, it played

  •   		ResetAnimSamples();

  •   		if (AnimState == EAnimState::IN_PROGRESS)

  •   		{

  •   			AnimState = EAnimState::ENDING;

  •   		}
  	}

  	return;