We’re putting some existing T4 cards to use for speech-to-text. We know the T4 is older hardware. But the fault below is in CPU-side text post-processing (std::regex inside ITN), not in GPU code, so we don’t think it’s specific to the T4.
**Environment**
- NIM: nvcr.io/nim/nvidia/parakeet-0-6b-ctc-en-us (pulled as :latest; RIVA_SPEECH_VERSION=2.22.0, Triton 25.03)
- GPU: 2 x Tesla T4, one NIM container per GPU, nginx gRPC load balancer in front
- Driver 550.163.01 (CUDA 12.8 compat libs), Debian 13, 80 CPU cores
- TLT/TAO: not used
- Model config as shipped: the offline profile runs with streaming: true internally (4.8 s chunks, greedy CTC endpointing), postprocessor_pool_threads: 8
**Workload**
- gRPC Recognize (offline), mono 16 kHz LINEAR_PCM (telephony audio resampled), one call channel per request. Requests up to about 17 minutes long.
- ITN and automatic punctuation on, word boosting on (about 33 phrases).
- Steady production load: roughly 3,000 requests per backend per day.
**Symptom**
- About once per 3,000 requests per backend, tritonserver segfaults and dumps core; the container restarts.
- Requests in flight at the moment of the crash can return OK with an empty result (no error status), which is easy to mistake for silence.
- Re-sending the same audio afterwards succeeds, so it does not look input-specific (3 of 3 re-sent recordings completed normally).
**Backtrace (same in 5 of 5 cores)**
- Crashing thread: pool task StreamingAsrEnsemble::FilterAndNormalizeTranscripts → ExecuteITN → std::regex _M_dfs, faulting on the first character of a short (12-byte) subject string.
- Another thread is in StreamingAsrEnsemble::PostProcessTranscripts(…), blocked on that task’s std::future with no timeout. It takes the transcript vectors by reference.
- The batch thread StreamingAsrEnsemble::ExecuteBatchLoop has already returned to SequenceDynamicBatcher::GetBatch.
Our reading (not confirmed): the transcript buffer is freed or reallocated while the asynchronous ITN task still references it. That looks like a race between the batch loop moving to the next window and post-processing of the previous one. Long requests would expose it more, because each request goes through many 4.8 s windows.
Possibly related older thread: NVIDIA RIVA ASR going down frequently for en-US
**Questions**
1. Is this a known issue, and is it fixed in a later Speech NIM / Riva release?
2. Is there a supported setting that avoids it, e.g. postprocessor_pool_threads: 1? Disabling ITN isn’t an option for us.
3. We can share the full gdb backtrace (all threads) privately if that helps.