TRT 10.14: dynamic-shape InstanceNormalization crashes with CUDA 720 in __myl_Inor kernel

AI & Data Science Deep Learning (Training & Inference) TensorRT
,
1

Engines built on TensorRT 10.14.1.48 from any ONNX containing InstanceNormalization with a dynamic batch dimension crash at inference time:

[E] Error[1]: [exec_instruction.cpp:1274: exec] CUDA error 720 launching __myl_Inor kernel.
terminate called after throwing an instance of 'nvinfer1::MyelinError'
Aborted (core dumped)

The Myelin-fused kernel only succeeds when the runtime batch equals --optShapes. Any other batch in [minShapes, maxShapes] faults with CUDA 720. The same ONNX and the same trtexec flags work on TRT 10.9.0.34 at every batch in the dynamic range.

compute-sanitizer confirms the OOB access is inside the Myelin-generated __myl_Inor kernel, dependent on the dynamic batch dim.

Environment

  • TensorRT Version: 10.14.1.48 (cuda13.0)
  • GPU Type: RTX 4060 (sm_89)
  • Nvidia Driver Version: 590.48.01
  • CUDA Version: 13.1
  • CUDNN Version: 9.17.1.4
  • Operating System + Version: Ubuntu 24.04.3 LTS (in container)
  • Python Version: 3.12.3
  • Container: nvcr.io/nvidia/deepstream:9.0-triton-multiarch

Relevant Files

OSNet AIN x1.0 ONNX

Steps To Reproduce

Build the engine:

trtexec --onnx=osnet_ain_x1_0.onnx --saveEngine=osnet_ain.engine --fp16 \
  --minShapes=input:1x3x256x128 \
  --optShapes=input:16x3x256x128 \
  --maxShapes=input:16x3x256x128

Run at opt batch → succeeds:

trtexec --loadEngine=osnet_ain.engine --shapes=input:16x3x256x128

Run at any other batch in range → crashes with the trace above:

trtexec --loadEngine=osnet_ain.engine --shapes=input:8x3x256x128

Reverting only the TensorRT version to 10.9.0.34 (DeepStream 8 image, same hardware, same ONNX, same flags) makes every batch 1..16 work, so the regression is isolated to TRT 10.14’s Myelin path for InstanceNormalization under dynamic shapes.

3

*** Please note that this reply is generated by LLM automatically ***

The issue reported is a crash at inference time when running a TensorRT engine built from an ONNX model with a dynamic batch dimension. The crash occurs when the runtime batch size is not equal to 1, and it is specific to TensorRT version 1014148.

To troubleshoot this issue, let’s analyze the provided information:

  1. TensorRT Version: The issue is specific to TensorRT version 1014148. Reverting to version 109034 resolves the issue.
  2. ONNX Model: The ONNX model contains a dynamic batch dimension, which is supported by TensorRT.
  3. GPU Type: The issue occurs on an RTX 4060 GPU with sm89 architecture.
  4. CUDA Version: The CUDA version is 13.1, which is compatible with the RTX 4060 GPU.
  5. cuDNN Version: The cuDNN version is 9.1.7.14, which is a compatible version with the CUDA version.
  6. Operating System: The issue occurs on Ubuntu 24.04.3 LTS in a container.
  7. Python Version: The Python version is 3.12.3, which is a compatible version with the TensorRT version.

Based on the analysis, the issue is likely related to a regression in the Myelin kernel, which is specific to TensorRT version 1014148. The Myelin kernel is responsible for optimizing the engine for dynamic shapes.

To resolve the issue, you can try the following:

  1. Revert to a previous TensorRT version: Reverting to TensorRT version 109034 resolves the issue.
  2. Update the Myelin kernel: If possible, update the Myelin kernel to a version that is compatible with TensorRT version 1014148.
  3. Disable dynamic shapes: If the issue is specific to dynamic shapes, you can try disabling dynamic shapes in the ONNX model or in the TensorRT engine.

In comparison to other frameworks, the issue is specific to TensorRT and is not related to the ONNX model or the GPU architecture. The issue is likely related to a regression in the Myelin kernel, which is specific to TensorRT version 1014148.

To provide more information, you can try the following:

  1. Provide the ONNX model: Share the ONNX model that is causing the issue.
  2. Provide the TensorRT engine: Share the TensorRT engine that is causing the issue.
  3. Provide the inference code: Share the inference code that is causing the issue.
  4. Provide the CUDA trace: Share the CUDA trace that is generated when the issue occurs.

By providing more information, you can help the development team to identify the root cause of the issue and provide a fix for the regression in the Myelin kernel.

*** Please give the thumbs up if you get answers you like. Or provide feedback to help us improve the answer. ***

4

Hi, thanks for the response — I think this might be the auto-triage reply, since it mostly restates my report.

Quick notes on the three suggestions:

  1. Revert TensorRT — I can’t, the container nvcr.io/nvidia/deepstream:9.0-triton-multiarch ships TRT 10.14.1.48 and downgrading would break DeepStream 9.

  2. Update the Myelin kernel — Myelin is internal to libnvinfer, so not something I can update on my side.

  3. Disable dynamic shapes — the model is used by DeepStream’s NvMultiObjectTracker ReID stage, which sends inference at the actual per-batch object count, so dynamic batch is required.

5

Hey @moishizra,

You’re exactly right, that earlier reply was an automated triage pass, and your three counters are completely valid: the DeepStream 9 image is pinned to TRT 10.14.1.48, Myelin isn’t user-tunable, and dynamic batching is non-negotiable for the NvMultiObjectTracker ReID stage. Apologies for the noise on that one!

The repro you posted (works at opt batch, faults everywhere else in range, clean on 10.9, OOB confirmed by compute-sanitizer inside __myl_Inor) lines up perfectly with a Myelin-side regression on the InstanceNormalization fusion under dynamic shapes. I’m pulling this in to file internally so the compiler team can take a look. I will follow up here with a tracking ID once it’s logged.

In the meantime, here are a few workarounds that should keep you unblocked without having to leave the DeepStream 9 container:

  1. Multiple optimization profiles, static per profile: Build the engine with separate profiles for the batch sizes you actually see (e.g., 1, 2, 4, 8, 16), each with min == opt == max. The tracker can select the matching profile at runtime, and since each profile is effectively static, the buggy Myelin dynamic-shape path never fires. This is usually the lowest-risk option for ReID since the per-frame batch distribution is fairly bounded.
  2. Decompose InstanceNormalization in the ONNX: Using onnx-graphsurgeon, replace the InstanceNormalization node with its primitive form (per-channel mean and variance via ReduceMean, normalize, then apply scale and bias). With the fused op gone, Myelin can’t form the problematic kernel. There might be a small throughput hit, but it routes around the bug on 10.14 today.
  3. InstanceNormalization plugin from TRT-OSS: If you’d rather keep the graph at the op level, swapping the standard op for the plugin node also bypasses the Myelin fusion.

One quick data point that would really help the bug report: if you can rebuild without --fp16 and let me know whether FP32 also crashes at non-opt batches, that will help narrow down whether the regression is in the InstanceNorm FP16 path specifically or the fusion in general.

Thanks for the thorough write-up. I’ll be trying to repro this on my end over the next couple of days as time allows.

Thanks,
Atharva

6

Hi Atharva,

Thanks for confirming this and for the workaround suggestions.

FP32 data point you asked for: I rebuilt the engine from the original osnet_ain_x1_0.onnx (5 InstanceNormalization nodes, dynamic batch 1–16, opt=16) on TRT 10.14.1.48, RTX 4060 Laptop (sm89):

Precision Batch 1 Batch 4 Batch 8 Batch 16 (opt)
FP16 CRASH (CUDA error 720 __myl_Inor) CRASH CRASH 203 qps / 5.7ms
FP32 760 qps / 1.4ms 340 qps / 3.2ms 179 qps / 6.1ms 80 qps / 13.4ms

FP32 passes at all batch sizes. The regression is FP16-specific.

Re: workarounds — we’re already unblocked. We replaced InstanceNormalization with LayerNormalization in the ONNX before filing this, similar to your Option 2.

Happy to run any additional tests if needed.

Thanks, Moishi