VI ATOMP_FRAME_TRUNCATED Error When Running DLA-Based ONNX Model on Orin-X

[zaiwei]

DRIVE OS Version: 6.0.10

Issue Description: VI ATOMP_FRAME_TRUNCATED Error When Running DLA-Based ONNX Model on Orin-X

HI, NVIDIA

We are running an ONNX model on the Orin platform. The model executes a specific FP16 high-channel 3x3 convolution workload and uses the DLA. We found that this triggers a VI error with the code ATOMP_FRAME_TRUNCATED.

 /usr/src/tensorrt/bin/trtexec \
     --onnx=models/special_vehicle_classifier/special_vehicle_classifier.onnx \
     --int8 \
     --fp16 \
     --calib=models/special_vehicle_classifier/calibration_table \
     --useDLACore=1 \
     --allowGPUFallback \
     --shapes=image_view:40x3x96x96,features:40x14x1x1,projected_lidar_view:40x3x96x96 \
     --duration=3600 \
     --warmUp=5000

/**
 * Frame truncated writing to system memory. Maps to VI
 * ATOMP_FRAME_TRUNCATED event. This error typically indicates
 * memory back-pressure due to insufficient memory bandwidth
 * or an excessive memory access latency.
 *
 * This error sets @ref CAPTURE_STATUS_NOTIFY_BIT_ATOMP_FRAME_TRUNCATED
 * in @ref notify_bits.
 */
#define CAPTURE_STATUS_ATOMP_FRAME_TRUNCATED	MK_U32(9)

Based on the comment, this error seems to be related to memory bandwidth or memory access latency. How should we understand this issue systematically? Are there any recommended solutions?

Thanks in advance.
Best regards,

zaiwei

[SivaRamaKrishnaNV]

Dear @zaiwei ,
Is this error encountered when running trtexec ? Could you share the log?

[zaiwei]

trtexec is just our minimal offline reproduction solution. The initial discovery occurred during application runtime. Logs and reproduction methods are provided in the attachment. Thank you.

mre_script_nsys_20260527_202817.zip (79.0 MB)

[zaiwei]

Hi SivaRamaKrishna,

Any update for this issue?

[SivaRamaKrishnaNV]

Dear @zaiwei ,
From my understading, You are running nvsipl_camera in parallel to trtexec. trtexec uses DLA with GPU fallback model enabled.
Once you hit the issue you don’t see camera frames in nvsipl_camera further or you notice few framedrops till trtexec runs?
Does the model layer fallback to GPU? Do you notice same error when using DLA core 0? How frequently you notice this issue?

[zaiwei]

@SivaRamaKrishnaNV

From my understading, You are running nvsipl_camera in parallel to trtexec. trtexec uses DLA with GPU fallback model enabled.
Once you hit the issue you don’t see camera frames in nvsipl_camera further or you notice few framedrops till trtexec runs?
Does the model layer fallback to GPU? Do you notice same error when using DLA core 0? How frequently you notice this issue?

1. The current model cannot disable GPU fallback.

2. While trtexec runs with the DLA1, sensor0–3 and sensor8–11 experience a gradual FPS drop down to 0 FPS. Once trtexec finishes, all sensors recover to normal frame rates.

3. Regarding the model layer, please refer to the attachment.

4. We do not reproduce this problem when using DLA core 0.

[zaiwei]

Hi SivaRamaKrishna,

Any update for this nvbug?

[SivaRamaKrishnaNV]

Dear @zaiwei ,
Note that there are no releases for orin platform in devzone. Is it ok to use DLA core 0 for inference for GPU fallback?
May I know if you have any other supported channel.

[carolyuu]

Could you please provide any update for this topic?

[zaiwei]

Yes, we plan to use DLA core 0 for inference.