Jetson Orin NX 8GB hard reboots during TensorRT engine build on fresh JetPack 7.2.1 / DeepStream 9.1

Robotics & Edge Computing Jetson Systems Jetson Orin NX
,
1

I have a Jetson Orin NX 8GB that reproducibly hard reboots, with no kernel panic or orderly shutdown visible, while TensorRT is building an engine.

orin-nx-jp721-tensorrt-reboot-diagnostics.tar.gz (21.6 KB)

I have reproduced this on a completely fresh JetPack 7.2.1 installation using the stock TrafficCamNet ONNX model included with DeepStream 9.1.

Hardware

  • Jetson Orin NX 8GB
  • 256 GB NVMe
  • 12 V / 5 A LiteOn HA-1600-12 power supply
  • Current test is on the carrier board that supports Force Recovery and was used for the clean flash

I have also reproduced similar hard reboots on a second carrier board and with a second 12 V / 5 A power supply.

Software

I performed a clean Direct Flash using SDK Manager 2.4.1.13536, selecting JetPack 7.2.1 and DeepStream 9.1. I did not run Software Updater or apt upgrade afterward.

The freshly installed system reports:



Jetson Linux: R39.2.1
Kernel: 6.8.12-1021-tegra
CUDA: 13.2
TensorRT: 10.16.2
DeepStream: 9.1.0

The Orin NX is in the default 15 W power mode:



NV Power Mode: 15W
2

Minimal reproduction

I used the stock DeepStream files:



/opt/nvidia/deepstream/deepstream-9.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx
/opt/nvidia/deepstream/deepstream-9.1/samples/configs/deepstream-app/source2_1080p_dec_infer-resnet_demux.txt
/opt/nvidia/deepstream/deepstream-9.1/samples/configs/deepstream-app/config_infer_primary.txt

I copied the two config files into my home directory and changed the model, label and output paths to absolute/writable paths. No TensorRT engine existed initially.

I then ran:



cd /opt/nvidia/deepstream/deepstream-9.1/samples/configs/deepstream-app

deepstream-app -c \
  ~/workspace/deepstream-test/source2_1080p_dec_infer-resnet_demux.txt

DeepStream correctly determines that the engine does not exist and starts rebuilding it:



WARNING: Deserialize engine failed because file path:
.../resnet18_trafficcamnet_pruned.onnx_b2_gpu0_fp16.engine open error

NvDsInferContext[UID 1]: ... deserialize backend context ... failed, try rebuild

NvDsInferContext[UID 1]: ... Trying to create engine from model files

The system then spends roughly 30–60 seconds building the engine and hard reboots. There is no clean shutdown or application error—the SSH connection simply drops and the board boots again.

After reboot there is still no engine file:



$ ls -lh /opt/nvidia/deepstream/deepstream-9.1/samples/models/Primary_Detector/

cal_trt.bin
labels.txt
resnet18_trafficcamnet_pruned.onnx

System behavior

At idle, tegrastats looks normal. For example:



RAM 993/7546MB
CPU [0%@729,0%@729,0%@729,0%@729,off,off]
GR3D_FREQ 0%@[305]
gpu ~45C
VDD_IN ~4632mW

I have not observed an OOM or thermal condition preceding the resets.

The kernel does repeatedly emit messages of this form:



NVRM: GPU0 rpcRmApiControl_dce:
NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341
result 0xffff: Failure: Generic Error [NV_ERR_GENERIC]

Importantly, these messages occur on the newly flashed JetPack 7.2.1 installation before I install or build any additional models.

During earlier investigation on the previous installation I also reproduced hard reboots while building/running CitySemSegFormer and while building PeopleNet Transformer. However, I am mentioning those only as additional context; the stock DeepStream TrafficCamNet case above is the clean reproduction.

I also tested different 12 V / 5 A power supplies, 10 W and 15 W power modes, and two carrier boards. The failure followed the Orin NX module.

Diagnostics available

I have captured:



dmesg-current.txt
nvidia-packages.txt
nvpmodel.txt
nv_tegra_release.txt
uname.txt
config_infer_primary.txt
source2_1080p_dec_infer-resnet_demux.txt

Persistent journaling was not enabled on the fresh installation before the reboot, so unfortunately I do not have the previous boot’s kernel journal for this reproduction.

Is this a known R39.2.1 / TensorRT 10.16.2 issue on Orin NX 8GB? Are there additional firmware/GPU diagnostics I should capture, or a recommended way to obtain crash information for a hard reset occurring during TensorRT engine compilation?

3

I reduced the reproducer further and got a new result.

Running the stock TrafficCamNet ONNX directly with trtexec at the default builder optimization level causes the Orin NX to hard reboot.

However, with:



trtexec \
  --onnx=/opt/nvidia/deepstream/deepstream-9.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx \
  --fp16 \
  --skipInference \
  --builderOptimizationLevel=0 \
  --maxAuxStreams=0 \
  --saveEngine=test_opt0.engine

the system does not reboot. Instead TensorRT exits with:



Linkable shader sm80_xmma_fprop_implicit_gemm_f16f16_f16f16_f16_nhwckrsc_nhwc_tilesize64x256x64_stage3_warpsize1x4x1_g1_tensor16x8x16_simple_t1r1s1_linkable
doesn't have base shader

Error[1]: Unexpected exception KTM assertion failure:
/_src/externals/ktm/src/caskTimingModels/convolutionTimingModel.cpp:65
shader != nullptr

Created engine with size: 0 MiB
Assertion failure: false && "Attempting to access an empty engine!"

This is on a clean JetPack 7.2.1 / TensorRT 10.16.2 installation using NVIDIA’s stock DeepStream TrafficCamNet ONNX.

Does this indicate a known TensorRT 10.16.2 tactic/shader database issue on Orin NX?

4

Standalone trtexec FP32 build of the same stock TrafficCamNet ONNX succeeds on the same system with --builderOptimizationLevel=0 --maxAuxStreams=0, producing a 9.11 MiB engine. The corresponding FP16 build fails with the KTM shader assertion, and FP16 with the default builder search has caused hard reboots. This appears to isolate the problem to the FP16 TensorRT build/tactic path rather than general TensorRT engine construction.

5

I also tested FP16 with --builderOptimizationLevel=0 --maxAuxStreams=0 while disabling the optional convolution tactic sources:



--tacticSources=-JIT_CONVOLUTIONS,-EDGE_MASK_CONVOLUTIONS

The build still fails with exactly the same error:



Linkable shader ... doesn't have base shader ...
KTM assertion failure:
/_src/externals/ktm/src/caskTimingModels/convolutionTimingModel.cpp:65
shader != nullptr

I also tested --maxTactics=1 with the same result. FP32 on the same ONNX still builds successfully. This suggests the failure is in an internal FP16 TensorRT convolution/timing-model path rather than the optional JIT or edge-mask tactic sources.I also tested FP16 with --builderOptimizationLevel=0 --maxAuxStreams=0 while disabling the optional convolution tactic sources:



--tacticSources=-JIT_CONVOLUTIONS,-EDGE_MASK_CONVOLUTIONS

The build still fails with exactly the same error:



Linkable shader ... doesn't have base shader ...
KTM assertion failure:
/_src/externals/ktm/src/caskTimingModels/convolutionTimingModel.cpp:65
shader != nullptr

I also tested --maxTactics=1 with the same result. FP32 on the same ONNX still builds successfully. This suggests the failure is in an internal FP16 TensorRT convolution/timing-model path rather than the optional JIT or edge-mask tactic sources.

6

I reproduced the same issue on a second device: a brand-new Jetson Orin Nano Developer Kit 8GB, freshly flashed with JetPack 7.2.1 / TensorRT 10.16.2.

Using the exact same resnet18_trafficcamnet_pruned.onnx:

  • FP32 with --builderOptimizationLevel=0 --maxAuxStreams=0 builds successfully.
  • FP16 with the same settings fails with the exact same KTM assertion:


Linkable shader ... doesn't have base shader ...
Error[1]: Unexpected exception KTM assertion failure:
/_src/externals/ktm/src/caskTimingModels/convolutionTimingModel.cpp:65
shader != nullptr

This reproduces identically on both:

  • Jetson Orin NX 8GB
  • Jetson Orin Nano 8GB Developer Kit

Both are on JetPack 7.2.1 / TensorRT 10.16.2.

That seems to rule out a hardware fault specific to the NX module and points to a TensorRT 10.16.2 FP16 builder issue on Orin.

7

I repeated the default FP16 trtexec build on a brand-new Jetson Orin Nano 8GB, freshly flashed with the same JetPack 7.2.1 / TensorRT 10.16.2 stack and using the exact same TrafficCamNet ONNX.

Result:



Engine generation completed in 69.9489 seconds.
Created engine with size: 3.04167 MiB
Engine built in 70.9736 sec.
&&&& PASSED TensorRT.trtexec

So the comparison is now:



Orin NX 8GB:
  FP32 opt0      PASS
  FP16 opt0      KTM shader assertion
  FP16 default   HARD REBOOT

Orin Nano 8GB:
  FP32 opt0      PASS
  FP16 opt0      same KTM shader assertion
  FP16 default   PASS

This suggests the builderOptimizationLevel=0 FP16 assertion is common to both platforms, while the hard reboot under the normal FP16 builder path appears specific to the NX or to an NX-specific software/hardware interaction.

8

Hi,

We are going to reproduce the OrinNX 8GB issue.
Will update with more information with you.

If possible, could you also help us to check if you can convert the same model into a TensorRT engine with JetPack 6.2.2?

Thanks.

9

Hey, send me the exact command you normally use to start llama-server.

I can send you a slightly different way to run it that should make it obvious when CUDA dies and it silently falls back to CPU.

10

Hi,

Does Orin NX 8GB also hard reboot or fail with a rate?

We try the following TensorRT command three times:

$ trtexec --onnx=/opt/nvidia/deepstream/deepstream-9.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx --fp16 --skipInference --saveEngine=test_opt0.engine

All passed and the system is alive.
Our device is set up with 15W (id=2) mode and without locking clocks with jetson_clocks.

Thanks.

12

I completed the requested JetPack 6.2.2 A/B test on the same Jetson Orin NX 8GB module.

Environment:

  • JetPack 6.2.2 / L4T R36.5.0
  • Kernel 5.15.185-tegra
  • TensorRT 10.3.0
  • DeepStream 7.1
  • 15 W mode, ID 2
  • Four SMs visible to TensorRT
  • No jetson_clocks
  • Same 12 V / 5 A supply and NX module

Stock TrafficCamNet result

I ran the requested default FP16 command three times using the stock DeepStream 7.1 ONNX:

/usr/src/tensorrt/bin/trtexec \
  --onnx=/opt/nvidia/deepstream/deepstream-7.1/samples/models/Primary_Detector/resnet18_trafficcamnet_pruned.onnx \
  --fp16 \
  --skipInference \
  --saveEngine=<fresh-engine-path>

ONNX SHA-256:

d92f97bd840b68850a33c83e480fc6bd83b3097ff4cb517e0ea12671046bab7d

Results:

  • Run 1: PASS, 131.812 seconds
  • Run 2: PASS, 131.795 seconds
  • Run 3: PASS, 137.149 seconds

The system remained alive, and no matching Xid, NVRM, machine-check, PBDMA, watchdog, thermal-throttle, or device-not-ready message appeared during these builds.

One resulting engine also completed a 120-second inference run:

  • 120.335 inferences/s
  • 8.734 ms mean host latency
  • 8.744 ms p95 host latency
  • 13.790 W mean VDD_IN
  • 13.888 W peak VDD_IN
  • 60.3 C maximum reported temperature
  • no post-run kernel fault

Therefore, the original stock TrafficCamNet default-FP16 reboot does not reproduce on JetPack 6.2.2.

YOLO26 INT8 result

I also repeated the more demanding model that had exposed additional failures on 7.2.1.

The model is an explicit-Q/DQ INT8 YOLO26n ONNX:

e0304e71736f29f7550831cee82a58b07e98a9dc07ca2a95c2454377b813acec

A fixed-batch engine built successfully with:

--stronglyTyped
--builderOptimizationLevel=0
--maxAuxStreams=0
--memPoolSize=workspace:1024

It then completed three independent 60-second inference processes. All three returned status zero, delivered approximately 28.8 inferences/s, and produced no kernel fault.

However, changing only:

--builderOptimizationLevel=0

to:

--builderOptimizationLevel=5

caused another hard reboot during the compiler/tactic phase.

The build started at 13:31:45 and telemetry stopped abruptly at 13:33:31. It produced no engine, exit-status file, completion timestamp, or TensorRT error. The next SSH connection showed uptime below one minute.

Immediately before the reset:

  • peak sampled VDD_IN: 8.737 W
  • maximum reported temperature: 53.625 C
  • RAM and swap were not exhausted

The fresh boot again logged:

NVRM_RPC_DCE: Failed RM ctrl call cmd:0x731341 result 0xffff
NVRM_RPC_DCE: Failed RM ctrl call cmd:0x730282 result 0x1f

A conservative CUDA inference ran successfully after the reboot.

A 90-second DeepStream 7.1 single-stream smoke test also passed using hardware decode, the optimization-level-0 INT8 engine, custom parser, and fake sink. It exited cleanly at 27.67 FPS with no kernel fault.

Interpretation

JetPack 6.2.2 fixes or avoids the original stock TrafficCamNet default-FP16 reboot, but it does not eliminate the hard reset under the YOLO26 optimization-level-5 TensorRT builder workload.

Because the high-optimization reset now reproduces on both JetPack 7.2.1 and 6.2.2, this no longer appears to be solely an R39.2.1/TensorRT 10.16.2 regression. It may be a module-specific hardware problem or a lower-level firmware/platform interaction triggered by the compiler workload.

Would you recommend additional serial-console or firmware diagnostics, or does this cross-version result justify testing/replacing the NX module?