I want to ask NVIDIA dev a question:
Start from Jetpack version 6.1, I cannot re-use TensorRT models between inside a docker container and outside on-device. Both environments have same version of TensorRT. Which of reasons are for this change? It is pretty inconvenient.
Hi,
Do you have logs or errors that can share with us?
This helps us to know more about your issue.
Thanks.
Hi @AastaLLL
This is log when I run a repo inside docker container with TensorRT 10.3.0:
root@b3e5458de517:/app# ./build/app
[2025-08-26 10:33:59.019] [info] This main function!
[2025-08-26 10:33:59.020] [info] Source path: src/configs/input_sources.json
[2025-08-26 10:33:59.021] [info] Number of sources: 1
[2025-08-26 10:33:59.130] [info] Loader::LoadVideo -> Open video file materials/input/video/5_stay_persons.mp4
[2025-08-26 10:33:59.131] [info] DetectorFactory::create_detector -> YoloV5
[2025-08-26 10:33:59.131] [info] Loading engine from file: materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine
[2025-08-26 10:33:59.428] [warning] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[2025-08-26 10:33:59.580] [info] TrackManager::Tracker -> Init OCSort Tracker of Class 0
[2025-08-26 10:33:59.581] [info] TrackManager::Tracker -> Done Init OCSort Tracker of Class 0
[2025-08-26 10:33:59.581] [info] DirectionFactory::create_direction -> Rankpose
[2025-08-26 10:33:59.581] [info] Loading engine from file: materials/weights/direction/rankpose_v2.0_mnv3s_96x96_fp32.engine
[2025-08-26 10:33:59.592] [warning] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[2025-08-26 10:33:59.622] [info] AgmFactory::create_agm -> MobilenetV2
[2025-08-26 10:33:59.622] [info] Loading engine from file: materials/weights/agm/ver79_39_WITH_PADDING_generalmodel.engine
[2025-08-26 10:33:59.699] [warning] Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
As soon as that, I run it directly on the same device (same version TensorRT) and this is log:
giapvn@ubuntu:/app$ sudo ./build/app
[2025-08-26 17:38:17.052] [info] This main function!
[2025-08-26 17:38:17.052] [info] Source path: src/configs/input_sources.json
[2025-08-26 17:38:17.052] [info] Number of sources: 1
[2025-08-26 17:38:17.075] [info] Loader::LoadVideo -> Open video file materials/input/video/5_stay_persons.mp4
[2025-08-26 17:38:17.075] [info] DetectorFactory::create_detector -> YoloV5
[2025-08-26 17:38:17.075] [info] Loading engine from file: materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine
[2025-08-26 17:38:17.290] [warning] IRuntime::deserializeCudaEngine: Error Code 6: API Usage Error (The engine plan file is not compatible with this version of TensorRT, expecting library version 10.3.0.30 got
..)
Segmentation fault
It will work well if I re-convert model directly on the device instead of inside a docker container.
Hi,
Could you double-confirm with the trtexec?
/usr/src/tensorrt/bin/trtexec --loadEngine=[engine] --getPlanVersionOnly
Thanks.
@AastaLLL Yes. It is without docker container:
sudo /usr/src/tensorrt/bin/trtexec --loadEngine=materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine --getPlanVersionOnly
&&&& RUNNING TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --loadEngine=materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine --getPlanVersionOnly
[08/28/2025-14:08:56] [I] === Model Options ===
[08/28/2025-14:08:56] [I] Format: *
[08/28/2025-14:08:56] [I] Model:
[08/28/2025-14:08:56] [I] Output:
[08/28/2025-14:08:56] [I]
[08/28/2025-14:08:56] [I] === System Options ===
[08/28/2025-14:08:56] [I] Device: 0
[08/28/2025-14:08:56] [I] DLACore:
[08/28/2025-14:08:56] [I] Plugins:
[08/28/2025-14:08:56] [I] setPluginsToSerialize:
[08/28/2025-14:08:56] [I] dynamicPlugins:
[08/28/2025-14:08:56] [I] ignoreParsedPluginLibs: 0
[08/28/2025-14:08:56] [I]
[08/28/2025-14:08:56] [I] === Inference Options ===
[08/28/2025-14:08:56] [I] Batch: Explicit
[08/28/2025-14:08:56] [I] Input inference shapes: model
[08/28/2025-14:08:56] [I] Iterations: 10
[08/28/2025-14:08:56] [I] Duration: 3s (+ 200ms warm up)
[08/28/2025-14:08:56] [I] Sleep time: 0ms
[08/28/2025-14:08:56] [I] Idle time: 0ms
[08/28/2025-14:08:56] [I] Inference Streams: 1
[08/28/2025-14:08:56] [I] ExposeDMA: Disabled
[08/28/2025-14:08:56] [I] Data transfers: Enabled
[08/28/2025-14:08:56] [I] Spin-wait: Disabled
[08/28/2025-14:08:56] [I] Multithreading: Disabled
[08/28/2025-14:08:56] [I] CUDA Graph: Disabled
[08/28/2025-14:08:56] [I] Separate profiling: Disabled
[08/28/2025-14:08:56] [I] Time Deserialize: Disabled
[08/28/2025-14:08:56] [I] Time Refit: Disabled
[08/28/2025-14:08:56] [I] NVTX verbosity: 0
[08/28/2025-14:08:56] [I] Persistent Cache Ratio: 0
[08/28/2025-14:08:56] [I] Optimization Profile Index: 0
[08/28/2025-14:08:56] [I] Weight Streaming Budget: 100.000000%
[08/28/2025-14:08:56] [I] Inputs:
[08/28/2025-14:08:56] [I] Debug Tensor Save Destinations:
[08/28/2025-14:08:56] [I] === Reporting Options ===
[08/28/2025-14:08:56] [I] Verbose: Disabled
[08/28/2025-14:08:56] [I] Averages: 10 inferences
[08/28/2025-14:08:56] [I] Percentiles: 90,95,99
[08/28/2025-14:08:56] [I] Dump refittable layers:Disabled
[08/28/2025-14:08:56] [I] Dump output: Disabled
[08/28/2025-14:08:56] [I] Profile: Disabled
[08/28/2025-14:08:56] [I] Export timing to JSON file:
[08/28/2025-14:08:56] [I] Export output to JSON file:
[08/28/2025-14:08:56] [I] Export profile to JSON file:
[08/28/2025-14:08:56] [I]
[08/28/2025-14:08:56] [I] === Device Information ===
[08/28/2025-14:08:56] [I] Available Devices:
[08/28/2025-14:08:56] [I] Device 0: "Orin" UUID: GPU-b9e50b48-752d-567b-bcb2-c4ab32da3bb9
[08/28/2025-14:08:57] [I] Selected Device: Orin
[08/28/2025-14:08:57] [I] Selected Device ID: 0
[08/28/2025-14:08:57] [I] Selected Device UUID: GPU-b9e50b48-752d-567b-bcb2-c4ab32da3bb9
[08/28/2025-14:08:57] [I] Compute Capability: 8.7
[08/28/2025-14:08:57] [I] SMs: 16
[08/28/2025-14:08:57] [I] Device Global Memory: 62840 MiB
[08/28/2025-14:08:57] [I] Shared Memory per SM: 164 KiB
[08/28/2025-14:08:57] [I] Memory Bus Width: 256 bits (ECC disabled)
[08/28/2025-14:08:57] [I] Application Compute Clock Rate: 1.3 GHz
[08/28/2025-14:08:57] [I] Application Memory Clock Rate: 1.3 GHz
[08/28/2025-14:08:57] [I]
[08/28/2025-14:08:57] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[08/28/2025-14:08:57] [I]
[08/28/2025-14:08:57] [I] TensorRT version: 10.3.0
[08/28/2025-14:08:57] [I] Loading standard plugins
[08/28/2025-14:08:57] [I] Plan was created with TensorRT version 10.3.0.26
&&&& PASSED TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --loadEngine=materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine --getPlanVersionOnly
And this is within docker container:
root@b3e5458de517:/app# /usr/src/tensorrt/bin/trtexec --loadEngine=materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine --getPlanVersionOnly
&&&& RUNNING TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --loadEngine=materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine --getPlanVersionOnly
[08/28/2025-07:12:09] [I] === Model Options ===
[08/28/2025-07:12:09] [I] Format: *
[08/28/2025-07:12:09] [I] Model:
[08/28/2025-07:12:09] [I] Output:
[08/28/2025-07:12:09] [I]
[08/28/2025-07:12:09] [I] === System Options ===
[08/28/2025-07:12:09] [I] Device: 0
[08/28/2025-07:12:09] [I] DLACore:
[08/28/2025-07:12:09] [I] Plugins:
[08/28/2025-07:12:09] [I] setPluginsToSerialize:
[08/28/2025-07:12:09] [I] dynamicPlugins:
[08/28/2025-07:12:09] [I] ignoreParsedPluginLibs: 0
[08/28/2025-07:12:09] [I]
[08/28/2025-07:12:09] [I] === Inference Options ===
[08/28/2025-07:12:09] [I] Batch: Explicit
[08/28/2025-07:12:09] [I] Input inference shapes: model
[08/28/2025-07:12:09] [I] Iterations: 10
[08/28/2025-07:12:09] [I] Duration: 3s (+ 200ms warm up)
[08/28/2025-07:12:09] [I] Sleep time: 0ms
[08/28/2025-07:12:09] [I] Idle time: 0ms
[08/28/2025-07:12:09] [I] Inference Streams: 1
[08/28/2025-07:12:09] [I] ExposeDMA: Disabled
[08/28/2025-07:12:09] [I] Data transfers: Enabled
[08/28/2025-07:12:09] [I] Spin-wait: Disabled
[08/28/2025-07:12:09] [I] Multithreading: Disabled
[08/28/2025-07:12:09] [I] CUDA Graph: Disabled
[08/28/2025-07:12:09] [I] Separate profiling: Disabled
[08/28/2025-07:12:09] [I] Time Deserialize: Disabled
[08/28/2025-07:12:09] [I] Time Refit: Disabled
[08/28/2025-07:12:09] [I] NVTX verbosity: 0
[08/28/2025-07:12:09] [I] Persistent Cache Ratio: 0
[08/28/2025-07:12:09] [I] Optimization Profile Index: 0
[08/28/2025-07:12:09] [I] Weight Streaming Budget: 100.000000%
[08/28/2025-07:12:09] [I] Inputs:
[08/28/2025-07:12:09] [I] Debug Tensor Save Destinations:
[08/28/2025-07:12:09] [I] === Reporting Options ===
[08/28/2025-07:12:09] [I] Verbose: Disabled
[08/28/2025-07:12:09] [I] Averages: 10 inferences
[08/28/2025-07:12:09] [I] Percentiles: 90,95,99
[08/28/2025-07:12:09] [I] Dump refittable layers:Disabled
[08/28/2025-07:12:09] [I] Dump output: Disabled
[08/28/2025-07:12:09] [I] Profile: Disabled
[08/28/2025-07:12:09] [I] Export timing to JSON file:
[08/28/2025-07:12:09] [I] Export output to JSON file:
[08/28/2025-07:12:09] [I] Export profile to JSON file:
[08/28/2025-07:12:09] [I]
[08/28/2025-07:12:09] [I] === Device Information ===
[08/28/2025-07:12:09] [I] Available Devices:
[08/28/2025-07:12:09] [I] Device 0: "Orin" UUID: GPU-b9e50b48-752d-567b-bcb2-c4ab32da3bb9
[08/28/2025-07:12:09] [I] Selected Device: Orin
[08/28/2025-07:12:09] [I] Selected Device ID: 0
[08/28/2025-07:12:09] [I] Selected Device UUID: GPU-b9e50b48-752d-567b-bcb2-c4ab32da3bb9
[08/28/2025-07:12:09] [I] Compute Capability: 8.7
[08/28/2025-07:12:09] [I] SMs: 16
[08/28/2025-07:12:09] [I] Device Global Memory: 62840 MiB
[08/28/2025-07:12:09] [I] Shared Memory per SM: 164 KiB
[08/28/2025-07:12:09] [I] Memory Bus Width: 256 bits (ECC disabled)
[08/28/2025-07:12:09] [I] Application Compute Clock Rate: 1.3 GHz
[08/28/2025-07:12:09] [I] Application Memory Clock Rate: 1.3 GHz
[08/28/2025-07:12:09] [I]
[08/28/2025-07:12:09] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[08/28/2025-07:12:09] [I]
[08/28/2025-07:12:09] [I] TensorRT version: 10.3.0
[08/28/2025-07:12:09] [I] Loading standard plugins
[08/28/2025-07:12:10] [I] Plan was created with TensorRT version 10.3.0.26
&&&& PASSED TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --loadEngine=materials/weights/detection/yolov5s-pfg_dynamic_fp32.engine --getPlanVersionOnly
Hi,
Thanks a lot for the information.
We want to check this issue further.
Could you share which container you are using? Do you the l4t-tensorrt container below?
More, how do you generate the engine?
Could we reproduce this issue with a public model or is this model-specific?
Thanks.
No, I pulled the image from nvcr.io/nvidia/l4t-jetpack:r36.4.0
The below is my dockerfile used to build my image
ARG BASE_IMAGE=nvcr.io/nvidia/l4t-jetpack:r36.4.0
FROM ${BASE_IMAGE}
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Japan
ARG LIB_PREFIX=/usr/local
ENV PATH="/usr/local/cuda-12.6/bin:${PATH}"
ENV LD_LIBRARY_PATH="${LIB_PREFIX}/lib/:/usr/lib/aarch64-linux-gnu/tegra/:${LD_LIBRARY_PATH}"
RUN rm /etc/apt/sources.list.d/cuda.list || true
RUN apt-get update --fix-missing && \
apt-get install -y --no-install-recommends --allow-change-held-packages \
build-essential pkg-config rsyslog tzdata yasm wget git curl \
nano python3-pip python3-dev libopenblas-dev qtbase5-dev uuid-dev \
libjson-glib-dev libjson-glib-1.0-0 \
libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libgstreamer-plugins-good1.0-dev libgstreamer-plugins-bad1.0-dev \
libboost-all-dev libgstrtspserver-1.0-dev
RUN apt-get install -y libavcodec-dev libavformat-dev libswscale-dev uuid-dev zlib1g-dev openssl
RUN apt-get install -y libyaml-cpp-dev tesseract-ocr libtesseract-dev lsb-release
RUN apt-get install -y nvidia-cuda-dev
RUN apt-get clean && rm -rf /var/lib/apt/lists/*
#RUN ln -s /usr/lib/aarch64-linux-gnu/libgstrtspserver-1.0.so.0 /usr/lib/aarch64-linux-gnu/libgstrtspserver-1.0.so
RUN ln -sf /usr/bin/python3 /usr/bin/python
RUN python3 -m pip install --upgrade pip
RUN pip install numpy torch torchvision matplotlib
ENV CMAKE_VERSION=3.26.0
RUN wget https://github.com/Kitware/CMake/releases/download/v${CMAKE_VERSION}/cmake-${CMAKE_VERSION}-linux-aarch64.tar.gz
RUN tar -xf cmake-${CMAKE_VERSION}-linux-aarch64.tar.gz -C /usr/local --strip-components=1 --no-same-owner \
&& rm -rf cmake-${CMAKE_VERSION}*
RUN wget https://developer.download.nvidia.com/compute/cudnn/9.3.0/local_installers/cudnn-local-tegra-repo-ubuntu2204-9.3.0_1.0-1_arm64.deb && \
dpkg -i cudnn-local-tegra-repo-ubuntu2204-9.3.0_1.0-1_arm64.deb && \
cp /var/cudnn-local-*/cudnn-*-keyring.gpg /usr/share/keyrings/ && \
apt-get update && rm cudnn-local-tegra-repo-ubuntu2204-9.3.0_1.0-1_arm64.deb && \
apt-get -y install cudnn9-cuda-12
#
# Install nvidia-tensorrt-dev for TensorRT developer packages
# Use nvidia-tensorrt if need TensorRT runtime only
#
RUN apt-get update && apt-get install -y --no-install-recommends \
nvidia-tensorrt-dev \
&& rm -rf /var/lib/apt/lists/* \
&& apt-get clean
# Update libraries
RUN ldconfig
#
# Setup environment variables
#
ENV CUDA_HOME="/usr/local/cuda"
ENV LD_LIBRARY_PATH="/usr/local/cuda/lib64:${LD_LIBRARY_PATH}"
ENV NVIDIA_VISIBLE_DEVICES=all
ENV NVIDIA_DRIVER_CAPABILITIES=all
RUN apt --fix-broken install
WORKDIR /app
I use a .onnx model and then convert it to TensorRT engine by using command-line:
/usr/src/tensorrt/bin/trtexec --onnx=materials/weights/detection/yolov5s_640x384_pfg.onnx --saveEngine=materials/weights/detection/yolov5s_640x384_pfg.engine --minShapes=images:1x3x384x640 --optShapes=images:2x3x384x640 --maxShapes=images:4x3x384x640 --memPoolSize=workspace:6400 --tacticSources=-cublasLt,+cublas --sparsity=disable --verbose
You can use a public model to re-produce this issue.
Hi,
Thanks for sharing the details.
We will try to reproduce this issue and update.
Thanks.
Hi,
Thanks for your patience.
We are not able to reproduce this issue locally.
The engine built with the same TensorRT version can work without re-generating.
Below are our steps:
1. Generate the TensorRT engine
Convert the model inside the nvcr.io/nvidia/l4t-jetpack:r36.4.0 container:
$ sudo docker run -it --rm --runtime nvidia nvcr.io/nvidia/l4t-jetpack:r36.4.0
# /usr/src/tensorrt/bin/trtexec --onnx=./ResNet50.onnx --saveEngine=ResNet50.engine --memPoolSize=workspace:6400 --tacticSources=-cublasLt,+cublas --sparsity=disable
2. Copy the engine file outside of the container
3. Deploy with trtexec
The model can work without issue.
$ /usr/src/tensorrt/bin/trtexec --loadEngine=ResNet50.engine --memPoolSize=workspace:6400 --tacticSources=-cublasLt,+cublas --sparsity=disable
&&&& RUNNING TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --loadEngine=ResNet50.engine --memPoolSize=workspace:6400 --tacticSources=-cublasLt,+cublas --sparsity=disable
...
[09/18/2025-05:29:48] [I] === Performance summary ===
[09/18/2025-05:29:48] [I] Throughput: 547.105 qps
[09/18/2025-05:29:48] [I] Latency: min = 1.85333 ms, max = 1.91309 ms, mean = 1.87272 ms, median = 1.87238 ms, percentile(90%) = 1.88086 ms, percentile(95%) = 1.8844 ms, percentile(99%) = 1.89349 ms
[09/18/2025-05:29:48] [I] Enqueue Time: min = 0.342407 ms, max = 1.41345 ms, mean = 0.827923 ms, median = 0.842087 ms, percentile(90%) = 0.894043 ms, percentile(95%) = 0.914917 ms, percentile(99%) = 0.962555 ms
[09/18/2025-05:29:48] [I] H2D Latency: min = 0.0292969 ms, max = 0.0837402 ms, mean = 0.0431653 ms, median = 0.0429993 ms, percentile(90%) = 0.0491943 ms, percentile(95%) = 0.0515747 ms, percentile(99%) = 0.0622559 ms
[09/18/2025-05:29:48] [I] GPU Compute Time: min = 1.81036 ms, max = 1.84882 ms, mean = 1.82411 ms, median = 1.82373 ms, percentile(90%) = 1.8291 ms, percentile(95%) = 1.83081 ms, percentile(99%) = 1.83661 ms
[09/18/2025-05:29:48] [I] D2H Latency: min = 0.00317383 ms, max = 0.00772095 ms, mean = 0.00544371 ms, median = 0.00537109 ms, percentile(90%) = 0.0065918 ms, percentile(95%) = 0.00683594 ms, percentile(99%) = 0.00720215 ms
[09/18/2025-05:29:48] [I] Total Host Walltime: 3.00491 s
[09/18/2025-05:29:48] [I] Total GPU Compute Time: 2.99884 s
[09/18/2025-05:29:48] [I] Explanations of the performance metrics are printed in the verbose logs.
[09/18/2025-05:29:48] [I]
&&&& PASSED TensorRT.trtexec [TensorRT v100300] # /usr/src/tensorrt/bin/trtexec --loadEngine=ResNet50.engine --memPoolSize=workspace:6400 --tacticSources=-cublasLt,+cublas --sparsity=disable
It looks like you have reinstalled some CUDA libraries on top of the l4t-jetpack container, which might cause the issue.
Could you try it with the l4t-jetpack image directly?
Thanks.
Thanks @AastaLLL
Did you test on Jetpack 6.1?
As in my Dockerfile above, I see that I installed nvidia-cuda-dev package. It is only CUDA libraries I re-installed. I tried to remove installing this package when building docker image but it still not work right on Jetpack 6.1 and 6.2.
However, I tested on Jetpack 6.2.1 and it is ok even if I reinstalled nvidia-cuda-dev package. I think it is fixed in the latest Jetpack.