# Inference error with custom train

**URL:** <https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846>\
**Category:** TAO Toolkit\
**Tags:** tao, deepstream\
**Created:** [October 24, 2025, 1:48pm UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846 "2025-10-24T13:48:35Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![henrique.cerqueira](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@henrique.cerqueira](https://forums.developer.nvidia.com/u/henrique.cerqueira)\
**Post date:** [October 24, 2025, 1:48pm UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/1 "2025-10-24T13:48:36Z")

</div>

OS: Linux 24.04

Docker container: [nvcr.io/nvidia/deepstream:8.0-gc-triton-devel](http://nvcr.io/nvidia/deepstream:8.0-gc-triton-devel)

Tao Version:

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/1/d/2/1d24eeb3d1648045f900389ff33c44c08ffa66e5.png)

i’m run efficientdet notebook and train a custom dataset, but on the moment run inference returns a errors in two situation:

First - Running the .engine generated for TAO:  
gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_nvmultiobjecttracker.so  
[NvMultiObjectTracker] Initialized  
Stack trace (most recent call last) in thread 176:  
#31 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898191fc87, in gst\_element\_change\_state  
#30 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898191fc43, in gst\_element\_change\_state  
#29 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898194b43e, in  
#28 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e89818f75f7, in  
#27 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e89819204f8, in  
#26 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898191fc43, in gst\_element\_change\_state  
#25 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e8981920b22, in  
#24 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e89819209ad, in  
#23 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e89819a245d, in  
#22 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898192ccb4, in gst\_iterator\_fold  
#21 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898192084a, in  
#20 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898193bcb4, in gst\_pad\_set\_active  
#19 Object “/usr/lib/x86\_64-linux-gnu/libgstreamer-1.0.so.0”, at 0x7e898193b2ef, in  
#18 Object “/usr/lib/x86\_64-linux-gnu/libgstbase-1.0.so.0”, at 0x7e89818758b6, in  
#17 Object “/usr/lib/x86\_64-linux-gnu/libgstbase-1.0.so.0”, at 0x7e89818756a3, in  
#16 Object “/usr/lib/x86\_64-linux-gnu/gstreamer-1.0/deepstream/libnvdsgst\_infer.so”, at 0x7e899b4a2e15, in  
#15 Object “///opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_infer.so”, at 0x7e899b15eb7d, in createNvDsInferContext(INvDsInferContext\*\*, \_NvDsInferContextInitParams&, void\*, void (_)(INvDsInferContext_, unsigned int, NvDsInferLogLevel, char const\*, void\*))  
#14 Object “///opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_infer.so”, at 0x7e899b15e141, in nvdsinfer::NvDsInferContextImpl::initialize(\_NvDsInferContextInitParams&, void\*, void (_)(INvDsInferContext_, unsigned int, NvDsInferLogLevel, char const\*, void\*))  
#13 Object “///opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_infer.so”, at 0x7e899b15613d, in nvdsinfer::NvDsInferContextImpl::generateBackendContext(\_NvDsInferContextInitParams&)  
#12 Object “///opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_infer.so”, at 0x7e899b15541c, in nvdsinfer::NvDsInferContextImpl::deserializeEngineAndBackend(std::\_\_cxx11::basic\_string\<char, std::char\_traits, std::allocator \>, int, std::shared\_ptrnvdsinfer::TrtEngine&, std::unique\_ptr\<nvdsinfer::BackendContext, std::default\_deletenvdsinfer::BackendContext \>&)  
#11 Object “///opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_infer.so”, at 0x7e899b182dd8, in  
#10 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e8939bec4cd, in  
#9 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e8939bec435, in  
#8 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e8939beb9f2, in  
#7 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e8939bb4b6d, in  
#6 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e8939bb43eb, in  
#5 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e8939bb404c, in  
#4 Object “/usr/lib/x86\_64-linux-gnu/libnvinfer.so.10”, at 0x7e893929712f, in  
#3 Object “/usr/lib/x86\_64-linux-gnu/libunwind.so.8”, at 0x7e89829151ac, in \_\_libunwind\_Unwind\_Resume  
#2 Object “/usr/lib/x86\_64-linux-gnu/libstdc++.so.6”, at 0x7e89ac34d71b, in \_\_gxx\_personality\_v0  
#1 Object “/usr/lib/x86\_64-linux-gnu/libstdc++.so.6”, at 0x7e89ac34d590, in  
#0 Object “/usr/lib/x86\_64-linux-gnu/libstdc++.so.6”, at 0x7e89ac34d210, in  
Segmentation fault (Signal sent by the kernel [(nil)])  
/opt/nvidia/deepstream/deepstream-8.0/entrypoint.sh: line 15: 41 Segmentation fault (core dumped) /opt/nvidia/nvidia\_entrypoint.sh $@

Second - Running .onnx generated for TAO:

gstnvtracker: Loading low-level lib at /opt/nvidia/deepstream/deepstream-8.0/lib/libnvds\_nvmultiobjecttracker.so  
[NvMultiObjectTracker] Initialized  
WARNING: ../nvdsinfer/nvdsinfer\_model\_builder.cpp:1261 Deserialize engine failed because file path: /app/Inference/configs/deepstream-app/models/tao-test/efficientdet-d0.fp16.engine open error  
0:00:21.323347765 41 0x7919e8bdbc90 WARN nvinfer gstnvinfer.cpp:682:gst\_nvinfer\_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::deserializeEngineAndBackend() \<nvdsinfer\_context\_impl.cpp:2097\> [UID = 1]: deserialize engine from file :/app/Inference/configs/deepstream-app/models/tao-test/efficientdet-d0.fp16.engine failed  
0:00:21.323379299 41 0x7919e8bdbc90 WARN nvinfer gstnvinfer.cpp:682:gst\_nvinfer\_logger: NvDsInferContext[UID 1]: Warning from NvDsInferContextImpl::generateBackendContext() \<nvdsinfer\_context\_impl.cpp:2202\> [UID = 1]: deserialize backend context from engine from file :/app/Inference/configs/deepstream-app/models/tao-test/efficientdet-d0.fp16.engine failed, try rebuild  
0:00:21.323399979 41 0x7919e8bdbc90 INFO nvinfer gstnvinfer.cpp:685:gst\_nvinfer\_logger: NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::buildModel() \<nvdsinfer\_context\_impl.cpp:2123\> [UID = 1]: Trying to create engine from model files  
INFO: Dispositivo 19443010A15D842F00 conectado com sucesso.  
INFO: Usando preset de sensor: 1080p @ 60 FPS (1920x1080 @ 58.0 FPS)  
INFO: Usando preset de corte: 640x640 (1:1) (640x640)  
INFO: Modo de Stream OAK: Bruto (NV12)  
AVISO: Tentativa de atualizar controle desconhecido: update\_crop\_size\_for\_ds  
Connected cameras: [\<CameraBoardSocket.CAM\_A: 0\>, \<CameraBoardSocket.CAM\_B: 1\>, \<CameraBoardSocket.CAM\_C: 2\>]  
INFO: Configurando modo ‘Esticar’. Redimensionando de 1920x1080 para 640x640.  
INFO: Profundidade desabilitada. Pulando a criação dos nós estéreo.  
INFO: Aplicando configurações iniciais da câmera…  
INFO: Pipeline V3 iniciado no dispositivo.  
**WARNING: [TRT]: onnxOpImporters.cpp:6521: Attribute class\_agnostic not found in plugin node! Ensure that the plugin creator has a default value defined or the engine may fail to build.**

the zip file contain TAO notebook, Pipeline and files for inference.

[tao-test.zip](https://forums.developer.nvidia.com/uploads/short-url/yMB1SMqcNE81upPmJGhTkt7nVzs.zip) (56.3 MB)

---

<div class="post-metadata">

**Author:** ![henrique.cerqueira](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@henrique.cerqueira](https://forums.developer.nvidia.com/u/henrique.cerqueira)\
**Post date:** [October 24, 2025, 1:56pm UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/2 "2025-10-24T13:56:47Z")

</div>

The inference test on TAO notebook after train runing correctly, we can see the images results bellow:

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/c/8/8/c88e3e9b9f78b04fb823add6c572f536b7c4a929.jpeg)

 ![camera_1_20250703_101124_349_jpg.rf.32f780555139da5d7d306e1bd7ba5706](https://global.discourse-cdn.com/nvidia/original/4X/c/e/6/ce6cfd585e83290e128639ae0d7357333eab94b8.jpeg)

 ![camera_1_20250703_101534_438_jpg.rf.1906b71ff85ddd495f1cbf9d4edd028d](https://global.discourse-cdn.com/nvidia/original/4X/7/c/a/7cac849546dcb536c5ff378887e330e07e059d94.jpeg)

---

<div class="post-metadata">

**Author:** ![henrique.cerqueira](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@henrique.cerqueira](https://forums.developer.nvidia.com/u/henrique.cerqueira)\
**Post date:** [October 29, 2025, 10:52am UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/3 "2025-10-29T10:52:31Z")

</div>

Hi, Anyone can help me?

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [November 27, 2025, 6:18am UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/4 "2025-11-27T06:18:05Z")

</div>

Moving to TAO forum.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [November 27, 2025, 6:26am UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/5 "2025-11-27T06:26:30Z")

</div>

> [@henrique.cerqueira](#):
>
> The inference test on TAO notebook after train runing correctly,

> [@henrique.cerqueira](#):
>
> [tao-test.zip](https://forums.developer.nvidia.com/uploads/short-url/yMB1SMqcNE81upPmJGhTkt7nVzs.zip) (56.3 MB)

The inference result is correct when you run inference with the tf model.  
How about run inference against the tensorrt engine? i.e., What is the result in below cell?

 ![image](https://global.discourse-cdn.com/nvidia/original/4X/8/3/d/83d6c7e45e5ab028d999b43e2006ce5b52ed0cec.jpeg)

---

<div class="post-metadata">

**Author:** ![henrique.cerqueira](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@henrique.cerqueira](https://forums.developer.nvidia.com/u/henrique.cerqueira)\
**Post date:** [December 1, 2025, 11:12am UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/6 "2025-12-01T11:12:41Z")

</div>

I tried to use 2 way the custom model, first i use the onnx model converted for TAO on deepstream from gen trt engine and second i use trt engine gen from TAO. i can’t understand why dosen’t works, because after custom train on jupyter the model works correctly.

---

<div class="post-metadata">

**Author:** ![Morganh](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/morganh/32/9748_2.png) [@Morganh](https://forums.developer.nvidia.com/u/Morganh)\
**Post date:** [December 2, 2025, 3:11am UTC](https://forums.developer.nvidia.com/t/inference-error-with-custom-train/348846/7 "2025-12-02T03:11:07Z")

</div>

Hi @henrique.cerqueira ,  
As mentioned above, to narrow down, could you please check if `tao deploy efficientdet_tf2 inference xxx` can generate expected results?  
If this works, that means the tao pipeline works. Then we can continue to check the gap in deepstream side.