# TRT8 serialize() return nullptr

**URL:** <https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818>\
**Category:** Jetson AGX Orin\
**Tags:** tensorrt\
**Created:** [July 14, 2023, 2:40am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818 "2023-07-14T02:40:26Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 14, 2023, 2:40am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/1 "2023-07-14T02:40:26Z")

</div>

`nvinfer1::ICudaEngine *engine = builder_->buildEngineWithConfig(*network_, *builder_config_);`  
I used my custom plugin to build the engine, but it took a long time, about 90 seconds, compared to only 10 seconds on x86.

`nvinfer1::IHostMemory *serialized_model = engine->serialize();`  
So I want to solve this problem by serializing engine and save it in advance, but it returns null pointer on trt8.5.

Please Help

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 14, 2023, 6:42am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/3 "2023-07-14T06:42:47Z")

</div>

Hi,

Is the engine successfully built?  
If the engine is NULL, the serialization might fail and returns a null pointer.

Thanks.

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 14, 2023, 6:44am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/4 "2023-07-14T06:44:19Z")

</div>

> [@AastaLLL](#):
>
> engine

I checked it and it was successfully built, so I want to know why.

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 14, 2023, 6:52am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/5 "2023-07-14T06:52:18Z")

</div>

Hi,

Which JetPack version do you use?  
For JetPack 5, the API should look like this:  
[https://elinux.org/Jetson/L4T/TRT\_Customized\_Example#OpenCV\_with\_ONNX\_model](https://elinux.org/Jetson/L4T/TRT_Customized_Example#OpenCV_with_ONNX_model)

Thanks.

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 17, 2023, 1:46am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/7 "2023-07-17T01:46:50Z")

</div>

JetPack 5.0.2 GA  
Our model was trained in caffe serval years ago and we use C++ to load model and complete deployment.  
The following code is work but it cost so much time when building engine.

```auto
  nvinfer1::ICudaEngine *engine =
      builder_->buildEngineWithConfig(*network_, *builder_config_);
  context_ = engine->createExecutionContext();

```

So I want to serialize model and load it next time to skip building stage:

```auto
  nvinfer1::ICudaEngine *engine =
      builder_->buildEngineWithConfig(*network_, *builder_config_);

  // generate custom.engine
  nvinfer1::IHostMemory *serialized_model(engine->serialize());
  std::ofstream f("custom.engine", std::ios::binary);
  f.write(reinterpret_cast<const char *>(serialized_model->data()),
          serialized_model->size());

```

I want to know can this work without trtexec tool?

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 17, 2023, 6:51am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/8 "2023-07-17T06:51:55Z")

</div>

Hi,

Do you try to serialize the model with TensorRT API or trtexec?  
Does it work with the below command?

```auto
$ /usr/src/tensorrt/bin/trtexec --saveEngine=[file] ...
$ /usr/src/tensorrt/bin/trtexec --loadEngine=[file] ...

```

Thanks.

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 17, 2023, 9:04am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/9 "2023-07-17T09:04:53Z")

</div>

> [@AastaLLL](#):
>
> `--loadEngine=`

```auto
  nvinfer1::IHostMemory *serialized_model(engine->serialize());

```

I have tried with TensorRT8.4 API which returned nullptr, but Tensorrt7.4 works fine, it’s this a bug in trt8?

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 19, 2023, 4:10am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/10 "2023-07-19T04:10:51Z")

</div>

Hi,

Could you try the trtexec command with TensorRT 8?

There are lots of API changes between TensorRT 7 and TensorRT 8.  
Testing it with _trtexec_ can narrow down the issue that comes from implementation or the library itself.

Thanks.

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 19, 2023, 4:12am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/11 "2023-07-19T04:12:08Z")

</div>

Hi,

Just want to confirm the environment first.

Which device are you using for the TensorRT 7?  
We don’t have TensorRT 7 for Orin.

Thanks.

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 19, 2023, 5:28am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/13 "2023-07-19T05:28:15Z")

</div>

> [@AastaLLL](#):
>
> Hi,
> 
> Just want to confirm the environment first.
> 
> Which device are you using for the TensorRT 7?  
> We don’t have TensorRT 7 for Orin.
> 
> Thanks.

I use TensorRT 7 on x86 machine and everything goes well, but it took a long time when I use TensorRT 8 to build Engine on Orin, I want to know why there is a big gap between these two machine, could you tell me why?

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 21, 2023, 2:23am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/14 "2023-07-21T02:23:27Z")

</div>

> [@AastaLLL](#):
>
> There are lots of API changes between TensorRT 7 and TensorRT 8.

Is the change in API the root cause of trt8 serialization failure? Has there been a change in compatibility?

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 21, 2023, 4:03am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/15 "2023-07-21T04:03:49Z")

</div>

> [@AastaLLL](#):
>
> There are lots of API changes between TensorRT 7 and TensorRT 8

Our model uses a lot of custom plugins, and we have adapted pluginV2 in Tensorrt8. and we were able to successfully generate engine, but the loading time has been extended too much. We have never used trtexec to generate an engine. Considering the time cost, can we solve our problem by using trt API(engine-\>serialize())?

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 21, 2023, 6:53am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/16 "2023-07-21T06:53:06Z")

</div>

Hi,

Have you tried to serialize and deserialize with _trtexec_ on Orin+TensorRT 8?  
We would like to know if it is working first.

Thanks.

---

<div class="post-metadata">

**Author:** ![xunjtech](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/xunjtech/32/254831_2.png) [@xunjtech](https://forums.developer.nvidia.com/u/xunjtech)\
**Post date:** [July 21, 2023, 11:51am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/17 "2023-07-21T11:51:35Z")

</div>

> [@AastaLLL](#):
>
> Have you tried

I tried and it kept reminding me that there was an error loading custom plugin. Although the plugin has already been adapted to trt8, it cannot be loaded directly into it yet.

> <https://github.com/ApolloAuto/apollo/blob/a3c851fc5844e0684b9c5108231fcc2c15cebb8e/modules/perception/inference/tensorrt/rt_net.cc#L794>

---

<div class="post-metadata">

**Author:** ![AastaLLL](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/aastalll/32/14043_2.png) [@AastaLLL](https://forums.developer.nvidia.com/u/AastaLLL)\
**Post date:** [July 24, 2023, 5:42am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/18 "2023-07-24T05:42:16Z")

</div>

Hi,

Thanks for the testing.

We need to reproduce this issue in our environment to gather more info.  
Could you share a reproducible source and the model?

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\
**Post date:** [August 7, 2023, 5:42am UTC](https://forums.developer.nvidia.com/t/trt8-serialize-return-nullptr/259818/19 "2023-08-07T05:42:30Z")

</div>

Closed on August 7, 2023, 5:42am UTC

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.
