TensorRT with Jetpack 4.6.2 on Xavier NX emmc 16gb version

[SooMSooM]

Hi,

i am trying to use tensorrt on a Xavier NX with emmc and 16gb ram and flashed Jetpack 4.6.1 and 4.6.2 onto it.
Sadly the trt engine cannot be build from an onnx file… the following error message is outputted:

[05/09/2022-10:29:16] [E] Error[2]: [utils.cpp::checkMemLimit::380] Error Code 2: Internal Error (Assertion upperBound != 0 failed. Unknown embedded device detected. Please update the table with the entry: {{1794, 6, 16}, 12653},)
[05/09/2022-10:29:16] [E] Error[2]: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
[05/09/2022-10:29:16] [E] Engine could not be created from network
[05/09/2022-10:29:16] [E] Building engine failed
[05/09/2022-10:29:16] [E] Failed to create engine from model.
[05/09/2022-10:29:16] [E] Engine set up failed

The module is placed on an dev kit carrier and flashed via the sdkmanager onto a nvme in the m2 slot.
How do I update the table entry?

Kind regards,
Nils

[SivaRamaKrishnaNV]

Dear @SooMSooM,
Does the target has enough memory(at write location)? Could you share your ONNX- model and used command

[SooMSooM]

Hi Siva,

thanks for taking time for me.
I am working on an 250 gb SSD, so space should not be a problem and I just installed everything coming with the jetpack via the sdkmanager to the nvme.

Due to nda reasons I cannot share my onnx file but you can use any and you will get the error.
For example:

git clone https://github.com/PINTO0309/PINTO_model_zoo.git
cd PINTO_model_zoo/061_U-2-Net/30_human_segmentation/
./download_320x320.sh

and if you check

/usr/src/tensorrt/bin/trtexec --onnx=./saved_model_320x320/u2net_human_seg_320x320.onnx

you will get my error. So it is just the call for trtexec with an onnx file.

Kind regards,
Nils

[SivaRamaKrishnaNV]

Dear @SooMSooM,
I don’t see any issue with Jetpack 4.6.1 for the above model. Attached the
log.txt (26.9 KB).
.

[SooMSooM]

HI @SivaRamaKrishnaNV,

the device info in your log shows, that you are not using the 16gb ram version of the xavier nx.
On the dev kit version with the sd slot I do not have this issue…
Just on the 16 gb module with emmc memory.

See my full log attached.

log.txt (21.3 KB)

edit: I just realized, that the error part is missing at the end of the log…
This comes after the last line

[05/11/2022-08:47:14] [E] Error[2]: [utils.cpp::checkMemLimit::380] Error Code 2: Internal Error (Assertion upperBound != 0 failed. Unknown embedded device detected. Please update the table with the entry: {{1794, 6, 16}, 12653},)
[05/11/2022-08:47:14] [E] Error[2]: [builder.cpp::buildSerializedNetwork::609] Error Code 2: Internal Error (Assertion enginePtr != nullptr failed. )
[05/11/2022-08:47:14] [E] Engine could not be created from network
[05/11/2022-08:47:14] [E] Building engine failed
[05/11/2022-08:47:14] [E] Failed to create engine from model.
[05/11/2022-08:47:14] [E] Engine set up failed
&&&& FAILED TensorRT.trtexec [TensorRT v8201] # /usr/src/tensorrt/bin/trtexec --onnx=u2net_human_seg_320x320.onnx

[SivaRamaKrishnaNV]

Dear @SooMSooM,
Thanks for clarification. It looks like a issue with 16GB Xavier NX devices as there is no table entry for this configuration TensorRT code. I will file a bug internally. Does this block your development?

[SooMSooM]

Hi @SivaRamaKrishnaNV ,

that’s for the help :)

In our project we shifted modules around to get something running, but have more of the 16gb versions lying around.
It would be beneficial to have more functioning setups.
Can you estimate how long the bugfix will take and will there be a intermediate update or a new jetpack version for that?

Kind regards,
Nils

[forumuser]

Hello, @SooMSooM

Thank you for good question.
Can I know you what carrier board you use?

Thank you.

[SivaRamaKrishnaNV]

Dear @SooMSooM,
The fix is in jetpack 5.0.1. Could you test with Jetpack 5.0.1 and confirm if it is working?

[SooMSooM]

@forumuser we have currently 3 to try it.
The original one from the dev kit, this one https://www.antratek.de/a206-carrier-board-for-jetson-nano-xavier-nx ,and one I cant remember.
But it is not a problem of the carrier!

@SivaRamaKrishnaNV I will be back in the office on Monday and will try it out! Thanks for the help in advance!

[forumuser]

Hello, @SooMSooM

Thank you very much

[SooMSooM]

@SivaRamaKrishnaNV
Yes it is building and running.
Thanks for the fix!