Tao toolkit transfer learning with Trafficcamnet -> .etlt files

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Hello!

Using tao toolkit and the detectnetv2 notebook, I am training the trafficcamnet unpruned model on a my own dataset of car images. The trafficcamnet unpruned model is a .tlt model. At each model checkpoint, I am producing a .hdf5 file and a .ckzip file. From reading through tao documentation, I would expect to be producing .tlt files at each model checkpoint. This is critical because I would like to produce a .etlt model which I can feed to the tao-converter after I am done training. How can I produce .tlt files from training?

Thanks

2

The tlt file is actually encryption of hdf5 file.
The etlt file is actually encryption of onnx file.

Before TAO 5.0, tlt file is generated during training. The etlt file is generated during exporting.
Since TAO 5.0, hdf5 file is generated during training. The onnx file is generated during exporting.

To encrypt hdf5 file to tlt file, or to encrypt onnx file to etlt file, you can refer to
https://github.com/NVIDIA/tao_tensorflow2_backend/blob/main/internal/encode.eff.py.

Also, you can use trtexec instead of tao-converter now. We can use trtexec to generate tensorrt engine against onnx file.

3

Thank you!

4

Using tao deploy is the same as trtexec, correct?

Also, my end goal is to run a Triton server with my generated tensorrt engine. Ive created a tensorrt engine with tao deploy and attempted to run it on a Triton server, but received this error:
Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 0)

In creating my triton server docker image, I was building from nvcr.io/nvidia/tritonserver:21.10-py3. Is there a more updated docker image?

Long story short, I want to run my tensorrt engine (generated with tao deploy) on a triton server. What is the best way to do this?

5

The tao deploy can also generate tensorrt engine. But they are not the same thing. The tao deploy is a package for deploying deep learning models from TAO Toolkit. Its source code is GitHub - NVIDIA/tao_deploy: Package for deploying deep learning models from TAO Toolkit.
The trtexec is part of TensorRT.

For Error Code 1: Serialization (Serialization assertion safeVersionRead == safeSerializationVersion failed.Version tag does not match. Note: Current Version: 43, Serialized Engine Version: 0), it is a common error when TensorRT version is not the same between building engine and running engine.

If you build engine with tao deploy, it will use docker nvcr.io/nvidia/tao/tao-toolkit:5.0.0-deploy. The tensort version should be different from the version in nvcr.io/nvidia/tritonserver:21.10-py3.

Two options for you.
You can find another tritonserver docker with the same TensorRT version as nvcr.io/nvidia/tao/tao-toolkit:5.0.0-deploy.
Or, you can user trtexec to build tensort engine inside nvcr.io/nvidia/tritonserver:21.10-py3. Refer to TRTEXEC with DetectNet-v2 - NVIDIA Docs.

6

I think it makes sense to use trtexec to build tensorrt engine inside nvcr.io/nvidia/tritonserver:21.10-py3 or any version of triton server.

For generating an INT8 calibration file, does it make sense to use ‘tao deploy detectnet_v2 gen_trt_engine’?

7

Yes, it makes sense.

8

Now I am trying to decode the trafficcamnet pruned etlt model to onnx so I can run it through the new pipeline (trtexec → triton server). I am using trying to use decode_eff (https://github.com/NVIDIA/tao_tensorflow2_backend/blob/main/internal/decode_eff.py).

root@ip-172-31-3-32:/workspace/tao-tf2/internal# python3 decode_eff.py -m /workspace/tao-experiments/path/to/model.etlt -o /workspace/path/to/output/ -k model_key

I am seeing this error:
Traceback (most recent call last):
File “decode_eff.py”, line 65, in
main()
File “decode_eff.py”, line 60, in main
decode_eff(args.model, args.output, args.key)
File “decode_eff.py”, line 21, in decode_eff
eff_art = Archive.restore_artifact(
File “”, line 519, in restore_artifact
File “/usr/lib/python3.8/tarfile.py”, line 1621, in open
return func(name, filemode, fileobj, **kwargs)
File “/usr/lib/python3.8/tarfile.py”, line 1678, in gzopen
raise ReadError(“not a gzip file”)
tarfile.ReadError: not a gzip file

What is the correct input? Is this the right code to use for decoding an etlt model to onnx?

9

I’m closing this topic due to there is no update from you for a period, assuming this issue was resolved.
If still need the support, please open a new topic. Thanks

Please refer to https://github.com/NVIDIA/tao_tensorflow1_backend/blob/c7a3926ddddf3911842e057620bceb45bb5303cc/nvidia_tao_tf1/cv/detectnet_v2/inferencer/trt_inferencer.py#L279-L286 instead.

Please note that, the undecrypted trafficcamnet pruned file is a uff file instead of onnx file.

10

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.