Problem in tlt export

p.vahidinia · August 3, 2021, 7:06am

Hi.
According to the documentation and select the first option for ingesting training data, we do not need to include
**--cal_image_dir** and **--cal_data_file**.

https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/object_detection/fasterrcnn.html#int8-mode-overview

so I executed this command:
https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/object_detection/fasterrcnn.html#exporting-a-model

But I got this error:

Traceback (most recent call last):
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/yolo_v3/scripts/export.py", line 12, in <module>
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py", line 219, in launch_export
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py", line 201, in run_export
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/keras_exporter.py", line 360, in export
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/base_exporter.py", line 184, in get_calibrator
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/base_exporter.py", line 284, in generate_tensor_file
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/base_exporter.py", line 327, in generate_random_tensorfile
File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/tensorfile.py", line 54, in __init__
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 312, in __init__
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/usr/local/lib/python3.6/dist-packages/h5py/_hl/files.py", line 148, in make_fid
fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py/h5f.pyx", line 98, in h5py.h5f.create
ValueError: Invalid file name (invalid file name)
Using TensorFlow backend.
write export results to /workspace/tlt/local_dir/tlt_yolov3_pv_resnet18/unpruned_model/export

Morganh · August 3, 2021, 7:57am

Can you share your command?

p.vahidinia · August 3, 2021, 8:03am

This is my command:

export_cmd = f'''{algorithm_name} export -m {final_model}  \
                        -o {output_folder_address}/final_model.etlt \
                        -e {spec_dir} \
                        -k $KEY \
                        --data_type {data_type} \
                        --batch_size {batch_size} \
                        --batches {batches} \
                        --cal_cache_file {output_folder_address}/cal.bin  \
                        --engine_file {output_folder_address}/trt.engine \
                        --gpu_index {gpu_num}'''

Morganh · August 3, 2021, 8:12am

Can you double check below?
-m {final_model}

To narrow down, I suggest you run the explicit command with “tlt faster_rcnn export xxx” .

p.vahidinia · August 3, 2021, 8:19am

Thanks for help.
I use tlt version3 inside the docker and I wrote all task commands in the same way (train, evaluate, inference, prune) Also there is no problem when I set --cal_image_dir and --cal_data_file. But I think these are not needed.

Morganh · August 3, 2021, 9:00am

So, can you run the explicit command with “faster_rcnn export xxx” inside the docker?

p.vahidinia · August 3, 2021, 9:45am

I executed this command. But I got the same error:

!yolo_v3 export -m /workspace/tlt/local_dir/tlt_yolov3_pv_resnet18/unpruned_model/weights/final_model.tlt \
-o /workspace/tlt/local_dir/tlt_yolov3_pv_resnet18/unpruned_model/export/final_model.etlt \
-e /workspace/tlt/local_dir/tlt_yolov3_pv_resnet18/unpruned_model/final_spec.txt \
-k $KEY \
--data_type int8 \
--batch_size 2 \
--batches 10 \
--cal_cache_file /workspace/tlt/local_dir/tlt_yolov3_pv_resnet18/unpruned_model/export/cal.bin \
--engine_file /workspace/tlt/local_dir/tlt_yolov3_pv_resnet18/unpruned_model/export/trt.engine \
--gpu_index 0

Morganh · August 3, 2021, 10:11am

For yolo_v3, can you follow NVIDIA TAO Documentation and retry?

p.vahidinia · August 3, 2021, 10:35am

Yes, it works according to yolov3 documentation:

But since it is also mentioned that we can use option 1 and using the training data loader, I thought the export is the same for all object detection models.

I am new to TLT. Excuse me if my questions are confusing.

p.vahidinia · August 3, 2021, 7:44pm

I executed this command:

!faster_rcnn export -m /workspace/tlt/local_dir/tlt_faster_rcnn_resnet18/unpruned_model/weights/final_model.tlt \
-o /workspace/tlt/local_dir/tlt_faster_rcnn_resnet18/unpruned_model/export/final_model.etlt \
-e /workspace/tlt/local_dir/tlt_faster_rcnn_resnet18/unpruned_model/final_spec.txt \
-k $KEY \
--data_type int8 \
--batch_size 2 \
--batches 10 \
--cal_image_dir /workspace/tlt/tlt-experiments/all_object_detection_approaches/dataset_1_temp/train/images \
--cal_data_file /workspace/tlt/local_dir/tlt_faster_rcnn_resnet18/unpruned_model/export/cal.tensorfile \
--cal_cache_file /workspace/tlt/local_dir/tlt_faster_rcnn_resnet18/unpruned_model/export/cal.bin  \
--engine_file /workspace/tlt/local_dir/tlt_faster_rcnn_resnet18/unpruned_model/export/trt.engine \
--force_ptq \
--gpu_index 1

and I got this error:

Traceback (most recent call last):
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/faster_rcnn/scripts/export.py", line 12, in <module>
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py", line 219, in launch_export
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/app.py", line 201, in run_export
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/keras_exporter.py", line 360, in export
TypeError: get_calibrator() got an unexpected keyword argument 'image_mean'

Morganh · August 4, 2021, 10:35am

The etlt model is generated successfully.

But the cal.bin is not generated successfully due to above error. It is a regression issue in tlt 3.0-py3.
For workaround, please login 3.0-dp-py3 docker and run the same command to generate cal.bin.
Sorry for inconvenience.

p.vahidinia · August 4, 2021, 10:40am

Thank you. I will try

system · October 3, 2021, 10:40am

This topic was automatically closed 60 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to export QAT yolov3 in int8 TAO Toolkit	7	550	April 25, 2023
Tlt lprnet export error, TypeError: set_data_preprocessing_parameters() got an unexpected keyword argument 'image_mean' TAO Toolkit	7	1242	October 12, 2021
Train with my own tlt model #2 TAO Toolkit	42	2776	February 8, 2022
Problem about installing TLT TAO Toolkit	9	1306	October 12, 2021
Error in detectnet_v2 - 10. Model Export TAO Toolkit	4	547	October 12, 2021
Error doing inference with yolo exported enginge TAO Toolkit yolo	4	668	October 4, 2021
OSError: Specfile not found plz help TAO Toolkit	16	1584	October 12, 2021
Preparing Models for Object Detection with Real and Synthetic Data and the NVIDIA Transfer Learning Toolkit Technical Blog	10	670	June 29, 2021
TLT YOLOv3 Int8 can not detect anything TAO Toolkit	17	1689	October 12, 2021
Inference YOLO_v4 int8 mode doesn't show any bounding box TAO Toolkit	31	2542	November 12, 2021

Problem in tlt export

Related topics