Tlt lprnet export error, TypeError: set_data_preprocessing_parameters() got an unexpected keyword argument 'image_mean'

Please provide the following information when requesting support.

• Hardware V100
• Network Type Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)

dockers: 
        nvidia/tlt-streamanalytics: 
                docker_registry: nvcr.io
                docker_tag: v3.0-py3
                tasks: 
                        1. augment
                        2. bpnet
                        3. classification
                        4. detectnet_v2
                        5. dssd
                        6. emotionnet
                        7. faster_rcnn
                        8. fpenet
                        9. gazenet
                        10. gesturenet
                        11. heartratenet
                        12. lprnet
                        13. mask_rcnn
                        14. multitask_classification
                        15. retinanet
                        16. ssd
                        17. unet
                        18. yolo_v3
                        19. yolo_v4
                        20. tlt-converter
        nvidia/tlt-pytorch: 
                docker_registry: nvcr.io
                docker_tag: v3.0-py3
                tasks: 
                        1. speech_to_text
                        2. speech_to_text_citrinet
                        3. text_classification
                        4. question_answering
                        5. token_classification
                        6. intent_slot_classification
                        7. punctuation_and_capitalization
format_version: 1.0
tlt_version: 3.0
published_date: 04/16/2021

• Training spec file(If have, please share here)


random_seed: 42
lpr_config {
  hidden_units: 512
  max_label_length: 8
  arch: "baseline"
  nlayers: 18 #setting nlayers to be 10 to use baseline10 model
}
training_config {
  batch_size_per_gpu: 32
  num_epochs: 24
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 1e-6
    max_learning_rate: 1e-5
    soft_start: 0.001
    annealing: 0.5
  }
  }
  regularizer {
    type: L2
    weight: 5e-4
  }
}
eval_config {
  validation_period_during_training: 5
  batch_size: 1
}
augmentation_config {
    output_width: 96
    output_height: 48
    output_channel: 3
    keep_original_prob: 0.3
   # transform_prob: 0.5
   # rotate_degree: 5
}
dataset_config {
  data_sources: {
    label_directory_path: "/workspace/tlt-experiments/data/openalpr/train/label"
    image_directory_path: "/workspace/tlt-experiments/data/openalpr/train/image"
  }
  characters_list_file: "/workspace/tlt-experiments/lprnet/specs/us_lp_characters.txt"
  validation_data_sources: {
    label_directory_path: "/workspace/tlt-experiments/data/openalpr/val/label"
    image_directory_path: "/workspace/tlt-experiments/data/openalpr/val/image"
  }
}

• How to reproduce the issue ?
I just followed Creating a Real-Time License Plate Detection and Recognition App | NVIDIA Developer Blog and trained LPD and LPR networks. Now am trying to make etlt file by the below command which gives me the below error

$ tlt lprnet export -m /workspace/tlt-experiments/lprnet/weights/lprnet_epoch-24.tlt -k nvidia_tlt -e /workspace/tlt-experiments/lprnet/tutorial_spec.txt
2021-07-02 10:27:19,956 [INFO] root: Registry: ['nvcr.io']
Matplotlib created a temporary config/cache directory at /tmp/matplotlib-78fgwu6_ because the default path (/.config/matplotlib) is not a writable directory; it is highly recommended to set the MPLCONFIGDIR environment variable to a writable directory, in particular to speed up the import of Matplotlib and to better support multiprocessing.
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2021-07-02 10:27:31,438 [INFO] iva.common.export.keras_exporter: Using input nodes: ['image_input']
2021-07-02 10:27:31,438 [INFO] iva.common.export.keras_exporter: Using output nodes: ['tf_op_layer_ArgMax', 'tf_op_layer_Max']
2021-07-02 10:27:31,438 [INFO] iva.lprnet.utils.spec_loader: Merging specification from /workspace/tlt-experiments/lprnet/tutorial_spec.txt
The ONNX operator number change on the optimization: 132 -> 61
2021-07-02 10:27:47,248 [INFO] keras2onnx: The ONNX operator number change on the optimization: 132 -> 61
Traceback (most recent call last):
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/export.py", line 215, in <module>
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/export.py", line 142, in main
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/export.py", line 211, in run_export
  File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/keras_exporter.py", line 371, in export
TypeError: set_data_preprocessing_parameters() got an unexpected keyword argument 'image_mean'
2021-07-02 10:27:49,446 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

any suggestions will be appreciated

Actually the etlt model is generated. Please check.
There is no issue in 3.0-dp-py3 version. It happened in 3.0-py3 version. You can ignore this error. TLT team will improve this in next release.

1 Like

Hi

Actually, the notebook was created the “lprnet_epoch-24.etlt” after this step.
So, I ignored this error but I met the another error at the next evaluate step on notebook.

# Verify the tensorrt engine accuracy on the validation dataset
!lprnet evaluate --gpu_index=$GPU_INDEX -e $SPECS_DIR/tutorial_spec.txt \
                     -m $USER_EXPERIMENT_DIR/export/lprnet_epoch-24.engine \
                     --trt
    Using TensorFlow backend.
    WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
    Using TensorFlow backend.
    2021-07-26 05:16:47,074 [INFO] iva.lprnet.utils.spec_loader: Merging specification from /workspace/data/lprnet/specs/tutorial_spec.txt
    Traceback (most recent call last):
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/evaluate.py", line 152, in <module>
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/evaluate.py", line 148, in main
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/evaluate.py", line 105, in evaluate
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/inferencer/trt_inferencer.py", line 31, in __init__
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/inferencer/engine.py", line 113, in load_engine
    FileNotFoundError: [Errno 2] No such file or directory: '/workspace/data/lprnet/export/lprnet_epoch-24.engine'
    Exception ignored in: <bound method TRTInferencer.__del__ of <iva.common.inferencer.trt_inferencer.TRTInferencer object at 0x7fdb63ddc518>>
    Traceback (most recent call last):
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/inferencer/trt_inferencer.py", line 139, in __del__
      File "/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/inferencer/trt_inferencer.py", line 96, in clear_trt_session
    AttributeError: 'TRTInferencer' object has no attribute 'context'

It seems that the name was not match on the tutorial code as below.

!tlt lprnet evaluate --gpu_index=$GPU_INDEX -e $SPECS_DIR/tutorial_spec.txt \
                     -m $USER_EXPERIMENT_DIR/export/lprnet_epoch-24.**engine** \
                     --trt

So, I modified the file name but it showed the another error. Would you please check it on your side?
Docker image: nvidia/tlt-streamanalytics: 3.0-py3
Sample: tlt_cv_samples_v1.1.0

If you have any questions, please let me know.
Best regards.
Kaka

@Kaka_m ,
Please create a new forum topic since it is not the same error. Thanks.

I see.

Here I am also facing the same issue,
Using TensorFlow backend.
WARNING:tensorflow:Deprecation warnings have been disabled. Set TF_ENABLE_DEPRECATION_WARNINGS=1 to re-enable them.
Using TensorFlow backend.
2021-08-30 07:38:58,070 [INFO] iva.common.export.keras_exporter: Using input nodes: [‘image_input’]
2021-08-30 07:38:58,070 [INFO] iva.common.export.keras_exporter: Using output nodes: [‘tf_op_layer_ArgMax’, ‘tf_op_layer_Max’]
2021-08-30 07:38:58,070 [INFO] iva.lprnet.utils.spec_loader: Merging specification from /workspace/tlt-experiments/lprnet/specs/tutorial_spec.txt
The ONNX operator number change on the optimization: 132 → 61
2021-08-30 07:39:08,650 [INFO] keras2onnx: The ONNX operator number change on the optimization: 132 → 61
Traceback (most recent call last):
File “/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/export.py”, line 215, in
File “/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/export.py”, line 142, in main
File “/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/lprnet/scripts/export.py”, line 211, in run_export
File “/opt/tlt/.cache/dazel/_dazel_tlt/2b81a5aac84a1d3b7a324f2a7a6f400b/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/common/export/keras_exporter.py”, line 371, in export
TypeError: set_data_preprocessing_parameters() got an unexpected keyword argument ‘image_mean’
2021-08-30 13:09:10,361 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

Can you update to latest docker? nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.08-py3

There should be no issue now.

https://docs.nvidia.com/tao/tao-toolkit/text/migrating_to_tao_toolkit.html