LPRNet with double line license plate

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (dGPU: NVIDIA RTX 3050 6GB)
• Network Type (LPRNet)
• TLT Version:

Configuration of the TAO Toolkit Instance

task_group:         
    model:             
        dockers:                 
            nvidia/tao/tao-toolkit:                     
                5.5.0-pyt:                         
                    docker_registry: nvcr.io
                    tasks: 
                        1. action_recognition
                        2. centerpose
                        3. visual_changenet
                        4. deformable_detr
                        5. dino
                        6. grounding_dino
                        7. mask_grounding_dino
                        8. mask2former
                        9. mal
                        10. ml_recog
                        11. ocdnet
                        12. ocrnet
                        13. optical_inspection
                        14. pointpillars
                        15. pose_classification
                        16. re_identification
                        17. classification_pyt
                        18. segformer
                        19. bevfusion
                5.0.0-tf1.15.5:                         
                    docker_registry: nvcr.io
                    tasks: 
                        1. bpnet
                        2. classification_tf1
                        3. converter
                        4. detectnet_v2
                        5. dssd
                        6. efficientdet_tf1
                        7. faster_rcnn
                        8. fpenet
                        9. lprnet
                        10. mask_rcnn
                        11. multitask_classification
                        12. retinanet
                        13. ssd
                        14. unet
                        15. yolo_v3
                        16. yolo_v4
                        17. yolo_v4_tiny
                5.5.0-tf2:                         
                    docker_registry: nvcr.io
                    tasks: 
                        1. classification_tf2
                        2. efficientdet_tf2
    dataset:             
        dockers:                 
            nvidia/tao/tao-toolkit:                     
                5.5.0-data-services:                         
                    docker_registry: nvcr.io
                    tasks: 
                        1. augmentation
                        2. auto_label
                        3. annotations
                        4. analytics
    deploy:             
        dockers:                 
            nvidia/tao/tao-toolkit:                     
                5.5.0-deploy:                         
                    docker_registry: nvcr.io
                    tasks: 
                        1. visual_changenet
                        2. centerpose
                        3. classification_pyt
                        4. classification_tf1
                        5. classification_tf2
                        6. deformable_detr
                        7. detectnet_v2
                        8. dino
                        9. dssd
                        10. efficientdet_tf1
                        11. efficientdet_tf2
                        12. faster_rcnn
                        13. grounding_dino
                        14. mask_grounding_dino
                        15. mask2former
                        16. lprnet
                        17. mask_rcnn
                        18. ml_recog
                        19. multitask_classification
                        20. ocdnet
                        21. ocrnet
                        22. optical_inspection
                        23. retinanet
                        24. segformer
                        25. ssd
                        26. trtexec
                        27. unet
                        28. yolo_v3
                        29. yolo_v4
                        30. yolo_v4_tiny
format_version: 3.0
toolkit_version: 5.5.0
published_date: 08/26/2024

• Training spec file:

random_seed: 42
lpr_config {
  hidden_units: 512
  max_label_length: 12
  arch: "baseline"
  nlayers: 18 #setting nlayers to be 10 to use baseline10 model
}
training_config {
  batch_size_per_gpu: 16
  num_epochs: 250
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 1e-6
    max_learning_rate: 1e-5
    soft_start: 0.001
    annealing: 0.5
  }
  }
  regularizer {
    type: L2
    weight: 5e-4
  }
}
eval_config {
  validation_period_during_training: 50
  batch_size: 1
}
augmentation_config {
    output_width: 120
    output_height: 48
    output_channel: 3
    max_rotate_degree: 5
    rotate_prob: 0.5
    gaussian_kernel_size: 5
    gaussian_kernel_size: 7
    gaussian_kernel_size: 15
    blur_prob: 0.5
    reverse_color_prob: 0.5
    keep_original_prob: 0.3
}
dataset_config {
  data_sources: {
    label_directory_path: "/data/tlt-experiments/data/ocr_dataset/train/label"
    image_directory_path: "/data/tlt-experiments/data/ocr_dataset/train/image"
  }
  characters_list_file: "/data/tlt-experiments/lprnet/specs/vn_lp_characters.txt"
  validation_data_sources: {
    label_directory_path: "/data/tlt-experiments/data/ocr_dataset/val/label"
    image_directory_path: "/data/tlt-experiments/data/ocr_dataset/val/image"
  }
}
  • vn_lp_characters.txt:
0
1
2
3
4
5
6
7
8
9
A
B
C
D
E
F
G
H
I
J
K
L
M
N
P
Q
R
S
T
U
V
W
X
Y
Z
-
.

I have a dataset which have both single line and double line of license plate:

If I use above dataset and train from the US pretrain model, the accuracy is low (~66%). Is there anyway that I can achieve a higher accuracy, so I can use that model in my GStreamer pipeline for inference with Triton Inference Server.

My previous post in Deepstream forum: https://forums.developer.nvidia.com/t/custom-processor-on-2-line-license-plate-recognition/335273?u=hungtv

3

Actually I already saw that topic and How to train LPRnet for license plates with two lines.
For 2-lines license plate, the discussion can be found in How to train LPRnet for license plates with two lines.
Also, as Levi confirmed, the idea can work on his side.

4

Hi @Morganh,

The first line and second line model have been trained and converted to TensorRT format so I can import them into Triton Inference Server. The hardest part is how to crop each line on the detected plate (if that plate is double line plate) to recognize them, and concatenate their output. The single line plate model works perfectly with gst-launch-1.0 command (since my custom backend only work with gst pipeline), but I didn’t find a way to work on double line plate. A license plate recognition model can use on both type of license plate is perfect.

5

There is no crop; the model will handle this (extract from each line on full plate). Inference/Training uses the same image (full plate) twice, one for each line, and it will work.
What you can’t achieve is character-level localization. If you need that functionality, you’ll need a different approach. However, if you only need OCR, you can crop the license plate from the LPD detection and extract the text using LPR.

6

Yes, as mentioned from Levi, there is no crop. The model will handle this.