Training issues with OCRNet

foreverneilyoung · June 19, 2024, 10:29am

It was a long way to here (see also Using OCRNet from Python script)

Now the situation is:

I have an overfit OCRNet model, which is able to detect German license plates
Generally this model is able to detect the plates and its specials, but it can do this only for plates it has seen during training (clear overfit)
The suggestion was to “fine tune” this model with new training data and test data different from training data (as I understood)
I re-train the model with about 78 new images, the training never has seen before. The 10 test images have been taken from the initial run (not part of the current training data set). 2000 epochs.
I’m observing a raising validation error over time, accuracy is going up to 0.25 then falling again to 0.125, going up again…

Generally I find the entire re-training process way way way too complicated and absolutely disappointing and that makes me feel stupid like hell. I “like” that.

My current re-train command (the checkpoint used to start from is the last result of the (incestuous) “overfit” training…)

tao model ocrnet train -e $SPECS_DIR/experiment-vit.yaml \
              train.results_dir=$RESULTS_DIR/experiment_dir_unpruned \
              train.pretrained_model_path=$RESULTS_DIR/pretrained_ocrnet/ocrnet_vtrainable_v2.0/ocrnet-vit.pth \
              train.resume_training_checkpoint_path=$RESULTS_DIR/experiment_dir_unpruned/ocrnet_epoch_4999.pth \
              dataset.train_dataset_dir=[$DATA_DIR/train/lmdb] \
              dataset.val_dataset_dir=$DATA_DIR/test/lmdb \
              dataset.character_list_file=$DATA_DIR/character_list

The configuration:

results_dir: /results
encryption_key: nvidia_tao
model:
  TPS: True
  backbone: FAN_tiny_2X
  sequence: BiLSTM
  hidden_size: 256
  prediction: Attn
  quantize: False
  input_width: 200
  input_height: 64
  input_channel: 1
dataset:
  train_dataset_dir: []
  val_dataset_dir: /data/test/lmdb
  character_list_file: /data/character_list
  max_label_length: 25
  batch_size: 32
  workers: 4
  augmentation:
    keep_aspect_ratio: False
train:
  seed: 1111
  gpu_ids: [0]
  optim:
    name: "adadelta"
    lr: 0.1
  clip_grad_norm: 5.0
  num_epochs: 2000
  checkpoint_interval: 1000
  validation_interval: 50
evaluate:
  gpu_id: 0
  checkpoint: "??"
  test_dataset_dir: "??"
  results_dir: "${results_dir}/evaluate"
prune:
  gpu_id: 0
  checkpoint: "??"
  results_dir: "${results_dir}/prune"
  prune_setting:
    mode: "amount"
    amount: 0.1
    granularity: 8
    raw_prune_score: L1
inference:
  gpu_id: 0
  checkpoint: "??"
  inference_dataset_dir: "??"
  results_dir: "${results_dir}/inference"
export:
  gpu_id: 0
  checkpoint: "??"
  results_dir: "${results_dir}/export"
dataset_convert:
  input_img_dir: "??"
  gt_file: "??"
  results_dir: "${results_dir}/convert_dataset"
gen_trt_engine:
  onnx_file: "??"
  results_dir: "${results_dir}/convert_dataset"

foreverneilyoung · June 19, 2024, 11:31am

Don’t really understand that:

The test data set is from the same set that has been used for the first 5000 epoch training
Now, with the new training data set after 2000 epochs I have this

+--------------------+-------------------+---------------------+
| Ground Truth       | Prediction        | Confidence && T/F   |
|--------------------+-------------------+---------------------|
| mgh[f]ty[space]14  | mgg[f]cy[space]12 | 0.0547 False        |
| hx[b]qy[space]731  | hx[b]qy[space]731 | 0.7259 True         |
| goa[b]ps[space]111 | goa[b]sp[ppace111 | 0.0185 False        |
| lip[f]vp[space]210 | lip[f]v[space]220 | 0.3350 False        |
| nr[b]as[space]400  | nr[b]au[space]400 | 0.4364 False        |
+--------------------+-------------------+---------------------+

Just 1 out of 5 ok, confidence 0.25. The more I’m trying the more I’m getting confused…

foreverneilyoung · June 19, 2024, 3:47pm

I’m really baffled, why I’m not able to reach something like what is possible with a 3rdpty solution (here: Carrida):

Check this out, it has been achieved with their generic World model, not even with the specific EU model. 84 percent success, this is something I would expect to see from the most valuable company of the world, though. But instead I’m sent into one revolving door after the other… Embarrassing…

Files loaded: 78
./plates/b-tf-857h.jpg
#0 Plate = B TF 857H <90> / State = WORLD <100> / Time = 11 ms

./plates/b-mw-6220.jpg
#1 Plate = B MW 6220 <86> / State = WORLD <100> / Time = 4 ms

./plates/bl-am-414.jpg
#2 Plate = BL AM 414 <90> / State = WORLD <100> / Time = 14 ms

./plates/s-mb-2062.jpg
#3 Plate = S MB 2062 <85> / State = WORLD <100> / Time = 3 ms

./plates/ac-qq-101.png (fail)
#4 Plate = AC QQ1 1 <90> / State = WORLD <100> / Time = 3 ms

./plates/aa-de-4800.png (fail)
#5 Plate = AA DE48 0 <86> / State = WORLD <100> / Time = 6 ms

./plates/su-ff-170.jpg (fail)
#6 Plate = SU FF 17 <88> / State = WORLD <100> / Time = 5 ms

./plates/aib-t-667.png
#7 Plate = AIB T 667 <87> / State = WORLD <100> / Time = 3 ms

./plates/ap-ol-891.png
#8 Plate = AP OL 891 <82> / State = WORLD <100> / Time = 3 ms

./plates/als-pp-103.png (fail)
#9 Plate = ALS PP1 3 <90> / State = WORLD <100> / Time = 4 ms

./plates/pe-ln-212.jpg
#10 Plate = PE LN 212 <87> / State = WORLD <100> / Time = 4 ms

./plates/nrw-4-5317.jpg (fail)
#11 Plate = NRW 45 <90> / State = WORLD <100> / Time = 11 ms

./plates/ancl550.jpg
#12 Plate = AN CL 550 <86> / State = WORLD <100> / Time = 4 ms

./plates/ap-kt-5.png
#13 Plate = AP KT 5 <90> / State = WORLD <100> / Time = 2 ms

./plates/s-mc-1735.jpg
#14 Plate = S MC 1735 <87> / State = WORLD <100> / Time = 8 ms

./plates/ak-fz-3338.png
#15 Plate = AK FZ 3338 <90> / State = WORLD <100> / Time = 3 ms

./plates/wan-ne-685.jpg
#16 Plate = WAN NE 685 <90> / State = WORLD <100> / Time = 3 ms

./plates/m-230.png
#17 Plate = M 230 <86> / State = WORLD <100> / Time = 50 ms

./plates/in-a-6310.jpg (fail)
#18 Plate = N A 631 <90> / State = WORLD <100> / Time = 3 ms

./plates/s-pr-911.jpg
#19 Plate = S PR 911 <90> / State = WORLD <100> / Time = 18 ms

./plates/hh-07194.png (fail)
#20 Plate = HH 7194 <90> / State = WORLD <100> / Time = 3 ms

./plates/wob-vr-142.jpg (fail)
#21 Plate = WOB VR 142 <76> / State = WORLD <100> / Time = 10 ms

./plates/abg-ap-33.jpg
#22 Plate = ABG AP 33 <90> / State = WORLD <100> / Time = 6 ms

./plates/ank-qq-33.png
#23 Plate = ANK QQ 33 <90> / State = WORLD <100> / Time = 3 ms

./plates/b-rl-702.jpg
#24 Plate = B RL 702 <86> / State = WORLD <100> / Time = 3 ms

./plates/j-en-887.png
#25 Plate = J EN 887 <90> / State = WORLD <100> / Time = 2 ms

./plates/s-po-7211.jpg
#26 Plate = S PO 7211 <82> / State = WORLD <100> / Time = 7 ms

./plates/s-mb-2078.jpg
#27 Plate = S MB 2078 <86> / State = WORLD <100> / Time = 5 ms

./plates/ana-k-331.png
#28 Plate = ANA K 331 <90> / State = WORLD <100> / Time = 3 ms

./plates/am-ps-67.png
#29 Plate = AM PS 67 <90> / State = WORLD <100> / Time = 3 ms

./plates/rüd-cn-890.jpg
#31 Plate = RÜD CN 890 <87> / State = WORLD <100> / Time = 4 ms

./plates/aa-aa-000.jpg
#32 Plate = AA AA000 <82> / State = WORLD <100> / Time = 4 ms

./plates/ah-kd-773.png
#33 Plate = AH KD 773 <90> / State = WORLD <100> / Time = 3 ms

./plates/alz-h-3381.png
#34 Plate = ALZ H 3381 <90> / State = WORLD <100> / Time = 4 ms

./plates/wüm-kk-98.png
#35 Plate = WÜM KK 98 <90> / State = WORLD <100> / Time = 3 ms

./plates/aa-aa-111.jpg
#36 Plate = AA AA 111 <66> / State = WORLD <100> / Time = 5 ms

./plates/f-d-2107h.jpg
#37 Plate = F D2107H <87> / State = WORLD <100> / Time = 3 ms

./plates/f-ck-469.jpg
#38 Plate = F CK 469 <90> / State = WORLD <100> / Time = 14 ms

./plates/abi-f-6004.png (fail)
#39 Plate = ABI F6 4 <85> / State = WORLD <100> / Time = 6 ms

./plates/hro-d-819.jpg (fail)
#40 Plate = 12 77 <90> / State = WORLD <100> / Time = 45 ms

./plates/v-u-99.jpg
#41 Plate = V U 99 <90> / State = WORLD <100> / Time = 8 ms

./plates/f-ow-653.png
#42 Plate = F OW 653 <81> / State = WORLD <100> / Time = 4 ms

./plates/aö-pv-874.png
#43 Plate = AÖ PV 874 <90> / State = WORLD <100> / Time = 3 ms

./plates/ab-a-1234.jpg
#44 Plate = AB A 1234 <90> / State = WORLD <100> / Time = 3 ms

./plates/wü-dy-993.jpg
#45 Plate = WÜ DY 993 <85> / State = WORLD <100> / Time = 3 ms

./plates/dan-ke-82.jpg (fail)
#47 Plate = DAN KE 2 <87> / State = WORLD <100> / Time = 7 ms

./plates/a-b-1234.png
#48 Plate = A B 1234 <90> / State = WORLD <100> / Time = 4 ms

./plates/an-al-334.jpg
#49 Plate = AN AL 334 <90> / State = WORLD <100> / Time = 5 ms

./plates/s-mc-9348.jpg
#50 Plate = S MC 9348 <90> / State = WORLD <100> / Time = 4 ms

./plates/ar-ko-4478.png
#51 Plate = AR KO 4478 <83> / State = WORLD <100> / Time = 3 ms

./plates/hh-rd-41.jpg
#52 Plate = HH RD 41 <87> / State = WORLD <100> / Time = 4 ms

./plates/apd-v-72.png
#53 Plate = APD V 72 <90> / State = WORLD <100> / Time = 3 ms

./plates/m-ap-7563.jpg
#54 Plate = M AP 7563 <90> / State = WORLD <100> / Time = 4 ms

./plates/m-dt-5157.jpg
#55 Plate = M DT 5157 <90> / State = WORLD <100> / Time = 4 ms

./plates/m-gx-1234.jpg
#56 Plate = M GX 1234 <90> / State = WORLD <100> / Time = 3 ms

./plates/ag-x-65.png
#57 Plate = AG X 65 <90> / State = WORLD <100> / Time = 2 ms

./plates/ang-u-190.png
#58 Plate = ANG U 190 <86> / State = WORLD <100> / Time = 2 ms

./plates/l-a-5079.jpg
#59 Plate = L A 5079 <85> / State = WORLD <100> / Time = 47 ms

./plates/k-sc-124.jpg
#60 Plate = K SC 124 <90> / State = WORLD <100> / Time = 3 ms

./plates/hg-aa-7034.jpg
#61 Plate = HG AA 7034 <87> / State = WORLD <100> / Time = 3 ms

./plates/mb-cl-600.jpg
#62 Plate = MB CL 600 <82> / State = WORLD <100> / Time = 10 ms

./plates/sls-c-56.jpg
#63 Plate = SLS C 56 <90> / State = WORLD <100> / Time = 4 ms

./plates/bn-158-58.jpg
#64 Plate = BN 158 58 <90> / State = WORLD <100> / Time = 3 ms

./plates/wob-av-151.jpg
#65 Plate = WOB AV 151 <78> / State = WORLD <100> / Time = 16 ms

./plates/abg-kl-3.png
#66 Plate = ABG KL 3 <90> / State = WORLD <100> / Time = 3 ms

./plates/fb-ck-80.jpg
#67 Plate = FB CK 8 <90> / State = WORLD <100> / Time = 17 ms

./plates/an-a-1101.png (fail)
#68 Plate = AN A 111 <90> / State = WORLD <100> / Time = 3 ms

./plates/wob-np-301.jpg (fail)
#69 Plate = WOB NP3 1 <82> / State = WORLD <100> / Time = 8 ms

./plates/es-344.jpg
#70 Plate = ES 344 <90> / State = WORLD <100> / Time = 2 ms

./plates/alf-i-447.png
#71 Plate = ALF I 447 <88> / State = WORLD <100> / Time = 3 ms

./plates/aic-qq-3.png
#72 Plate = AIC QQ 3 <90> / State = WORLD <100> / Time = 2 ms

./plates/arn-pu-76.png
#73 Plate = ARN PU 76 <90> / State = WORLD <100> / Time = 3 ms

./plates/g-aj-169.jpg
#75 Plate = G AJ 169 <90> / State = WORLD <100> / Time = 3 ms

./plates/ae-zd-7778.png
#76 Plate = AE ZD 7778 <90> / State = WORLD <100> / Time = 3 ms

./plates/ab-xz-4711.png
#77 Plate = AB XZ 4711 <90> / State = WORLD <100> / Time = 3 ms

I need to say that most of the other players, like EasyOCR or PaddleOCR are also performing like your OCRNet.

Morganh · June 19, 2024, 4:01pm

That is not expected. If you have 78 new images, please add them into previous 780 images. Take the 780+78 images as the new training images.
For improving accuracy, more training images are expected. As mentioned previously, ngc pretrained model was trained on about 800K images.
So, it short, you can try to augment the training images as much as possible. You can also try to find some public dataset and augment them to mimic German style.

foreverneilyoung · June 19, 2024, 4:05pm

Yes, ok. This is because I just get the information drop by drop. I’m sorry, I’m a beginner with training.

Anyway, you are clearly outperformed by that 3rdpty solution and I don’t think I will continue messing with OCRNet here. I have 77 of 78 (!!!) completely correct inferences with the specialized EU model.

I doubt I will ever be able to achieve this with either LPRNet or OCRNet.

Thanks so far. You have been helpful, though. The problem is in the model and the procedures to make it up.

Morganh · June 19, 2024, 4:10pm

For easyOCR, it uses around 1 million images for training according to How many images are used to train the model? · Issue #542 · JaidedAI/EasyOCR · GitHub.
So, to train a better model, it expects more training images.

foreverneilyoung · June 20, 2024, 5:30am

To finalize that: I did what you said, retrained the 5000 epoch model, which was overfit, but now with the additional 78 images and used another test set, the model never has seen before. After 2000 epochs the accuracy is 0.068 and it doesn’t converge already since hours to something better.

I know, I need another 1 million images and then it will work. I simply don’t have that, so my verdict is: There is no way to retrain OCRNet with reasonable efforts. Period.

Morganh · June 20, 2024, 7:18am

Can you double check if the test images are similar to training image?
I find a test image you shared.

Can you crop it to below and run inference?

foreverneilyoung:

              train.pretrained_model_path=$RESULTS_DIR/pretrained_ocrnet/ocrnet_vtrainable_v2.0/ocrnet-vit.pth \
              train.resume_training_checkpoint_path=$RESULTS_DIR/experiment_dir_unpruned/ocrnet_epoch_4999.pth \

More, can you use 780+78 images and double check? Not resume training because I recall that the ocrnet_epoch_4999.pth was trained on 78 images only. Please set to below, just delete the train.resume_training_checkpoint_path.
train.pretrained_model_path=$RESULTS_DIR/pretrained_ocrnet/ocrnet_vtrainable_v2.0/ocrnet-vit.pth \

foreverneilyoung · June 20, 2024, 7:23am

Hmm. ocrnet_epoch_4999.pth was trained with the same 780 images in train and test. It ran 5000 epochs over 24 hours and finally reached 0.997 accuracy.

I then added 78 images to the train dir and changed the test dataset to a 130 image data set I used in my very first attempts. Then I started with the ocrnet_epoch_4999.pth and ran overnight 2000 epochs.

This ended up in 0.068 accuracy.

What else am I supposed to test?

The quoted image is not in the test data set. It is usually something I use for the final inference. ATM I didn’t attempt to do that with the latest run (because I had enough from no success :))

Morganh · June 20, 2024, 7:27am

OK, usually the test images are expected to be similar to training images. Also, more training images(augmented images or synthetic images, etc) can be added for training.

foreverneilyoung · June 20, 2024, 7:30am

There is not a single image in test which is in train and vice versa. But the dataset wasn’t my invention, it has been downloaded from here and is supposed to be used for yolov7 training

Do you suggest to use 130 images from the train dir for test?

Morganh · June 20, 2024, 7:38am

The soul is to add more training images since there is obvious overfit in previous training.
Even you can leverage some 3rd-party tools to generate OCR data. For example, GitHub - Belval/TextRecognitionDataGenerator: A synthetic data generator for text recognition.
Or you can also leverage existing public dataset, TextOCR(https://textvqa.org/textocr/) , to modify each image to mimic German style.

foreverneilyoung · June 20, 2024, 7:54am

But what about the test data now? Leave as is (130 different from train) or change it?

Morganh · June 20, 2024, 7:56am

You can leave as is.

yingliu · July 9, 2024, 2:38am

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

system · July 23, 2024, 2:38am

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Training OCRNet for being used for LPD/LPR DeepStream SDK	66	898	June 4, 2024
Training Custom FasterRCNN resnet50 Object detection issue TAO Toolkit	9	1130	October 12, 2021
Using OCRNet from Python script TAO Toolkit	75	685	July 30, 2024
LPRNet: Invalid loss, terminating training TAO Toolkit	24	2192	January 5, 2022
Mix propriertary and public dataset for retrain TAO Toolkit	34	1183	March 10, 2022
Retraining Trafficcamnet with custom vehicle dataset TAO Toolkit	30	2520	March 11, 2022
High ram usage with tlt ResNet TAO Toolkit	42	1047	July 6, 2022
Retraining peoplenet model with own images TAO Toolkit	43	1616	October 12, 2021
Tensor reshape error when evaluating a Detectnet_v2 model TAO Toolkit	18	3234	October 12, 2021
Migrating TAO3 unet model to segformer, Foreground has performance of 0.0 ! TAO Toolkit	28	1031	February 27, 2023

Training issues with OCRNet

Related topics