Duplicate chracter in LPR Retraining using TLT-V3

Hi Team,

I have Re-trained LPRnet model on my custom data-set and test it but I am getting repeated character like my actual number plate is GJ06PB8703 but got GJ006PB88703 result. I am getting this issue on almost 5-10% of my testing data. How can I avoid this issue ? can someone please help ?

Actual - Predicted
GJ06JM5411- GJJ06JM5411
GJ06JM4411- GGJJ06JM4411
GJ06JM5422- GGJ06JMM5422

Thanks.

How about the validation result in the training log? Can you share the log?

On Validation data I am getting around 82-83% accuracy.

below are the logs.

epoch,accuracy,loss,lr
0,nan,2.587995486705344,1e-05
1,nan,1.225582666176917,1e-05
2,nan,0.8701144526878133,1e-05
3,nan,0.7836750434392018,1e-05
**4,0.8195488721804511,0.7131261605383624,1e-05**
5,nan,0.6340069563071916,1e-05
6,nan,0.5820798972014113,1e-05
7,nan,0.5220499521429425,1e-05
8,nan,0.5324309563618355,1e-05
**9,0.8270676691729323,0.40411772324460365,1e-05**
10,nan,0.39945308196213,1e-05
11,nan,0.39153773161414235,1e-05
12,nan,0.38394406497754857,1e-05
13,nan,0.3394874894683475,1e-05
epoch,accuracy,loss,lr
0,nan,2.587995486705344,1e-05
1,nan,1.225582666176917,1e-05
2,nan,0.8701144526878133,1e-05
3,nan,0.7836750434392018,1e-05
4,0.8195488721804511,0.7131261605383624,1e-05
5,nan,0.6340069563071916,1e-05
6,nan,0.5820798972014113,1e-05
7,nan,0.5220499521429425,1e-05
8,nan,0.5324309563618355,1e-05
**9,0.8270676691729323,0.40411772324460365,1e-05**
10,nan,0.39945308196213,1e-05
11,nan,0.39153773161414235,1e-05
12,nan,0.38394406497754857,1e-05
13,nan,0.3394874894683475,1e-05
14,0.8345864661654135,0.30403443910699723,1e-05
15,nan,0.3463592989229161,1e-05
16,nan,0.3108247216831219,1e-05
17,nan,0.27171264855994326,1e-05
18,nan,0.29416644960135174,1e-05
19,0.8270676691729323,0.26408787419475394,1e-05
20,nan,0.286938238503207,1e-05
21,nan,0.25302603460813783,1e-05
22,nan,0.2811309878571877,1e-05
23,nan,0.25566658006845705,1e-05
24,0.8270676691729323,0.25163370218859205,1e-05
25,nan,0.23378919742114762,9.120108e-06
26,nan,0.24462735261393848,8.317637e-06
27,nan,0.2586062539720093,7.5857756e-06
28,nan,0.21468088132152233,6.9183097e-06
29,0.8345864661654135,0.22113247379269077,6.309573e-06

15,nan,0.3463592989229161,1e-05
16,nan,0.3108247216831219,1e-05
17,nan,0.27171264855994326,1e-05
18,nan,0.29416644960135174,1e-05
**19,0.8270676691729323,0.26408787419475394,1e-05**
20,nan,0.286938238503207,1e-05
21,nan,0.25302603460813783,1e-05
22,nan,0.2811309878571877,1e-05
23,nan,0.25566658006845705,1e-05
**24,0.8270676691729323,0.25163370218859205,1e-05**
25,nan,0.23378919742114762,9.120108e-06
26,nan,0.24462735261393848,8.317637e-06
27,nan,0.2586062539720093,7.5857756e-06
28,nan,0.21468088132152233,6.9183097e-06
**29,0.8345864661654135,0.22113247379269077,6.309573e-06**

Thanks

So, can you run inference to get the predicted result of your validation dataset? Are the repeated ones almost 5-10% ?

NO, In the validation data-set it was not 5-10%.
I had 133 images in the validation data and 111 was correct the and remaining were wrong as an it was classifying P as R but there were no duplicate character.

How we can overcome this on the test data ?

Could you share the picture of below ? I am afraid it is necessary to add more training images which have similar feature to them.
GJ06JM5411
GJ06JM4411
GJ06JM5422

Thanks for the info. Can you please find below 4 images in your test data and share here?
GJ06PB8703
GJ06JM5411
GJ06JM4411
GJ06JM5422

May I know how did you get the inference result? With lprnet inference xxx ?

I got this result using python script.

Could you please run inference with lprnet inference xxx too?

Sure. I will run inference using lprnet and will update you.

Hi @Morganh,

I ran the inference with lprnet inference there I am not getting duplicate entries of character.

Can you please tell me where are the gaps in the script.
Below is my script:

import os
import time

import cv2
#import matplotlib.pyplot as plt
import numpy as np
import pycuda.autoinit
import pycuda.driver as cuda
import tensorrt as trt
from PIL import Image
import pdb
import codecs
import glob
import datetime
import shutil

NPR_LABEL = {'0':'0','1':'1','2':'2','3':'3','4':'4','5':'5','6':'6','7':'7','8':'8','9':'9','10':'A','11':'B','12':'C','13':'D','14':'E','15':'F','16':'G','17':'H','18':'I','19':'J','20':'K','21':'L','22':'M','23':'N','24':'P','25':'Q','26':'R','27':'S','28':'T','29':'U','30':'V','31':'W','32':'X','33':'Y','34':'z'}

class HostDeviceMem(object):
    def __init__(self, host_mem, device_mem):
        self.host = host_mem
        self.device = device_mem

    def __str__(self):
        return "Host:\n" + str(self.host) + "\nDevice:\n" + str(self.device)

    def __repr__(self):
        return self.__str__()


def load_engine(trt_runtime, engine_path):
    with open(engine_path, "rb") as f:
        engine_data = f.read()
    engine = trt_runtime.deserialize_cuda_engine(engine_data)
    return engine

# Allocates all buffers required for an engine, i.e. host/device inputs/outputs.
def allocate_buffers(engine, batch_size=-1):
    inputs = []
    outputs = []
    bindings = []
    stream = cuda.Stream()
    for binding in engine:
        # pdb.set_trace()
        size = trt.volume(engine.get_binding_shape(binding)) * batch_size
        dtype = trt.nptype(engine.get_binding_dtype(binding))
        # Allocate host and device buffers
        host_mem = cuda.pagelocked_empty(size, dtype)
        device_mem = cuda.mem_alloc(host_mem.nbytes)
        # Append the device buffer to device bindings.
        bindings.append(int(device_mem))
        # Append to the appropriate list.
        if engine.binding_is_input(binding):
            inputs.append(HostDeviceMem(host_mem, device_mem))
            # print(f"input: shape:{engine.get_binding_shape(binding)} dtype:{engine.get_binding_dtype(binding)}")
        else:
            outputs.append(HostDeviceMem(host_mem, device_mem))
            # print(f"output: shape:{engine.get_binding_shape(binding)} dtype:{engine.get_binding_dtype(binding)}")
    return inputs, outputs, bindings, stream



def do_inference(context, bindings, inputs, outputs, stream, batch_size=1):
    # Transfer input data to the GPU.
    [cuda.memcpy_htod_async(inp.device, inp.host, stream) for inp in inputs]
    # Run inference.
    context.execute_async(
        batch_size=batch_size, bindings=bindings, stream_handle=stream.handle
    )
    # Transfer predictions back from the GPU.
    [cuda.memcpy_dtoh_async(out.host, out.device, stream) for out in outputs]
    # Synchronize the stream
    stream.synchronize()
    # Return only the host outputs.
    return [out.host for out in outputs]

def post_processing(label_ids):
    # iterate label using label ids
    number = ''
    i = 0
    for label in label_ids[0]:
        if str(label) != '35':
            number = number + NPR_LABEL[str(label)]
            print("Number : {}  Accuracy : {}".format(NPR_LABEL[str(label)],label_ids[1][i]))
        i += 1
    return number

def model_loading(trt_engine_path):
    # TensorRT logger singleton
    os.environ["CUDA_VISIBLE_DEVICES"] = "1"
    TRT_LOGGER = trt.Logger(trt.Logger.WARNING)
    # trt_engine_path = "/opt/smarg/surveillance_gateway_prod/surveillance_ai_model/x86_64/Secondary_NumberPlateClassification/lpr_us_onnx_b16.engine"

    trt_runtime = trt.Runtime(TRT_LOGGER)
    # pdb.set_trace()
    trt_engine = load_engine(trt_runtime, trt_engine_path)
    # Execution context is needed for inference
    context = trt_engine.create_execution_context()
    # NPR input shape
    input_shape = (1,3,48,96)
    context.set_binding_shape(0, input_shape)
    # This allocates memory for network inputs/outputs on both CPU and GPU
    inputs, outputs, bindings, stream = allocate_buffers(trt_engine)
    return inputs, outputs, bindings, stream, context

trt_engine_path = "number_plate_classification_b8_fp16.engine"
inputs, outputs, bindings, stream, context = model_loading(trt_engine_path)


# pdb.set_trace()
# image = [cv2.imread("/home/smarg/Downloads/Images/resized/img/IMG_20210719_160022_cropped_batch_code_image_imgGB3_BATO007_.jpg")]

# Run inference on folder 
image_folder_path = "/home/smarg/Documents/Pritam/TRT-INFER-NPR/NPR_TEST/Cropped/"
output_folder_path = "/home/smarg/Documents/Pritam/TRT-INFER-NPR/OutPutImg/Cropped/"
image_count = 0
start_time = datetime.datetime.now()
for image_path in glob.glob(image_folder_path + "*.jpg"):
    
    print("Image name :",image_path)
    image = [cv2.imread(image_path)]

    image = np.array([(cv2.resize(img, ( 96 , 48 )))/ 255.0 for img in image], dtype=np.float32)
    image= image.transpose( 0 , 3 , 1 , 2 )

    np.copyto(inputs[0].host, image.ravel())

    output = do_inference(context, bindings=bindings, inputs=inputs, outputs=outputs, stream=stream)
    number = post_processing(output)
    image_count += 1
    shutil.copy(image_path,output_folder_path+str(image_count)+"_"+number+".jpg")
    print("Number : ",number)

end_time = datetime.datetime.now()
total_time = end_time - start_time

print("Total image processed : {} Total Time : {} ".format(image_count,total_time))

Training Config :

random_seed: 42
lpr_config {
  hidden_units: 512
  max_label_length: 12
  arch: "baseline"
  nlayers: 18 #setting nlayers to be 10 to use baseline10 model
}
training_config {
  batch_size_per_gpu: 4
  num_epochs: 50
  learning_rate {
  soft_start_annealing_schedule {
    min_learning_rate: 1e-6
    max_learning_rate: 1e-5
    soft_start: 0.001
    annealing: 0.5
  }
  }
  regularizer {
    type: L2
    weight: 5e-4
  }
}
eval_config {
  validation_period_during_training: 5
  batch_size: 1
}
augmentation_config {
    output_width: 96
    output_height: 48
    output_channel: 3
    keep_original_prob: 0.3
    transform_prob: 0.5
    rotate_degree: 5
}
dataset_config {
  data_sources: {
    label_directory_path: "/workspace/tlt-experiments/data/train/label"
    image_directory_path: "/workspace/tlt-experiments/data/train/image"
  }
  characters_list_file: "/workspace/examples/lprnet/specs/us_lp_characters.txt"
  validation_data_sources: {
    label_directory_path: "/workspace/tlt-experiments/data/val/label"
    image_directory_path: "/workspace/tlt-experiments/data/val/image"
  }
}

When you run lprnet inference , did you run inference against tlt model or trt engine?

Hi @Morganh

I had run lprnet inference against tlt file.

OK, how about inference result against the same number_plate_classification_b8_fp16.engine ?
More, please generate trt engine via below as well.

-p image_input,1x3x48x96,1x3x48x96,1x3x48x96

Previously, you generated one engine for b8. Please test b1 too.

Thanks @Morganh.
Will generate engine with batch-size 1 and will test.

lprnet inference can also run inference against trt engine. So, please run lprnet inference too.

with batch size one again getting duplicate entries of character.

will update you about the results on engine with lprnet inference.

Thanks

More, please note that
35 2 2 35 will be decoded to 2
35 2 2 2 35 will be decoded to 2 too.

Thanks @Morganh.