Test VehicleMakenet on CompCars dataset

Hi!
I am testing VehicleMakeNet(unpruned) model on CompCars sv dataset, where instead of model target label brand target label is used. For instance “audi Q3” target label is replaced by “audi”.
When launching evaluate function with config(below) the model scores 64.3 accuracy and 71.36 accuracy on cropped images. However, on Nvidia proprietary dataset the model hits 91 accuracy. I wonder what may cause such difference in accuracy and would appreciate any comments, concerning the issue.

model_config {
arch: “resnet”,
n_layers: 18
use_batch_norm: true
all_projections: true
freeze_blocks: 0
freeze_blocks: 1
input_image_size: “3,224,224”
}
eval_config {
eval_dataset_path: “compcars_crops/test/”
model_path: “tlt_vehiclemakenet_vunpruned_v1.0/resnet18_vehiclemakenet.tlt”
top_k: 1
batch_size: 256
n_workers: 8
}

How did you prepare the CompCars dataset for evaluation? This dataset contains 163 classes. Have you filtered the classes?

Please note that in VehicleMakeNet(NVIDIA NGC), it supports only 20 classes.

This model classifies following cars: Acura, Audi, BMW, Chevrolet, Chrysler, Dodge, Ford, GMC, Honda, Hyundai, Infiniti, Jeep, Kia, Lexus, Mazda, Mercedes, Nissan, Subaru, Toyota, and Volkswagen.