Test VehicleMakenet on CompCars dataset

Hi!
I am testing VehicleMakeNet(unpruned) model on CompCars sv dataset, where instead of model target label brand target label is used. For instance “audi Q3” target label is replaced by “audi”.
When launching evaluate function with config(below) the model scores 64.3 accuracy and 71.36 accuracy on cropped images. However, on Nvidia proprietary dataset the model hits 91 accuracy. I wonder what may cause such difference in accuracy and would appreciate any comments, concerning the issue.

model_config {
arch: “resnet”,
n_layers: 18
use_batch_norm: true
all_projections: true
freeze_blocks: 0
freeze_blocks: 1
input_image_size: “3,224,224”
}
eval_config {
eval_dataset_path: “compcars_crops/test/”
model_path: “tlt_vehiclemakenet_vunpruned_v1.0/resnet18_vehiclemakenet.tlt”
top_k: 1
batch_size: 256
n_workers: 8
}

How did you prepare the CompCars dataset for evaluation? This dataset contains 163 classes. Have you filtered the classes?

Please note that in VehicleMakeNet(https://ngc.nvidia.com/catalog/models/nvidia:tlt_vehiclemakenet), it supports only 20 classes.

This model classifies following cars: Acura, Audi, BMW, Chevrolet, Chrysler, Dodge, Ford, GMC, Honda, Hyundai, Infiniti, Jeep, Kia, Lexus, Mazda, Mercedes, Nissan, Subaru, Toyota, and Volkswagen.

I used surveillance-nature data of CompCars dataset, it contains 281 makes. I have filtered them and use only marks which belong to one of the 20 possible brands. There are 17 of such brands(except chevrolet and gmc) and 7137 pictures for them.

There is no update from you for a period, assuming this is not an issue any more.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Could you please list these 17 brands?