We have a market project targeting object detection scenarios, where the customer provided multiple scenario samples. Currently, we are using the TAO tool with the built-in GD-TINY CV model for fine-tuning. We’ve found that in some scenarios, the F1 score after testing reaches over 80, but in other scenarios, after the same level of fine-tuning, the F1 score is only around 10. We’re quite confused and would like to understand what the issue is. Since it’s not convenient to disclose details about the market project, we hope to organize a dedicated meeting to help us review this.
How many dataset did you run training? Please check the dataset distribution when you use TAO to do Grounding DINO finetuning. Especially for the low F1 scenario you have found, is this kind of data less than other scenarios? You can do some oversampling for this kind of data to do more experiments. More, please try to set a lower learning rate to do more experiments. More reference can be found in tao_tutorials/notebooks/tao_launcher_starter_kit/grounding_dino/specs/train.yaml at main · NVIDIA/tao_tutorials · GitHub.
Please share the training spec file and log as well.
Hi @1741027687
If you have any private yaml, json or dataset, please click my icon to send a private forum message to me. Thanks a lot!
From the dataset you shared,
- Two classes(N、S) have more data
- But E, H, M, O, P, Z class has obvious less data.
- And A、G、V、W have none of training images but have test images.
That means the training dataset distribution is not similar to test dataset distribution.
The dataset is obviously imbalance.
Suggest you to do below
- 1st priority: Add the training dataset to the classes which has none of training dataset.
- 2nd priority: Do oversampling or data augmentation against the classes which has less data.
- 3rd priority: Tune learning rate, batch size, freeze, etc。 Set
lr: 1e-4or5e-5. Setlr_backbone: 1e-5or5e-6. Setbatch_size: 4 - 4th priority: Change to larger backbone
Two classes(N、S) have more data
But E, H, M, O, P, Z class has obvious less data.
And A、G、V、W have none of training images but have test images.
—reply1 : It’s not that there is no data; it’s just that these are not easy to calculate. In fact, we have sample training, but the results are not obtained directly from the samples. Instead, we calculate them based on business logic, so they are derived indirectly.
1st priority: Add the training dataset to the classes which has none of training dataset.
—reply: the same with reply1
2nd priority: Do oversampling or data augmentation against the classes which has less data.
—reply : We previously ran some experiments involving rotations, resizing, cropping, etc., and found that they weren’t very useful. We’re not sure if there was something wrong with our approach. One question we have is: how many sample data points should we add for each category?"
3rd priority: True learning rate, batch size, freeze, etc。 Set lr: 1e-4or5e-5. Set lr_backbone: 1e-5 or 5e-6. Set batch_size: 4
—reply : We have also tested this — on our own model and toolchain — and found the results were not good. There was a document earlier that I sent to your colleague Yang Xuan
4th priority: Change to larger backbone
—reply : We tried switching to the GD BASE model, and the results were largely the same — the metrics didn’t improve much.
Based on your comments, the main bottleneck is not simply hyperparameter tuning, but rather the combination of class imbalance, insufficient training coverage for some classes.
For classes with very limited data, such as E, H, M, O, P, and Z, our recommendation is to raise them to a minimum learnable level of coverage.
Suggest running experiments on 3 to 6 of the low F1 classes firstly. For those classes, please add more explicitly annotated and directly trainable samples.
Besides augmentation you have done, you can use copy-paste style augmentation to improve diversity.
Above will verify whether targeted data balancing can deliver measurable gains.
Some suggestions in there worry me that a literal interpretation might be ambiguous. Is there any way we could have a voice call? How can I conveniently reach you by phone?
I mean you can select several low F1 classes, and do oversample(copy-paste style augmentation), then run experiments on them to monitor if the F1 can improve for these low F1 classes.
Is it based on the experiments provided to you, without any adjustments, only focusing on a few classes with low F1 scores, then performing sample augmentation to observe the situation of these classes? Could you specify exactly how many samples to augment to? The test set remains unchanged, right?
Not just focusing on a few classes with low F1 classes. Please use 1) the classes which you have found that have good F1 score 2) several few classes with low F1 classes, but use copy-paste style augmentation, to enlarge the dataset for these few classes. For your case, you can augment to about 1000 labels for each class.
In short, do oversampling against the few classes which have lower F1 class, to make dataset balanced, to check if the bottleneck is the insufficient training dataset for some classes.
Yes.
Regarding the data distribution, I would like to add some additional information:
We conducted experiments with balanced distributions using 32-shot and 128-shot random samples from the full dataset. The experimental results vary across different categories — some show an upward trend, some remain roughly flat, and others decline. What might be the reason for this?
In addition, we believe that simply copying and pasting for upsampling to expand the data does not bring valuable learnable features; it only makes the distribution appear more balanced.
Some of the targets we detect using object detection exhibit characteristics such as diverse appearance variations, the need for semantic understanding, and the need for degree judgment. Would these types of targets be suitable for object detection modeling?
The 32-shot and 128-shot may be too small. Based on your earlier result, some of the higher-F1 classes appear to have around 600 images and more than 2000 labels, so a 32-shot or 128-shot setting may be too far below the sample scale that supports stable learning in your task. Please consider running balanced experiments with more shots.
Yes, copy-paste should not be expected to generate truly new learnable features, but it may still be useful as an auxiliary step when the goal is to reduce sampling imbalance for certain low-frequency classes. It helps you trigger experiments to check where is the bottleneck.
For targets with large appearance variation, object detection can still be effective if the objects are visually localizable and the category definition is clear. However, if the decision depends heavily on semantic understanding, contextual reasoning, or degree judgment, then pure object detection may not be the most appropriate approach.
This is also why I suggested you to select some high-F1 classes and add some lower-F1 classes to run experiments instead of full class training.
You can set a baseline. (Select higher-F1 classes).
Then, more experiments as below.
- Add several lower-F1 classes and increase them to a more meaningful sample scale (for example, closer to 600+ images).
- Then gradually include more classes and compare the trend.
This may help clarify whether the main limitation comes from sample scale, class imbalance, or the suitability of pure object detection for certain categories.
Thank you for your detailed response.
The above discussion mainly focused on how to identify and explain the causes of performance differences. May I ask what methods are available to improve the corresponding performance for different causes? Thank you.
The improvement method depends on the root cause, because different causes require different actions.
-
If the issue is insufficient data or class imbalance, then the main action is to 1)add more directly trainable samples for the low-F1 classes, 2)Increase the sample scale of low-frequency classes to a more meaningful level 3) Apply moderate oversampling or class-balanced sampling.
-
If the issue is insufficient diversity, then the main action is to add samples with more realistic variation in appearance, scale, background, and lightning conditions. Ensure the added data better represents the real distribution of the class.
-
If the issue is weak or indirect supervision, then the main action is to 1)improve annotation quality and provide more explicit, directly trainable labels. 2)Improve annotation consistency and clarify class definitions.
-
If the issue is class ambiguity, then the class definition and annotation rules may need to be refined. Review whether some classes should be merged, split, or redefined. Refine annotation guidelines for visually similar categories.
-
If the issue is mainly training setup, then parameters such as learning rate, batch size, freeze strategy, and epochs can be further tuned.
In short, the recommended improvement path is:
- improve data quantity if the issue is insufficient coverage,
- improve diversity if the issue is weak representativeness,
- improve supervision if the issue is indirect labeling,
- refine task and label design if the issue is semantic ambiguity,
- and only then further optimize the training configuration.
A practical next step would be to group the low-F1 classes by cause. Once the causes are grouped, the corresponding optimization actions become much clearer.
One more question I’d like to ask:
Among the categories we are detecting, there are two with very different performance levels, defined as follows:
-
Damage: Missing areas or irregular holes on the object surface, which can vary in size, location, and shape.
-
Corrosion: Rust-colored patches on the object surface, close to reddish-brown or dark brown, with blurred edges and rough texture.
The specific F1 score comparisons are as follows:
32-shot: Damage (30 F1) vs. Corrosion (89)
128-shot: Damage (48) vs. Corrosion (90)
Full data (379 vs. 686 samples): Damage (59) vs. Corrosion (91)
What could be the reason for such a significant performance gap between these two categories? Do you have any suggestions for improvement? Thank you.
Based on the results you shared, Corrosion remains consistently around 89 to 91 under 32-shot, 128-shot, and full-data settings, while Damage improves from 30 to 59. This suggests that Damage is learnable, and it is more sensitive to sample scale, and is harder for the model to capture consistently under the current definition.
Damage is defined as missing areas or irregular holes whose size, location, and shape can all vary, which means its visual pattern space is very large, making it difficult for the model to learn a stable pattern.
By contrast, Corrosion has clearer color, texture, and visual characteristics, such as rust-colored, reddish-brown, dark brown, and rough texture, so it is easier for the model to form stable features.
Damage improves continuously from 30 at 32-shot, to 48 at 128-shot, and then to 59 with full data, which shows that adding data does help this category. However, it still does not approach the 90-level performance of Corrosion, which suggests that the issue is not only limited sample quantity, but a combination of insufficient data, lower learnability, and possibly less stable category definition.
Suggestions
- please review the annotation definition for Damage and confirm the following points:
- Is there a consistent rule for how large the annotation box should be?
- Are they being mixed into other categories?
- For Damage, the key is not only to add more samples, but to add more representative samples covering different types of variation, such as:
- variation in size
- variation in location
- variation in shape
- variation in background
- variation in severity level, such as mild / moderate / severe