There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks
You can try
experiment1: More deeper backbone
experiment2: Assume class 1 is less than class 2, you can oversample the class 1. It is fine to just copy the existing class 1. Then, make class 1 and class 2 be more balance.