I think you are running the experiments which is similar to your old topic Detectnet_v2(resnet50) low accuracy on 2 class dataset. For your training images and labels, I remember that you label the whole images as damage or healthy. If that is the case, actually you can train with classification network instead of detection network.
If you decide the use detection network, I suggest you to label to the “damage” class where the area is really damage. And label to the “healthy” class where is really healthy.