Deepstream Custom model with white background not complying with coloured and textured backgrounds


I got stuck with a problem, Thanks in advance for the solution.

The custom model is trained to detect person and faces with data set of the respective images in a White Background.

The model got an accuracy about 98% for both classes.

While checking the model accuracy in deep stream with coloured and textured background, the inferred bounding boxes seem to populate randomly all-over the screen.

My Questions :

  1. Does the quality of the data set plays a vital role ?
    [Such as sharpness/blurred edges, textures, colours]

  2. If the model is trained in a data set of simple background to detect object, will the performance of the get worse on inferring over complex background ?