Polygon inpainting

Is it possible for the inpainting model to ignore certain parts of the image when making predictions? For example, I have a picture like this. While inpainting certain parts of this image, I want black pixels not to be used when making predictions. Is this possible? Or is it possible to inpaint only inside the polygon and not include the pixels outside?