The ViewPortOut rectangle is a region within the display mode that the ViewPortIn pixels are scaled to. In your case, your display mode is 1920x1080 and the default ViewPortOut exactly fills that. The WxH numbers control the size of the box and the +x+y setting controls where the upper-left corner of it is displayed, relative to the display mode.
The ViewPortOut must fit entirely within the display mode, or it’ll get rejected by nvidia-settings. For your first attempt, you set 1920x1080+2+0. This is invalid because it asks for two blank columns on the left, and two columns past the right edge of the screen. What you probably meant to do was to shrink the width by double the amount of border you’re trying to apply. So for example, to get a 10-pixel border around the whole image, shrink both dimensions by 20:
If you want to shrink your desktop to match (instead of getting scaling from 1920x1080), set your ViewPortIn to 1900x1060 as well.