VPI Remap: Output of different size

Hello. I am using VPI’s remap algorithm and can not seem to get it to create an image of a different size than the input image. If I define my WarpMap like vpi.WarpMap(size=INPUT_SIZE) it works but if I define it like vpi.WarpMap(size=DESIRED_OUTPUT_SIZE) it won’t. Specifically, I get this error:

“ValueError: VPI_ERROR_INVALID_ARGUMENT: Output image must have same size corresponding to the warp map passed during payload creation”

This doesn’t make any sense to me because I am not declaring the output image size. Do I have to do that? The error throws when I am actually doing the remap:
output = vpi.asimage(input).convert(vpi.Format.NV12_ER).remap(warp_map, interp=vpi.Interp.NEAREST).convert(vpi.Format.RGB8)

and since the input is bigger than my desired output my only option is to fill the rest of the WarpMap with 0s which really slows down performance and makes the output image way bigger than it needs to be.

How can I fix this? Thanks!