Hi,
If the source is not able to meet constraint of data alignemnt, you would need to call Raw2NvBuffer() to copy data from CPU buffer to NvBuffer. This memory copy may trigger certain latency. Due to this there may not be benefit in switching from gstreamer to jetson_multimedia_api, but if you would like to give it a try, may refer to this patch:
tegra multimedia samples not working properly - #11 by DaneLLL
It demonstrates 12_camera_v4l2_cuda + NvVideoEncoder. You would also need to port the options –stats --max-perf -poc from 01_video_encode.