Using GstBuffer to transport data?

You can get the compressed stream in appsink. Please refer to this sample:
Latency issue: nvv4l2h265enc accumulates four images before releasing the first - #3 by DaneLLL

Since you don’t need existing sink plugins in gstreamer, you can consider to use jetson_multiemdia_api. May refer to 12_camera_v4l2_cuda sample for capturing video frames through v4l2 into NvBuffer. And can apply this patch for encoding into h264 stream:
TX2 Camera convert/encode using Multimedia API issue - #17 by DaneLLL