Hi,
The encoding is done on individual hardware engine NVENC. Not done on GPU so the loading is nearly 0% in tegrastats. Please check NVENC.
And you may try the commands and check if there is improvement:
Jetson 4k Encoding -> Decoding Pipeline and latency - #11 by DaneLLL
Gstreamer TCPserversink 2-3 seconds latency - #5 by DaneLLL
May also try h265 encoding.