Unresponsive/slow RTSP stream from USB Cam + Deepstream on Jetson Nano

I don’t have a Nano for trying, I only can check on AGX Xavier. You may boost your Nano with:

sudo nvpmodel -m 0
sudo jetson_clocks

I’d suggest to build test-launch example. You can find details in Nano FAQ with question:
Q: Is there any example of running RTSP streaming?

I’m using this command for streaming nvarguscamera:

./test-launch "nvarguscamerasrc do-timestamp=1 ! video/x-raw(memory:NVMM), width=1920, height=1080, framerate=30/1, format=NV12 ! nvv4l2h264enc insert-vui=true insert-sps-pps=1 insert-aud=1 maxperf-enable=1 ! h264parse ! video/x-h264, stream-format=byte-stream ! rtph264pay name=pay0 pt=96 "

Using VLC for reading from Jetson itself has seconds of delay.
However using this gstreamer pipeline I see much less latency:

gst-launch-1.0 rtspsrc location=rtsp:// latency=0 ! application/x-rtp,media=video,encoding-name=H264 ! rtph264depay ! h264parse ! nvv4l2decoder ! nvvidconv ! xvimagesink

For remote reading, you may increase the latency such as latency=500.