Hi,
Not sure but it seems like there is buffering mechanism in VLC. This would need other users to share experience.
It may help to run in CBR + setting virtual buffer size. It avoid burst bitrate between I and P frames and may offer better stability in limited network bandwidth. Please check the example of setting it:
Random blockiness in the picture RTSP server-client -Jetson TX2 - #5 by DaneLLL