Dual screen nvoverlaysink sluggish

On tx2 4g R32.2, with our custom-designed carrier board, when we capture one live camera and display it with 1 nvoverlaysink, the playback is smooth.
However when we capture 2 cameras and show them on 2 nvoverlaysink with different display-ids, the movement on both displays become a bit sluggish. This become obvious when pointing the camera to fast-moving cars at 50 meters away, just like they are dropping a frame every one second.
How can we fix it?

This looks to be duplicate. Please check this first:

I observed two phenomena and separated them. The other one is about latency and this one is about smoothness.

we have tested nvpmodel -m 0 and 2 and many other options but there was no improvement on this.