Thank you for your support! I was able to solve my problem. In the end, the FPGA was adding 16 columns I did not expect so I set the resolution in the device tree and driver were wrong. After correcting it, the Xavier NX was able to stream without errors and with correct frames. The same goes for the Nano.
2 things are important to mention:
- NANO was streaming without errors, even when the frames were corrupted while XAVIER NX was not streaming because of errors ( XAVIER NX VI Debug Codes ).
- On all systems jetson was receiving frames in the resolution I was giving him on the driver and device tree, not the solution the sensor was sending
- Wrong resolution in the driver seems to produce such errors we have been seeing.