Hi,
On the problematic unit, these had no effect.
Another development is that i witnessed the same problem on the other Xavier unit after running the program for ~30 minutes. On this unit the problem did not get big enough to crash the program, but many images were skipped.
While the problem persisted, i witnessed that ksoftirqd/0 was using ~30% of the cpu core.
After trying the settings on this unit, at the same time enabling jetson clocks, the problem seems to have disappeared.