Installing real time kernel on Xavier with jetpack version 5.1

I am not using Issac driver, and I want to use my Xavier for controlling a UR5 robotic arm. The robot is controlled by the Xavier but with a lot of jerky movements and we notice in the log there a lot of messages with (ur rtde overflow). The Xavier is flashed with jetpack version 5.1 and I saw that for ur_rtde to function correctly there is a need for real time kernel.

The problem is that by default there no real time kernel for the version of jetpack but if I add in /etc/apt/sources.list.d/nvidia-l4t-apt-source.list the dependency for a version like (r32.7) and I do the update I can see the kernel and it’s header available, my question is: is it possible to do this? Or will this break everything because it is from an older version of the jetpack? Do I have to re-flash the Xavier with a jetpack that contains this real-time kernel support by default (something like 4.6)?