It’s the later - you would need to rebuilt PyTorch for JetPack 4 and Python 3.8. This is because the versions of CUDA in JetPack 4 (CUDA 10.2) are different than JetPack 5 (CUDA 11.4).
There are Build from Source instructions under PyTorch for Jetson. Or since you are on Xavier, you could just reflash your device with JetPack 5 and use Ubuntu 20.04 / Python 3.8 straight away (which is what I’d probably recommend, baring other factors that you might have)