I installed jetpack 5 in my new AGX Xavier with sdkmanager. Then I installed Miniforge3-4.12.0-0-Linux-aarch64.sh and created a environment to install pytorch for jetpack 5 version, which it was downloaded from the nvidia page:
For JetPack 5.0, we provide the pre-built PyTorch wheel for Python 3.8 (as this is the default version of Python that comes with Ubuntu 20.04). For Python 3.9, you would need to build it from source, following the instructions from this post: https://forums.developer.nvidia.com/t/pytorch-for-jetson-version-1-11-now-available/72048 (that is, assuming you don’t want to just use Python 3.8 instead)