I initially bought AGX orin to test GDS. Installed Jetpack by default has cuda-11.4, inside which there is no gds folder. I downloaded and installed cuda-12.1 and 11.8 with aarch64 from official website which has gds folder and toolkit inside cuda. But the gdscheck shows that it doesn’t support NVMe or GPU. I struggled installing nvidia-fs.ko then NVMe is supported, but not GPU. Then struggled with some other drivers and gave up.
Now I bought Jetson AGX Xavier to have same device as the blog. I have the same issue as before. What jetpack and cuda version should I install to try GDS on Jetson Xavier. If there is some documentation, that would be great too. Please let me know.
They are different features.
GPUDirect RDMA vs GPUDirect Storage. Which feature do you want?
We don’t have too much experience with gds.
But RDMA is working on Xavier and it will need some manual changes for Orin (this is fixed in our next release).
Hi,
I connected NVMe SSD via PCIe to Xavier. Now how do I test GPUDirect RDMA for the transfer of data from NVMe to GPU? Is it possible to use NVMe with RDMA? Also, in general, is there any documentation or test command to test if RDMA is working on Xavier (even if it’s not with NVMe)? Please let me know.
Regarding GDS, I saw one Nvidia forum response that GDS is not supported on Xavier (almost 1.5 years ago) GPUDirect Storage in Nvidia jetson xavier
But the latest jetpack-5.1.1 documentation shows the cudatoolkit for l4t contains nvidia-gds in the documentation How to Install JetPack :: NVIDIA JetPack Documentation But sdkmanager doesn’t actually install nvidia-gds when I install cuda toolkit with jetpack-5.1.1. This is another separate issue, I wanted to clarify.
RDMA needs an external FPGA device.
Our previous users tested it with FPGA rather than NVMe SSD.
Below is a sample to test some basic functions for your reference:
We need to double-check with our internal team about the latest status of the GDS support of Jetson.
Will let you know the result.