Hello NVIDIA Team,
I would like to set up a NTB Bridge with two ORINs and a Microchip switch.
So first thing is, when i to ntb_perf on x86 to the orin i get like 3.5GB/s with Gen3 which is close to 90% of theoretical max but only 800MB/s when i do ntb_perf from the ORIN.
Strange but what really bothers me is that i cant do ntb_perf with two ORINs because after loading ntb_hw_switchtec i only get ntb link up but not the messages in the picture below which are being captured on the x86 machine
after proceeding with loading ntb_perf.ko always one of the input buffers are unallocated so when i reload ntb_perf on ORIN1 which allocates the addresses then ORIN 2 gets unallocated so i cant do a ntb_perf even if it says ntb link up always one buffer is unallocated.
I tried switchtec kernel 1.8 and 1.9 with kernel versions 5.10.120 on both ORINs respectively.
- To summarize:
- x86 to each ORIN is working
- Jetson ORIN AGX to DRIN ORIN AGX is not working
(x86 has 5.15.xxx generic and ORINs have either 5.10.120-rt70 oder 5.10.120 - DRIVE and Jetson)
Do you know what could cause this strange behaviour?
I think it has to do with kernel version 5.10 so i would like to upgrade to Jetpack 6.1 because it has kernel 5.15. The problem is that the installation is so basic that i dont even has a internet browser, nor the mainline ntb drivers.
What is the reason that Jetpack 5.1.x has such a complete installation but not Jetpack 6.x? How can i customize the Jetpack 6.x or upgrade Jetpack 5.1.x to have kernel 5.15?
Btw this is the procedure for loading all necessary drivers:
unload all modules
cd …/switchtec-kernel-1.8/
rmmod ntb_perf
rmmod ntb_hw_switchtec
rmmod switchtec
rmmod ntb
load modules
modprobe ntb dyndbg=+p
insmod switchtec.ko dyndbg=+p
sleep 1
insmod ntb_hw_switchtec.ko dyndbg=+p max_mw_size=0x8000000000000000 use_lut_mws=0
cd …/nt_test/ntb_perf/
insmod ntb_perf.ko chunk_order=20 total_order=36
check ntb_perf is loaded properly
cat /sys/kernel/debug/ntb_perf/$bdf/info
echo
run the test
echo 0 > /sys/kernel/debug/ntb_perf/$bdf/run
check test results
cat /sys/kernel/debug/ntb_perf/$bdf/run
echo
exit