I have had great success hosting some of the large language models like Alpaca and Vicuna (7B and 13B versions) on my Jetson Orin. I am upgrading my workstation with a 4090, which leaves my RTX3080 without a home.
I would like to try to use the 3080 in the PCIe slot available on the Orin dev kit. I have an extra ATX Power Source to supply the 3080, and I can rig and enclosure to hold it in place. What considerations am I going to have to take though in order to get this setup working?
You’d need a driver for arm64 instead of amd64. This would have to be in addition to the existing driver. The existing driver is for an integrated GPU (iGPU), which connects directly to the memory controller, whereas the drivers most people are used to are for a discrete GPU (dGPU) on the PCIe bus. Should you replace or modify the original driver you might find a need to reinstall (it would be difficult to fix). So back up the rootfs prior to any experiment.
Also, the nvidia-smi application is only for PCI based dGPUs, and I don’t know if software will get confused with both nvidia-smi and iGPU drivers. I’m kind of pessimistic. So I’ll double the original advice: Back it up before you start.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.