Hi guys,
I have an AGX Orin Developer Kit (Memory 32GB; Storage: 64GB), and its memory size limited the possibility to run larger LLM models. So I was wonder if it’s possible to replace/upgrade its memory, like from 32GB to 64GB or even bigger. For instance, to ask a hardware guy to manual replace the LPDDR5 chips on it, like via re-soldering?
Besides, what about replacing its eMMC chip ?
Any suggestions are welcomed.
Many thanks ~
Someone else will need to answer this, but I want to point out a few things regarding this topic…
You can use an external boot, e.g., to an SSD or NVMe for disk space. Or you could just mount any external drive to a particular mount point, e.g., to either “/usr/local” or to “/home”.
I will suggest that soldering more RAM might be possible, but very risky and not easy. Definitely not something to try if you are not willing to lose the entire unit.
Additionally, Jetsons have an integrated GPU (iGPU) which is wired directly to the memory controller. Desktop PCs use a discrete GPU (dGPU) with their own memory (VRAM). A dGPU can use whatever memory it has, whereas there are a lot of restrictions for the iGPU case which might not make upgrades as useful as it sounds.
GPU memory is always accessed at a physical address (the memory controller can map 1:1 and even if going through the controller, then the memory behavior would be that of physical address rather than a virtual memory address). Virtual memory can be fragmented, and the GPU needs contiguous memory. GPUs live in kernel space, and all of the devices there use physical RAM. It is user space which tends to be able to use virtual memory, and this is why swap doesn’t help a GPU (well, it could indirectly help if user space no longer uses as much physical RAM). But even if physical RAM is released by giving user space swap, if the GPU memory is not reserved ahead of time, then it might be fragmented such that the GPU still cannot use that RAM. There are also address line alignment requirements, and so even if there is contiguous memory, then if the memory does not start at the correct address, perhaps not all of it will be available so far as the GPU is concerned.
@dusty_nv is the resident expert on AI, and he could give you ways of improving memory use, but I doubt that adding memory chips will work (and if it does work, then you might kill a few Jetsons in the learning stage). If you are training models, then you could do that on another system with a dGPU that has lots of VRAM, and then actually run the model on the Jetson (which would take less RAM than training). So before anyone can answer methods of improving memory use you would need to state a more detailed description of the problem.
Forget it. It’s that easy. It’s ven hard work if NVidia decides to change memory chips (same size, different vendor or same vendor different type) in a module. This usually requires a new Jetpack since each RAM chip has its own parameters. Remember the EEPROMs on DIMM RAM modules?
If you can’t upgrade for some reason then you have a problem. Been there, done this. Took me a week each time to modify the old jetpack so it boots the new modules.
Plus: If you do manage to get this working at all you need to repeat this work for each new Jetpack version.
EMMC swap sould be easier but is mostly unnecessary. You can always add an SSD.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.