The tutorials in the Generative AI Lab do not list the Orin Nano as an available platform - only the AGX Orin. Can these tutorials run on the Orin Nano? I’m particularly interested in the LLaVA model.
Hi @ralphjy, Orin Nano is listed on the tutorial pages for which it was confirmed to work in 8GB memory. The Llava tutorial as shown uses a quantized 13B variant of Llava (which is too big for Orin Nano). Most/all of the quantized Llava models out there seem to be using the 13B models as opposed to the 7B ones (presumably because Llava accuracy has for all intensive purposes needed the 13B models to be realistically usable, but that seems to be getting better with Llava-1.5 and Mistral variants of Llava for 7B)
With Llava and multimodal models, the vision embedding model also needs loaded (in the case of Llava, CLIP-Large ViT-L/14@336px is an additional ~1.7GB)
If you were to quantize one of the Llava 7B models, it’s worth a shot to try though. Will add this to our todo list to investigate Llava on the lower-memory systems 👍
Hi @dusty_nv, thanks for the quick response. I realized one issue was that I was only looking at the featured tutorials on the Generative AI Home page and that I should have selected the “Explore Tutorials” button for a better overview of tutorials that run on the Orin Nano.
Thanks for the Llava explanation. It would be nice if the newer Llava versions would have reasonable accuracy with 7B models.