Hi, I’m using Jetson Orin Nano with Nano LLMs (TinyLLama1.1B chatv1.0) from jetson-containers, for my project with Pepper Robot since It’s Token per second is pretty fast for Orin Nano. I want to fine-tune the model, how can I do it inside jetson-containers or are there any other ways? Thank you so much for having me.
Hi @joshwa.phyl, sounds like a cool project! Fine-tuning LLMs onboard Jetson is a topic we have been kicking around in the Jetson AI Lab Research Group and seems likely that we’ll get into as edge AI agents are becoming more complex and mature.
That said, it would seem unlikely to be able to accomplish this in 8GB RAM, but hey worth a shot! What you would try is to essentially run the TinyLlama training or fine-tuning scripts in the Transformers container (or similar). Or maybe try a LoRA as I believe those use less memory. Regardless I have not done this myself yet, and the exact scripts/commands you would use for fine-tuning seem to be somewhat model-specific (and the dataset too)
The requirements for the TinyLlama scripts are pretty normal though, hopefully should at least be able to import/load these using that Transformers container as base (although you might want to try relaxing some of the specific version requirements in there if necessary)
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.