4/2/24 - local_llm migration to NanoLLM project
-
The local_llm container provided support for the most optimized LLM inferencing APIs (such as MLC/TVM), in addition to many of the advanced demos and multimodal agents on Jetson AI Lab (such as llamaspeak and Live Llava).
-
However the size of its codebase and need for more detailed documentation outgrew being hosted directly inside jetson-containers, so its source was moved to https://github.com/dusty-nv/NanoLLM (and now that it supports SLMs and mini-VLMs, renamed to NanoLLM for consistency with our other libraries for Nano like NanoOWL, NanoSAM, NanoDB, ect)
-
jetson-containers still provides the dockerfiles and container builds for NanoLLM, and the code/containers for legacy local_llm will remain up for a while as the roll-out of NanoLLM progresses (although local_llm is now deprecated and any new features/fixes will be going into NanoLLM)
-
There is also now improved documentation, API references, and examples for using local_llm / NanoLLM now that it’s transitioning out of experimental phase. Find the docs here: dusty-nv.github.io/NanoLLM