NIM for finetunning/custom models?

Accelerated Computing NGC GPU Cloud
1

Hello, I would like to know if NIM can be used to deploy fine tuning or custom models. I deployed the MIXTRAL-8x7b-INSTRUCT-2-A100.24.01 model through NIM following this tutorial and now I’m trying to launch the optimized mixtral engines (checkpoints from HF) with tensorRT-LLM (ensemble, postprocessing, preprocessing, tensorrt_llm and tensorrt_llm_bls) but I have not been able to, Is there a tutorial on how to launch finetunning/custom models through NIM? Is there documentation of the structure of the model repository that NIM needs to deploy a model?

2

Hi, I think this tutorial is tailored for you: Parameter-Efficient Fine-Tuning - NVIDIA Docs

It introduces how to obtain LoRA adapters from HF and then serve with NIM.

Hope it helps!