we have choice of VITA and GPT,
How can I use different endpoint from maybe build.nvidia.com or any other openAI compatible model
If you have another openAI compatible model you can run the following:
docker run … -e OPENAI_API_KEY=<OPENAI_API_KEY>
-e VIA_VLM_OPENAI_MODEL_DEPLOYMENT_NAME=gpt-4o
-e VLM_MODEL_TO_USE=openai-compat
You can also load a custom model:
- Mount the directory containing the inference.py file and the optional model files in the
container. - Set the MODEL_PATH env variable to the mount path. MODEL_PATH should point to the
directory containing the inference.py file in the container. - Set VLM_MODEL_TO_USE=custom
ls <MODEL_DIR_ON_HOST>
inference.py
docker run … -v <MODEL_DIR_ON_HOST>:<MODEL_DIR_IN_CONTAINER>
-e MODEL_PATH=<MODEL_DIR_IN_CONTAINER> -e VLM_MODEL_TO_USE=custom …
You can refer to pages 21 and 23 of the VIA DP User Guide for more details.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.