Any tools recommanded to quantize VLMs with jetson orin nano?

Robotics & Edge Computing Jetson & Embedded Systems Jetson Orin Nano
1

Any tools recommanded to quantize VLMs with jetson orin nano? I tried with mlc_llm with docker dustynv/nano_llm to quantize VILA1.5-3b successfully, but the mlc_llm version inside docker is too old to quantize Qwen models. I tried to build new version of mlc_llm from source but keep get this error: InternalError: Check failed: (it != type_key2index_.end()) is false: Cannot find type tir.Any. Did you forget to register the node by TVM_REGISTER_NODE_TYPE ? when importing mlc_llm. Any suggestion will be helpful, thanks.

3

Hi,

Please find below for the detailed info:

For example, you can use dustynv/vllm image to quantize Qwen2.5-VL-3B-Instruct to bnb4.

Thanks.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.