Failed to MLC-compile mlc-ai/Llama-3.1-8B-Instruct-fp8-MLC on Jetson AGX orin

Robotics & Edge Computing Jetson & Embedded Systems Jetson AGX Orin
, ,
6

Hi,

It looks like the model can work with q4 but fails with 8-bit quantization.
So the failure might be caused by the model requiring more memory resources than the Jetson device has.

You can verify this by monitoring the system with tegrastats:

$ sudo tegrastats

Thanks.