Early this year (2024), I was very satisfied with the performance of llama.cpp 2 and Mistral on Jetson Xavier AGX.
However, after I built the last llama.cpp code last week (August 25, 2024) to run Llama 3.1 and Phi 3.5, the loading time became unendurable. Llama-3.1-8B-Lexi-Uncensored_V2_Q8.gguf (8.5GB) took 7.5 minutes to load.
Once loaded, the inference speed is fine:
llama_print_timings: load time = 450482.81 ms
llama_print_timings: sample time = 3076.00 ms / 1308 runs ( 2.35 ms per token, 425.23 tokens per second)
llama_print_timings: prompt eval time = 14287.63 ms / 196 tokens ( 72.90 ms per token, 13.72 tokens per second)
llama_print_timings: eval time = 154754.42 ms / 1300 runs ( 119.04 ms per token, 8.40 tokens per second)
llama_print_timings: total time = 770548.21 ms / 1496 tokens
Anyone had similar experience? Not many people in llama.cpp community use Jetson, I think this may be the proper forum to ask.