VLM Refresh Rate

[alpuMUUU]

When using the VideoQuery agent to apply prompts to the video feed with the VLM, like in the Live LLaVA example from Live LLaVA - NVIDIA Jetson AI Lab, the refresh rate seems to be low. Is this a limitation of the Jetson AGX Orin or the camera?

With the liuhaotian/llava-v1.5-7b model, the refresh rate is around .2 FPS. When using the VILA1.5-3b model, it was around .8 FPS

[AastaLLL]

Hi

Have you maximized the device’s performance?

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

You can find our benchmark table below.
VLM on Orin with VILA 1.5-3B is ~7fps.

Benchmarks - NVIDIA Jetson AI Lab

Thanks.

[alpuMUUU]

After maximizing performance, I am getting around 0.6 FPS when using Llava-7b. The benchmarks are around 1.43 FPS for Llava-7b. I should have also specified that I am using the Jetson AGX Orin Developer Kit 32GB version. Would ~0.6 FPS be around the estimated performance?

[AastaLLL]

Hi,

The benchmark is done with the MLC container.
Do you also inference the model with the library?

Thanks.

[alpuMUUU]

I downloaded the MLC container, but I did not know how to run the llava-v1.5-7B model in the MLC container. I use the following command from the Live LLaVA tutorial and I’m getting approximately 0.6 FPS after maximizing performance.

jetson-containers run $(autotag nano_llm)
python3 -m nano_llm.agents.video_query --api=mlc
–model liuhaotian/llava-v1.5-7b
–max-context-len 256
–max-new-tokens 32
–video-input /dev/video0
–video-output display://0

[AastaLLL]

Hi,

Please check the benchmark script below:

dusty-nv/jetson-containers/blob/master/packages/llm/mlc/benchmark.sh

Thanks.