Flash Attention/Torch SDPA on Orin Nano Devleoper kit

[mr.naveenkumarjain]

Hi ,
I am trying to build convernsation ai tool on Nvidea jetson orin nano 8gb
using distil whisper medium

what all the optimization possible for shorter audio like <10 second
is the Flash Attenntion or torch sdpa is avaialble

also will the streaming will help for shorter audios ?

Thanks
Navene

[AastaLLL]

Hi,

Please find below related topics for info:

Flash_Attention_2 && BitsandBytes && more

Ideally I’d like to get these working but even after I docker exec -it /bin/bash; pip install bitsandbytes; pip install flash_attn; I setup the LD_LIBRARY_PATH so it knows where to look for the python modules both in the container as well as on my Jetson Orin. I even went as far and installed the flash_attn and bitsandbytes on my local Orin 64GB. It will still claim it can’t find it it. Also, does the model loader have an impact on how fast the model performs and how do I match the loader …

Thanks.