Jetpack6.2+TensorRT OOM issue

[DavidDDD]

Hi,

@VincentWang

SWAP will be used on GPU?

No.

TensorRT-LLM is verified by AGX Orin

TensorRT-LLM for Jetson

TensorRT-LLM is a high-performance LLM inference library with advanced quantization, attention kernels, and paged KV caching. Initial support for TensorRT-LLM in JetPack 6.1 has been included in the v0.12.0-jetson branch of the TensorRT-LLM repo for Jetson AGX Orin. We’ve made pre-compiled TensorRT-LLM wheels and containers available, along with these guides and additional documentation: > TensorRT-LLM Deployment on Jetson Orin

If you want to inference Llama-2-7-b, you could try other like

Small LLM (SLM) - NVIDIA Jetson AI Lab

Thanks