Drive AGX thor with Alpamayo

Autonomous Vehicles DRIVE AGX Thor DRIVE AGX Thor General
24

@SivaRamaKrishnaNV : can you please confirm is this a bug ?

25

I could not able to download the valid tokenizer file can preprocessor_config.json and because of that i am getting errors related to that can you support by sharing ways to download vaild files

27

Any update or feedback would be very much helpful

29

@SivaRamaKrishnaNV : Is there any update ?

30

@SivaRamaKrishnaNV : thanks for your support it is working fine

32

Dear @nandhini.ravikumar1 ,
As you have confirmed the steps provide via PM worked for you. Could you please share issues you noticed and complete followed steps to improve the documentation

33

Hello @SivaRamaKrishnaNV

Please find attached text file for the steps i followed and pre requisites that are needed in the THOR HOST

Alpamayo deployment steps.txt (6.9 KB)

regarding issues , i faced 3 major issues as below

  1. Building in the thor host has OOM error
    Solution: GPU carveout to 32 GB instead of 6GB in my case
  2. Tokenizer and pre processor config file missing
     Solution: Ensure venv environment is okay before export
  1. Input_action.json is not aligned with expectation of the VLA model and user guide need to be updated
 Solution: used  the script from PM team

Hope this would be helpful

34

Alpamayo verification steps for others reference

Step 1 : setting up the environment

#Set up virtual env 
python3 -m venv ~/edgellm-venv
source ~/edgellm-venv/bin/activate
pip install --upgrade pip
pip install .
pip install requests datasets
pip install "huggingface_hub==1.33.0"
python3 -c "import huggingface_hub; print(huggingface_hub.__version__)"   # 1.33.0

Noticed issues when installing tensorrt-edgellm-export in venv

Fix : python -m pip install --upgrade --force-reinstall “safetensors==0.7.0” “transformers==5.9.0”

Diagnosis commands:
python3 -c “import torchvision”
pip list | grep -iE "^(torch|torchvision|transformers) "
python3 -c “import torch; print(torch.version.cuda)” # 13.0

Fix (installs matching torchvision, leaves torch untouched):
pip install --no-deps “torchvision==0.27.*” --index-url https://download.pytorch.org/whl/cu130
replaced 0.25.0a0+ca221243 with 0.27.1+cu130

python3 -c “import torchvision, transformers; from transformers import AutoProcessor; print(‘ok’)”

Step 2 : Download the VLA model

export WORKSPACE_DIR=$HOME/tensorrt-edgellm-workspace

export MODEL_NAME=Alpamayo-R1-10B

mkdir -p “$WORKSPACE_DIR” && cd “$WORKSPACE_DIR”

hf auth login

hf download nvidia/Alpamayo-R1-10B --local-dir "$WORKSPACE_DIR/$MODEL_NAME"

**Step 3 : Export the model from venv**

tensorrt-edgellm-export "$WORKSPACE_DIR/$MODEL_NAME" "$WORKSPACE_DIR/$MODEL_NAME/onnx" --max-kv-cache-capacity 4096

#Verify following files are there otherwise it will not work

action/:
total 4459004
drwxr-xr-x 2 root root       4096 Sep 30 20:49 .
drwxr-xr-x 5 root root       4096 Sep 30 20:48 ..
-rw-r--r-- 1 root root        516 Sep 30 20:49 config.json
-rw-r--r-- 1 root root    4760340 Sep 30 20:49 model.onnx
-rw-r--r-- 1 root root 4561240064 Sep 30 20:49 model.onnx.data

llm/:
total 16073984
drwxr-xr-x 2 root root        4096 Sep 30 20:47 .
drwxr-xr-x 5 root root        4096 Sep 30 20:48 ..
-rw-r--r-- 1 root root        5292 Sep 30 20:47 chat_template.jinja
-rw-r--r-- 1 root root         740 Sep 30 20:47 config.json
-rw-r--r-- 1 root root  1275469920 Sep 30 20:47 embedding.safetensors
-rw-r--r-- 1 root root     4421841 Sep 30 20:46 model.onnx
-rw-r--r-- 1 root root 15167660032 Sep 30 20:46 model.onnx.data
-rw-r--r-- 1 root root         684 Sep 30 20:47 processed_chat_template.json
-rw-r--r-- 1 root root    12162986 Sep 30 20:47 tokenizer.json
-rw-r--r-- 1 root root         694 Sep 30 20:47 tokenizer_config.json

visual/:
total 1143720
drwxr-xr-x 2 root root       4096 Sep 30 20:48 .
drwxr-xr-x 5 root root       4096 Sep 30 20:48 ..
-rw-r--r-- 1 root root       5292 Sep 30 20:48 chat_template.jinja
-rw-r--r-- 1 root root       1247 Sep 30 20:47 config.json
-rw-r--r-- 1 root root    2404703 Sep 30 20:47 model.onnx
-rw-r--r-- 1 root root 1157300224 Sep 30 20:47 model.onnx.data
-rw-r--r-- 1 root root       1192 Sep 30 20:48 preprocessor_config.json
-rw-r--r-- 1 root root       1192 Sep 30 20:48 processor_config.json
-rw-r--r-- 1 root root   11422650 Sep 30 20:48 tokenizer.json
-rw-r--r-- 1 root root        858 Sep 30 20:48 tokenizer_config.json

#copy the model files to target
scp -r ~/tensorrt-edgellm-workspace/Alpamayo-R1-10B/onnx \

  thoruser@<target_ip>:~/tensorrt-edgellm-workspace/Alpamayo-R1-10B/

Step 3: Cross compile C++ runtime

docker run -it --privileged --net=host -v /path/to/workspace:/workspace nvcr.io/drive/driveos-sdk/drive-agx-linux-nsr-aarch64-sdk-build-x86-earlyaccess:7.2.5.0-0004
cd /workspace
git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git fetch --tags && git checkout v0.9.0
git submodule update --init --recursive
mkdir build && cd build
cmake .. \
  -DCMAKE_BUILD_TYPE=Release \
  -DTRT_PACKAGE_DIR=/usr \
  -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
  -DEMBEDDED_TARGET=auto-thor \
  -DCUDA_CTK_VERSION=13.3 \
  -DENABLE_CUTE_DSL=ALL
make -j"$(nproc)"

#copy the TensorRT edge LLM build directocry to target using scp. Change build directory perm on target if needed.
scp -r ../build <thoruser>@<target_ip>:~/TensorRT-Edge-LLM/build

**Step 4: Prepare the alpamayo input file **
use
prepare_alpamayo_input.py (6.5 KB)

#inside venv  
cd /workspace/tensorrt-edgellm-workspace
pip install physical_ai_av pillow
hf auth login
python3 /path/to/prepare_alpamayo_input.py \
  --workspace /workspace/tensorrt-edgellm-workspace \
  --container-workspace /home/nvidia/tensorrt-edgellm-workspace

#The first run downloads approximately 8 GB because the official dataset stores camera data in multi-clip chunks. The script creates:

```text
tensorrt-edgellm-workspace/
├── dataset-cache/
├── input_action.json
└── inputs/
    └── physical_ai_av/
        └── <clip-id>/
            ├── cross_left_t0.png
            ├── cross_left_t1.png
            ├── ...
            └── front_tele_t3.png

#copy the alpamayo input test files
scp <path-to>/files/files/alpamayo-input/input_action.json \
  <thoruser>@<target_ip>:~/tensorrt-edgellm-workspace/

scp -r <path-to>/files/files/alpamayo-input/inputs \
  <thoruser>@<target_ip><target_ip>:~/tensorrt-edgellm-workspace/

Now we have Alpamayo model ONNX files, input files, TensorRT edge LLM build copied to target already.

Steps 5: Prerequisites in thor Board:
Thor target has only 6GB GPU available memory , so we need to set it to 32GB to avoid OOM error. Run below commands on target.

		echo 16384 | sudo tee /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
		cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages
		cat /sys/devices/system/node/node0/hugepages/hugepages-2048kB/free_hugepages

Step 6: Run the inference on target

  cd /path/to/TensorRT-Edge-LLM/build
./examples/llm/llm_build \
  --onnxDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/onnx/llm \
  --engineDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/engines/llm \
  --maxInputLen 3424 \
  --maxKVCacheCapacity 4096 \
  --maxBatchSize 1
  
  ./examples/multimodal/visual_build \
  --onnxDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/onnx/visual \
  --engineDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/engines \
  --minImageTokens 160 \
  --maxImageTokens 18432 \
  --maxImageTokensPerImage 192

  
  ./examples/multimodal/action_build \
  --onnxDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/onnx/action \
  --engineDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/engines \
  --maxBatchSize 1
./build/examples/multimodal/action_inference \
  --engineDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/engines/llm \
  --multimodalEngineDir /home/nvidia/tensorrt-edgellm-workspace//Alpamayo-R1-10B/engines \
  --inputFile /home/nvidia/tensorrt-edgellm-workspace/input_action.json \
  --outputFile /home/nvidia/tensorrt-edgellm-workspace/output_action.json