Orin AGX, JP 7.2, Pytorch and sm_87 support?

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
,
1

Greetings - just a newbe question:

I just got an AGX Orin, flashed it to SP 7.2, and I would very much like to use it to run everyday LLMs at minimal energy consumption and maximum throughput. Now, when installing pytorch, I tend to get the warning that sm_87 / CC 8.7 was not supported.

My question: is this still the case, or did pytorch receive an update to include sm_87 support for the various kernels available for let’s say vllm ?

Do I need to build the toolchain from source (as in recompile everything with TORCH_CUDA_ARCH_LIST=“8.7” ?

If so, can you give me a few hints for the correct env variables to set for pytorch/vllm builds for JP 7.2 and the current pytorch versions ?

Thank you !

4

I ran exactly this experiment yesterday on an AGX Orin 64 GB devkit freshly flashed to JetPack 7.2 (r39.2, CUDA 13.2, driver 595.78). Short answer: you do not need to rebuild anything, and for the vLLM path the warning is cosmetic.

The upstream arm64 image runs on Orin as-is — no Jetson-specific rebuild, no custom wheels:

docker run --rm --runtime nvidia --network host \
  -v $HOME/hf-cache:/root/.cache/huggingface \
  -v $HOME/vllm-compile-cache:/root/.cache/vllm \
  vllm/vllm-openai:latest --model Qwen/Qwen2.5-7B-Instruct \
  --max-model-len 4096 --gpu-memory-utilization 0.60 --port 8000

That gives vLLM 0.26.0 / torch 2.11.0+cu130. Torch does print the warning you are seeing — its arch list is sm_80/90/100/110/120 and it explicitly states that 8.0 covers “>=8.0,<9.0 except {8.7}” — but in practice cuBLAS works, vLLM selects the FLASH_ATTN backend, torch.compile completes, and CUDA graphs (piecewise + full) capture cleanly. No PTX-JIT stall: the server was up in ~3 min for a 0.5B model and ~9 min for the 7B including the HuggingFace download.

Measured on that setup with Qwen2.5-7B, 200-token generations, wall-clock: 11.8 tok/s single stream, and 186 tok/s aggregate at 16 concurrent requests — which is 11.6 tok/s per stream, i.e. near-zero degradation under load.

One thing worth knowing given your “minimal energy consumption and maximum throughput” goal: on this board those two pull in opposite directions. Single-stream decode is memory-bandwidth-bound (204.8 GB/s on the 64 GB AGX Orin), and no engine escapes that wall. Ollama running the same model at Q4 gave me ~14 tok/s single-stream — slightly faster than vLLM at FP16, simply because it moves a third of the bytes per token. vLLM’s advantage is concurrency: 186 vs 58 tok/s aggregate at 16 parallel requests. So if you serve one request at a time, a Q4 GGUF runtime is the more efficient choice; if you fan out, vLLM is worth the extra memory.

Two practical notes. The image entrypoint is vllm serve, so pass --entrypoint python3 if you want to introspect torch inside the container. And mount /root/.cache/vllm as above, otherwise you discard the torch.compile cache on every --rm run and pay the compile cost again.

Caveat on what I did not test: FP16 only. The known SM 8.7 gap in the prebuilt Marlin kernels means --quantization gptq_marlin may still fall back to generic CUDA cores, so if you go quantized on vLLM, verify that separately.

5

Thanks - this shows me that it is possible. For various reasons (including the docker licensing stuff) I do not feel comfortable running docker, so I would like to have everything on the machine itself. Do the prebuilt wheels pull in working NCCL (if required by the vllm v1 engine) ?

I am aware of throughput vs minimum wattage. There is a huge difference in running a full blown RTX Pro in an x64 system vs running a “home use” model on an embedded box - I did get a Thor up and running with vllm 0.26 yesterday and it gave me ~ 3000 token/s during prefill and ~42 tokens/s for decode with Qwen3.6-35B-A3B in NVFP4 after parameter and engine tweaking (using flashinfer and triton).

But, I would like to see reasonable GPU usage and vllm throughput on the Orin as well, preferably without using docker (even though docker is so convenient …), so I wonder if I have to compile the tool chain myself, or if I can just pip install pytorch and vllm for cu13 and be done with it (plus have GPU utilization)

6

Hi,

Do you install the upstream PyTorch?

$ pip3 install torch torchvision --index-url https://download.pytorch.org/whl/cu132

If so, it is a harmless warning. You can turn it off with the following function:

import warnings
warnings.filterwarnings("ignore", message=".*Found GPU.*compute capability.*")

The package has been built with sm_80 and can be run on the Orin without triggering JIT compiling:

An example of the latter is that code compiled for the target sm_80 will run on all other CC 8.x GPUs, such as sm_86 or sm_89.*

This means the upstream packages contain the kernel code that can directly run on Orin.
The warning can be turned off without any performance or capability issues.

Thanks.

7

Thank you, I needed this clarification. I’ll do a clean install of that, tossing my build experiments and will check vllm. Which quantization would be best for Orin plus supported my the kernels ?

8

Direct answer to your last question — AWQ INT4 (W4A16). I tested the quantised path after posting the above, and it changes my earlier recommendation, so please treat this as a correction to my own post.

Qwen/Qwen2.5-7B-Instruct-AWQ (5.57 GB), same box, same 200-token benchmark, wall-clock:

single stream 32.8 tok/s (vs 11.8 at FP16), and 496 tok/s aggregate at 16 concurrent (vs 186 at FP16). It held 31 tok/s per stream at 16-way, so it barely degrades under load.

Marlin is selected on sm_87. The container log says so outright:

[auto_awq.py:473] Using MarlinLinearKernel for AutoAWQMarlinLinearMethod
[cuda.py:482]     Using FLASH_ATTN attention backend

So the “no SM 8.7 Marlin kernels” caveat I gave in my earlier post does not apply to the AWQ path — please disregard it. That was the one part of my answer I had flagged as untested, and it turned out to be wrong.

It also beats the Q4 GGUF route I suggested for single-stream (~14 tok/s under Ollama). The reason is that llama.cpp dequantises Q4 to FP16 and runs FP16 MMA, so it never touches the INT4 tensor-core path — it sits at roughly a third of the 204.8 GB/s roofline, while Marlin’s real W4A16 path reaches ~90%. Ollama was kernel-bound, not bandwidth-bound. So for your “minimal energy consumption and maximum throughput” goal, AWQ INT4 wins both axes on this hardware — there is no longer a workload where I would point you at the GGUF path.

NVFP4, which you used on Thor, needs SM110+, so INT4 AWQ is the Orin-side equivalent.

On avoiding docker: my numbers come from the container, but nothing about the speed is container-specific — same upstream wheels, same Marlin kernels — so a native install should land in the same place. I have not verified that myself, so I cannot answer your NCCL question from experience.

9

I actually tried this and then installed vllm. I am getting abysmal performance. It seems to completely fall back to CPU.

I’ll give it another try …

10

Hi,

Is the perf issue from PyTorch or vLLM?
Could you share some logs with us so we can check why it fallback to CPU?

Another alternative is to try our container. It has all the compatible packages installed.
vLLM: vLLM | NVIDIA NGC
PyTorch: PyTorch | NVIDIA NGC

Thanks.

11

@saskia.hold Before you retry — and to give @AastaLLL something concrete to look at in those logs — the symptom you describe has a specific and very common cause on aarch64, and it is worth checking before rebuilding anything.

pip3 install vllm after installing torch from the cu132 index will usually re-resolve torch and replace the CUDA build you just installed with whatever PyPI serves for linux_aarch64, which has historically been a CPU-only wheel. The install succeeds, nothing warns you, and everything then runs on CPU. That matches “completely falls back to CPU” exactly.

Check this before anything else:

python3 -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available(), torch.cuda.get_device_name(0))"

If torch.version.cuda is None, that is your answer — the vLLM install clobbered the cu132 torch. Install vLLM first, then force torch back from the cu132 index last, and pin it so nothing re-resolves it.

Two other things produce “abysmal” numbers on this box even when the GPU is being used:

1. jtop can under-report. Others in the r39.2 feedback thread saw GPU memory filling while jtop showed CPU-only. Read the load directly instead:

cat /sys/devices/platform/gpu.0/load

sampled during generation, not before. And check the vLLM startup log for the lines that prove the CUDA path was taken. On my box, with AWQ INT4:

[auto_awq.py:473] Using MarlinLinearKernel for AutoAWQMarlinLinearMethod
[cuda.py:482]     Using FLASH_ATTN attention backend

If those appear, you are not on CPU regardless of what the monitor says.

2. Discard your first benchmark pass. Cold torch.compile and CUDA-graph capture fire once per new batch shape, and on a fresh install there is no cache at all. My first AWQ run produced a non-monotonic concurrency curve — 8-way slower than 4-way — purely because capture cost landed inside the measured window. Warm, the same runs gave 262 and 496 tok/s. Treat any non-monotonic curve as a cold-cache artifact until proven otherwise. Natively, make sure ~/.cache/vllm persists between runs.

Also pass --gpu-memory-utilization 0.85. The default fails on Jetson because unified memory means the OS’s own share counts against the target, and it aborts with “Free memory on device cuda:0 (56.33/61.4 GiB) on startup is less than desired GPU memory utilization”. That one is not container-specific — it will bite you natively too.

Numbers to check yourself against — AGX Orin 64 GB, Qwen2.5-7B-Instruct-AWQ, 200-token generations, wall-clock: 33.5 tok/s single stream, 496 tok/s aggregate at 16 concurrent (31.0 per stream, so about 7% degradation at 16-way). If you are an order of magnitude below that, it is the torch wheel.

On NCCL: you are single-GPU at TP=1, so there are no collectives to perform. I have not run the native install myself, so I cannot confirm from experience that the wheels pull a working NCCL — only that it should not be on the critical path for your configuration.

12

thanks, folks -

  1. had some interesting issues. Stumbled into issue 6236259. The system was set to maxpower when rebooting. I should have reset the power mode prior to hitting reboot. Me bad. The system did not want to come back up, ethernet connections were dropped immediately. So I had to reflash. Now my Ethernet is flakey, I am seeing lots of connection timeouts. I could potentially attribute this to the switch (which is a low end no good netgear). sigh. On Wifi now.

  2. reinstalled jtop. It has “issues” and refrains from reporting the CPU.

  3. tried the docker as mentioned. Jtop did not report GPU usage for the docker. Looking at the GPU:

root@orin-agx:/home/saskia# cat /sys/devices/platform/gpu.0/load
998

also: I could see CUDA graph compile.

  1. will start pytorch recompile in a bit. I had it running with compile for the correct architecture.

  2. NCCL seems to be required by vllm v1, even if it is just routed to loopback.

13

ok, I got a baseline to start with.

I compiled the latest pytorch, torchaudio, torchvision for TargetArch 8.7.

Got VLLM to compile, too.

It is compiling the CUDA graphs.

image
image825×521 26.5 KB

It seems that I am getting somewhere.

Prefill is abysmal, the system is running at 50W power setting. Question to the experts: are there additional options to further speed up the inference (I know, the context length is way too high, this was just a first test.)

(test) saskia@orin-agx:~/build/vllm/dist$ vllm serve /home/saskia/models/Qwen2.5-7B-Instruct-AWQ --gpu-memory-utilization 0.5
W0729 21:46:13.790000 92517 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead
W0729 21:46:13.792000 92517 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead
W0729 21:46:13.793000 92517 torch/_library/opaque_object.py:211] typ=‘value’ is deprecated, use typ=‘constant’ instead
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:345]
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:345] █ █ █▄ ▄█
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:345] ▄▄ ▄█ █ █ █ ▀▄▀ █ version 0.26.1rc1.dev103+g381b69162.d20260729
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:345] █▄█▀ █ █ █ █ model /home/saskia/models/Qwen2.5-7B-Instruct-AWQ
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:345] ▀▀ ▀▀▀▀▀ ▀▀▀▀▀ ▀ ▀
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:345]
(APIServer pid=92517) INFO 07-29 21:46:32 [api_utils.py:273] non-default args: {‘model_tag’: ‘/home/saskia/models/Qwen2.5-7B-Instruct-AWQ’, ‘model’: ‘/home/saskia/models/Qwen2.5-7B-Instruct-AWQ’, ‘gpu_memory_utilization’: 0.5}
(APIServer pid=92517) INFO 07-29 21:46:53 [model.py:638] Resolved architecture: Qwen2ForCausalLM
(APIServer pid=92517) INFO 07-29 21:46:53 [model.py:1875] Using max model len 32768
(APIServer pid=92517) INFO 07-29 21:46:54 [vllm.py:1118] Asynchronous scheduling is enabled.
(APIServer pid=92517) INFO 07-29 21:46:54 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’])
W0729 21:47:03.768000 92634 torch/_opaque_base.py:6] torch._opaque_base is deprecated, use torch._custom_class_base instead
W0729 21:47:03.769000 92634 torch/_library/opaque_object.py:288] register_opaque_type is deprecated, use register_custom_class instead
W0729 21:47:03.771000 92634 torch/_library/opaque_object.py:211] typ=‘value’ is deprecated, use typ=‘constant’ instead
(EngineCore pid=92634) INFO 07-29 21:47:18 [core.py:121] Initializing a V1 LLM engine (v0.26.1rc1.dev103+g381b69162.d20260729) with config: model=‘/home/saskia/models/Qwen2.5-7B-Instruct-AWQ’, speculative_config=None, tokenizer=‘/home/saskia/models/Qwen2.5-7B-Instruct-AWQ’, skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32768, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=auto_awq, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend=‘auto’, disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser=‘’, reasoning_parser_plugin=‘’, enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False, jit_monitor_mode=‘warn’, jit_monitor_verbose=False), seed=0, served_model_name=/home/saskia/models/Qwen2.5-7B-Instruct-AWQ, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={‘mode’: <CompilationMode.VLLM_COMPILE: 3>, ‘debug_dump_path’: None, ‘cache_dir’: ‘’, ‘compile_cache_save_format’: ‘binary’, ‘backend’: ‘inductor’, ‘custom_ops’: [‘none’], ‘ir_enable_torch_wrap’: True, ‘splitting_ops’: [‘vllm::unified_attention_with_output’, ‘vllm::unified_mla_attention_with_output’, ‘vllm::mamba_mixer2’, ‘vllm::mamba_mixer’, ‘vllm::short_conv’, ‘vllm::linear_attention’, ‘vllm::qwen_gdn_attention_core’, ‘vllm::gdn_attention_core_xpu’, ‘vllm::olmo_hybrid_gdn_full_forward’, ‘vllm::kda_attention’, ‘vllm::sparse_attn_indexer’, ‘vllm::rocm_aiter_sparse_attn_indexer’, ‘vllm::deepseek_v4_attention’, ‘vllm::hpc_rope_norm_forward’, ‘vllm::unified_kv_cache_update’, ‘vllm::unified_mla_kv_cache_update’], ‘compile_mm_encoder’: False, ‘cudagraph_mm_encoder’: False, ‘encoder_cudagraph_token_budgets’: , ‘encoder_cudagraph_max_vision_items_per_batch’: 0, ‘encoder_cudagraph_max_frames_per_batch’: None, ‘compile_sizes’: , ‘compile_ranges_endpoints’: [2048], ‘inductor_compile_config’: {‘enable_auto_functionalized_v2’: False, ‘combo_kernels’: True, ‘benchmark_combo_kernel’: True}, ‘inductor_passes’: {}, ‘cudagraph_mode’: <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, ‘cudagraph_num_of_warmups’: 1, ‘cudagraph_capture_sizes’: [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], ‘cudagraph_copy_inputs’: False, ‘cudagraph_specialize_lora’: True, ‘use_inductor_graph_partition’: False, ‘pass_config’: {‘fuse_norm_quant’: False, ‘fuse_act_quant’: False, ‘fuse_attn_quant’: False, ‘enable_sp’: False, ‘fuse_gemm_comms’: False, ‘fuse_allreduce_rms’: False, ‘enable_qk_norm_rope_fusion’: False, ‘fuse_rope_kvcache_cat_mla’: False, ‘fuse_act_padding’: False, ‘fuse_qk_norm_rope_kvcache’: False}, ‘max_cudagraph_capture_size’: 512, ‘dynamic_shapes_config’: {‘type’: <DynamicShapesType.BACKED: ‘backed’>, ‘evaluate_guards’: False, ‘assume_32_bit_indexing’: False}, ‘local_cache_dir’: None, ‘fast_moe_cold_start’: False, ‘static_all_moe_layers’: }, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’]), enable_flashinfer_autotune=True, enable_cutedsl_warmup=True, enable_jit_warmup=True, enable_bf16x3_router_gemm=False, moe_backend=‘auto’, linear_backend=‘auto’)
(EngineCore pid=92634) INFO 07-29 21:47:18 [parallel_state.py:1640] world_size=1 rank=0 local_rank=0 distributed_init_method=tcp://192.168.178.41:45063 backend=nccl
(EngineCore pid=92634) INFO 07-29 21:47:18 [parallel_state.py:1977] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, PCP rank 0, TP rank 0, EP rank N/A, EPLB rank N/A
(EngineCore pid=92634) INFO 07-29 21:47:18 [gpu_worker.py:385] Using V2 Model Runner
(EngineCore pid=92634) INFO 07-29 21:47:19 [model_runner.py:295] Loading model from scratch…
(EngineCore pid=92634) INFO 07-29 21:47:19 [auto_awq.py:473] Using MarlinLinearKernel for AutoAWQMarlinLinearMethod
(EngineCore pid=92634) INFO 07-29 21:47:21 [cuda.py:482] Using FLASH_ATTN attention backend out of potential backends: [‘FLASH_ATTN’, ‘FLASHINFER’, ‘TRITON_ATTN’, ‘FLEX_ATTENTION’].
(EngineCore pid=92634) INFO 07-29 21:47:21 [flash_attn.py:789] Using FlashAttention version 2
(EngineCore pid=92634) INFO 07-29 21:47:22 [weight_utils.py:867] Filesystem type for checkpoints: EXT4. Checkpoint size: 5.19 GiB. Available RAM: 51.72 GiB.
(EngineCore pid=92634) INFO 07-29 21:47:22 [weight_utils.py:890] Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre). If you want to force prefetching, start vLLM with --safetensors-load-strategy=prefetch.
Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:01<00:01, 1.19s/it]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.22it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:01<00:00, 1.14it/s]
(EngineCore pid=92634)
(EngineCore pid=92634) INFO 07-29 21:47:24 [default_loader.py:430] Loading weights took 1.83 seconds
(EngineCore pid=92634) INFO 07-29 21:47:30 [model_runner.py:316] Model loading took 5.29 GiB and 10.699937 seconds
(EngineCore pid=92634) INFO 07-29 21:47:30 [topk_topp_sampler.py:55] Using FlashInfer for top-p & top-k sampling.
(EngineCore pid=92634) INFO 07-29 21:47:47 [backends.py:1094] Using cache directory: /home/saskia/.cache/vllm/torch_compile_cache/bb421a0b5e/rank_0_0/backbone for vLLM’s torch.compile
(EngineCore pid=92634) INFO 07-29 21:47:47 [backends.py:1155] Dynamo bytecode transform time: 15.79 s
(EngineCore pid=92634) INFO 07-29 21:48:02 [backends.py:393] Compiling a graph for compile range (1, 2048) takes 14.79 s
(EngineCore pid=92634) INFO 07-29 21:48:13 [backends.py:920] collected artifacts: 29 entries, 3 artifacts, 4049055 bytes total
(EngineCore pid=92634) INFO 07-29 21:48:13 [decorators.py:708] saved AOT compiled function to /home/saskia/.cache/vllm/torch_compile_cache/torch_aot_compile/060a295c7be4b828924994b87f9fcc9057b917b011652a16f16a31b22595e381/rank_0_0/model
(EngineCore pid=92634) INFO 07-29 21:48:13 [monitor.py:53] torch.compile took 41.91 s in total
(EngineCore pid=92634) INFO 07-29 21:48:13 [monitor.py:81] Initial profiling/warmup run took 0.18 s
(EngineCore pid=92634) INFO 07-29 21:48:17 [gpu_worker.py:563] Available KV cache memory: 22.32 GiB
(EngineCore pid=92634) INFO 07-29 21:48:17 [kv_cache_utils.py:2214] GPU KV cache size: 417,872 tokens
(EngineCore pid=92634) INFO 07-29 21:48:17 [kv_cache_utils.py:2215] Maximum concurrency for 32,768 tokens per request: 12.75x
(EngineCore pid=92634) INFO 07-29 21:48:23 [cutedsl_warmup.py:105] Skipping CuTeDSL warmup because no compile units were requested.
Capturing CUDA graphs (PIECEWISE): 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 51/51 [00:17<00:00, 2.88it/s]
Capturing CUDA graphs (FULL): 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 35/35 [00:06<00:00, 5.53it/s]
(EngineCore pid=92634) INFO 07-29 21:48:48 [model_runner.py:776] Graph capturing finished in 25 secs, took 1.17 GiB
(EngineCore pid=92634) INFO 07-29 21:48:48 [gpu_worker.py:789] Free memory on device (56.94/61.35 GiB) on startup. Desired GPU memory utilization is (0.5, 30.67 GiB). Actual usage is 7.32 GiB for consumed memory (weights + non-torch), 1.04 GiB for peak activation, and 1.17 GiB for CUDAGraph memory. Replace gpu_memory_utilization config with --kv-cache-memory=22545779200 (21.0 GiB) to fit into requested memory, or --kv-cache-memory=50754564608 (47.27 GiB) to fully utilize gpu memory. Current kv cache memory in use is 22.32 GiB.
(EngineCore pid=92634) INFO 07-29 21:51:59 [jit_monitor.py:79] Kernel JIT monitor activated; monitored JIT compilations during inference will use mode=warn.
(EngineCore pid=92634) INFO 07-29 21:52:00 [core.py:348] init engine (profile, create kv cache, warmup model) took 270.15 s (compilation: 41.91 s)
(EngineCore pid=92634) INFO 07-29 21:52:01 [vllm.py:1118] Asynchronous scheduling is enabled.
(EngineCore pid=92634) INFO 07-29 21:52:01 [kernel.py:306] Final IR op priority after setting platform defaults: IrOpPriorityConfig(rms_norm=[‘native’], fused_add_rms_norm=[‘native’])
(APIServer pid=92517) INFO 07-29 21:52:01 [api_server.py:676] Supported tasks: [‘generate’]
(APIServer pid=92517) WARNING 07-29 21:52:02 [model.py:1629] Default vLLM sampling parameters have been overridden by the model’s generation_config.json: {'repetition_penalty': 1.05, 'temperature': 0.7, 'top_k': 20, 'top_p': 0.8}. If this is not intended, please relaunch vLLM instance with --generation-config vllm.
(APIServer pid=92517) INFO 07-29 21:52:03 [hf.py:540] Detected the chat template content format to be ‘string’. You can set --chat-template-content-format to override this.
(APIServer pid=92517) INFO 07-29 21:52:03 [api_server.py:680] Starting vLLM server on http://0.0.0.0:8000
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:37] Available routes are:
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /openapi.json, Methods: HEAD, GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /docs, Methods: HEAD, GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /docs/oauth2-redirect, Methods: HEAD, GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /redoc, Methods: HEAD, GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /load, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /version, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /health, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /metrics, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /tokenize, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /detokenize, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/models, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /ping, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /ping, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /invocations, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/chat/completions, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/chat/completions/batch, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/responses, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/responses/{response_id}, Methods: GET
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/responses/{response_id}/cancel, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/completions, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/messages, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/messages/count_tokens, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /generative_scoring, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /scale_elastic_ep, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /is_scaling_elastic_ep, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/chat/completions/derender, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /v1/completions/derender, Methods: POST
(APIServer pid=92517) INFO 07-29 21:52:03 [launcher.py:46] Route: /inference/v1/generate, Methods: POST
(APIServer pid=92517) INFO: Started server process [92517]
(APIServer pid=92517) INFO: Waiting for application startup.
(APIServer pid=92517) INFO: Application startup complete.
(APIServer pid=92517) INFO 07-29 21:55:24 [loggers.py:310] Engine 000: Avg prompt throughput: 6.0 tokens/s, Avg generation throughput: 2.3 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=92517) INFO 07-29 21:55:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.6 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 0.0%
(APIServer pid=92517) INFO: 127.0.0.1:50056 - “POST /v1/chat/completions HTTP/1.1” 200 OK
(APIServer pid=92517) INFO 07-29 21:55:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 19.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=92517) INFO 07-29 21:55:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 0.0%
(APIServer pid=92517) INFO 07-29 22:00:14 [loggers.py:310] Engine 000: Avg prompt throughput: 8.3 tokens/s, Avg generation throughput: 12.2 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 10.1%
(APIServer pid=92517) INFO 07-29 22:00:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 10.1%
(APIServer pid=92517) INFO: 127.0.0.1:33114 - “POST /v1/chat/completions HTTP/1.1” 200 OK
(APIServer pid=92517) INFO 07-29 22:00:34 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 9.5 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 10.1%
(APIServer pid=92517) INFO 07-29 22:00:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 10.1%
(APIServer pid=92517) INFO 07-29 22:02:44 [loggers.py:310] Engine 000: Avg prompt throughput: 0.3 tokens/s, Avg generation throughput: 3.9 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 43.4%
(APIServer pid=92517) INFO 07-29 22:02:54 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.1%, Prefix cache hit rate: 43.4%
(APIServer pid=92517) INFO 07-29 22:03:04 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 29.5 tokens/s, Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.2%, Prefix cache hit rate: 43.4%
(APIServer pid=92517) INFO: 127.0.0.1:46764 - “POST /v1/chat/completions HTTP/1.1” 200 OK
(APIServer pid=92517) INFO 07-29 22:03:14 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 25.3 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 43.4%
(APIServer pid=92517) INFO 07-29 22:03:24 [loggers.py:310] Engine 000: Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Waiting: 0 reqs, GPU KV cache usage: 0.0%, Prefix cache hit rate: 43.4%

Summary

This text will be hidden

14

Hi,

The log seems to be truncated.
Based on the log shared so far, vLLM appears to be working as expected.

Are you still experiencing any performance issues after building it manually?

Thanks.

15

here is what I found during testing:

Orin AGX, 50W power setting.

Models like Huihui-Qwen3.5-9B-abliterated-AWQ give me about 600 token/s in prefill and roughly 30 tokens/s during decode for a single stream. This is the correct ballpark figure. I would now need to test for concurrency, but from what I can tell, VLLM and torch are now utilizing the GPU, even though JTOP is not reporting that. Why, I cannot say. With FP8 and Marlin kernels, throughput drops to ~9 tokens/s (which is not unexpected).

I would need to test for concurrent requests, but have not done so yet.

In case someone wants the wheels that I built for pytorch, torchaudio, torchvision, vllm: I’ll make them available on request, drop me a note.

Apart from that. I would consider vllm to be working as expected now, 4 or 5 models launched without issues, I am getting the expected throughput, and the vllm log shows that hw accelerated kernels are being utilized.

Thanks everybody

16

a quick update: I got my Orin AGX up and running with FenomAI-Qwen3.6-35B-AWQ-4bit. Prefill is above 600 token/s, decode constantly at about 26 tokens/s. The model is delivering good results for translations and stoichiometry. I am happy with that result, this is absolutely usable.

Thank you for looking into this.

17

For jtop to upgrade to current release please use:

curl -LsSf https://raw.githubusercontent.com/rbonghi/jetson_stats/master/scripts/upgrade-jtop.sh | bash
20

Thank you - I think I installed the latest jtop, but it did not report graphics utilization.

Anyway, that particular Orin is now at a friend’s to serve Qwen. I have another Orin AGX here that does not show me the UEFI when logging in. It is booting into Ubuntu 20.something, but no UEFI screen upon powerup. I guess this calls for a reflash from Software Manager …

sigh

21

ok, flash successful, it is back among the living. I’ll revisit the jtop issue …

22

Hi,

Do you meet an issue with jtop?
Or does the second device work well after reflashing?

Thanks.

23

ok, to sum up:

  • jtop works and shows GPU stats after updating it.
  • vllm compile for the sm_87 architecture works, rendering 27+ tokens/s with Qwen3.6-35B-A3B at 50W power setting. Multi Token Prediction works as well, however it does not give me any noticable performance boost on this architecture.
  • reflashing the 2nd Orin unit from SDK Manager got me the UEFI menu and the display output back. The unit is now at JP7.2 and running fine.

Thanks everybody.