Issue Running VSS on Jetson Thor

[abdelrahman.hosny]

• Hardware Platform (GPU model and numbers): Jetson Thor dev kit

• Ubuntu Version: Ubuntu 24.04

• NVIDIA GPU Driver Version (valid for GPU only): 580.00

• Issue Type( questions, new requirements, bugs): bugs

• How to reproduce the issue:

We are running VSS demo on a Jetson Thor device using the remote_llm_deployment

We are following resources from, deploying using docker compose:

• https://github.dev/NVIDIA-AI-Blueprints/video-search-and-summarization
• Deploy Using Docker Compose ARM — Video Search and Summarization Agent

To reproduce: docker compose up using the below.env file.

The UI on port 9100 is not opening.

docker logs -f remote_llm_deployment-via-server-1
GPU has 2 decode engines
Free GPU memory is [N/A] MiB
/opt/nvidia/via/start_via.sh: line 72: [: [N/A]: integer expression expected
Total GPU memory is 125772 MiB per GPU
Auto-selecting VLM Batch Size to 128
Using cosmos-reason1
Starting VIA server in release mode
2025-12-23 15:03:44,014 INFO Initializing VIA Stream Handler
INFO:     Started server process [160]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:60000 (Press CTRL+C to quit)
2025-12-23 15:14:57,173 INFO Loaded Guardrails
2025-12-23 15:14:57,177 INFO {'gdino_engine': '/root/.via/ngc_model_cache//cv_pipeline_models/swin.fp16.engine', 'tracker_config': '/tmp/via_tracker_config.yml', 'inference_interval': 1}
2025-12-23 15:14:57,178 INFO Initializing VLM pipeline
2025-12-23 15:14:57,188 INFO Have peer access: True
2025-12-23 15:14:57,189 INFO Using model cached at /root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic
2025-12-23 15:14:57,190 INFO GPUs per VLM instance: 1
2025-12-23 15:14:57,190 INFO num_vlm_procs set to 1
2025-12-23 15:15:02,983 INFO Initializing VlmProcess-0
2025-12-23 15:15:02,983 INFO Initializing DecoderProcess-0
2025-12-23 15:15:05,111 INFO Cosmos Reason1 default system prompt: Please provide captions of all the events in the video with timestamps using the following format: <start time> <end time> caption of event 1.
<start time> <end time> caption of event 2.
At each frame, the timestamp is embedded at the bottom of the video. You need to extract the timestamp and answer the user question.
2025-12-23 15:15:05,111 INFO Using VLLM model for cosmos-reason1
2025-12-23 15:15:05,178 INFO Warmup DecoderProcess-0
/bin/dash: 1: lsmod: not found
Opening in BLOCKING MODE 
2025-12-23 15:15:05,583 INFO Video stream found.
Opening in BLOCKING MODE 
2025-12-23 15:15:05,903 INFO Video stream found.
Opening in BLOCKING MODE 
2025-12-23 15:15:06,112 INFO Video stream found.
Opening in BLOCKING MODE 
2025-12-23 15:15:06,331 INFO Video stream found.
Opening in BLOCKING MODE 
2025-12-23 15:15:06,568 INFO Video stream found.
Opening in BLOCKING MODE 
2025-12-23 15:15:06,820 INFO Video stream found.
2025-12-23 15:15:06,954 INFO Warmup DecoderProcess-0 done
2025-12-23 15:15:06,955 INFO Initialized DecoderProcess-0
INFO 12-23 15:15:07 [__init__.py:244] Automatically detected platform cuda.
2025-12-23 15:15:10,539 INFO Initializing Cosmos-Reason1-7B from: /root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic
INFO 12-23 15:15:21 [config.py:841] This model supports multiple tasks: {'embed', 'classify', 'generate', 'reward'}. Defaulting to 'generate'.
INFO 12-23 15:15:21 [config.py:1472] Using max model len 20480
INFO 12-23 15:15:21 [config.py:2285] Chunked prefill is enabled with max_num_batched_tokens=5120.
INFO 12-23 15:15:28 [__init__.py:244] Automatically detected platform cuda.
INFO 12-23 15:15:30 [core.py:526] Waiting for init message from front-end.
INFO 12-23 15:15:30 [core.py:69] Initializing a V1 LLM engine (v0.9.2+03bb1cdd.nv25.08.post1-stage-alexandrem-vss) with config: model='/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic', speculative_config=None, tokenizer='/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=20480, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":null}
[W1223 15:15:31.002821246 ProcessGroupNCCL.cpp:990] Warning: TORCH_NCCL_AVOID_RECORD_STREAMS is the default now, this environment variable is thus deprecated. (function operator())
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
[Gloo] Rank 0 is connected to 0 peer ranks. Expected number of connected peer ranks is : 0
INFO 12-23 15:15:31 [parallel_state.py:1076] rank 0 in world size 1 is assigned as DP rank 0, PP rank 0, TP rank 0, EP rank 0
INFO 12-23 15:15:31 [topk_topp_sampler.py:49] Using FlashInfer for top-p & top-k sampling.
INFO 12-23 15:15:31 [gpu_model_runner.py:1770] Starting to load model /root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic...
INFO 12-23 15:15:32 [gpu_model_runner.py:1775] Loading model from scratch...
INFO 12-23 15:15:32 [cuda.py:284] Using Flash Attention backend on V1 engine.
Loading safetensors checkpoint shards: 100% 3/3 [00:03<00:00,  1.22s/it]
INFO 12-23 15:15:36 [default_loader.py:272] Loading weights took 3.79 seconds
INFO 12-23 15:15:37 [gpu_model_runner.py:1801] Model loading took 9.5227 GiB and 4.526460 seconds
INFO 12-23 15:15:37 [gpu_model_runner.py:2238] Encoder cache will be initialized with a budget of 16384 tokens, and profiled with 1 video items of the maximum feature size.
[rank0]:W1223 15:15:44.967000 71821 torch/_inductor/utils.py:1416] [3/0] Not enough SMs to use max_autotune_gemm mode
AUTOTUNE addmm(65536x3840, 65536x1280, 1280x3840)
strides: [0, 1], [1280, 1], [1, 1280]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16
  bias_addmm 5.8386 ms 100.0% 
  addmm 11.7133 ms 49.8% 
SingleProcess AUTOTUNE benchmarking takes 0.3072 seconds and 0.0002 seconds precompiling for 2 choices
AUTOTUNE addmm(16384x3584, 16384x5120, 5120x3584)
strides: [0, 1], [5120, 1], [1, 5120]
dtypes: torch.bfloat16, torch.bfloat16, torch.bfloat16
  bias_addmm 9.4843 ms 100.0% 
  addmm 12.0674 ms 78.6% 
SingleProcess AUTOTUNE benchmarking takes 4.9629 seconds and 0.0002 seconds precompiling for 2 choices
INFO 12-23 15:16:55 [backends.py:506] vLLM's torch.compile cache is disabled.
INFO 12-23 15:16:55 [backends.py:519] Dynamo bytecode transform time: 5.59 s
INFO 12-23 15:16:57 [backends.py:181] Cache the graph of shape None for later use
INFO 12-23 15:17:21 [backends.py:193] Compiling a graph for general shape takes 25.76 s
INFO 12-23 15:17:23 [monitor.py:34] torch.compile takes 31.35 s in total
INFO 12-23 15:19:04 [gpu_worker.py:232] Available KV cache memory: 32.58 GiB
INFO 12-23 15:19:04 [kv_cache_utils.py:716] GPU KV cache size: 610,080 tokens
INFO 12-23 15:19:04 [kv_cache_utils.py:720] Maximum concurrency for 20,480 tokens per request: 29.79x
Capturing CUDA graph shapes: 100% 67/67 [00:48<00:00,  1.39it/s]
INFO 12-23 15:19:53 [gpu_model_runner.py:2326] Graph capturing finished in 48 secs, took 0.72 GiB
INFO 12-23 15:19:53 [core.py:172] init engine (profile, create kv cache, warmup model) took 256.49 seconds
INFO 12-23 15:19:54 [loggers.py:137] Engine 000: vllm cache_config_info with initialization after num_gpu_blocks is: 38130
2025-12-23 15:19:55,472 INFO Max batch size: 128
2025-12-23 15:19:55,473 INFO Cosmos Reason1 VLLM model initialized successfully
2025-12-23 15:19:55,473 INFO Warmup VlmProcess-0
2025-12-23 15:19:55,474 INFO Starting model warmup...
INFO 12-23 15:20:01 [async_llm.py:270] Added request 453fd9f1-35c1-465a-b721-446badef0216.
ERROR 12-23 15:20:15 [dump_input.py:69] Dumping input data for V1 LLM engine (v0.9.2+03bb1cdd.nv25.08.post1-stage-alexandrem-vss) with config: model='/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic', speculative_config=None, tokenizer='/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config={}, tokenizer_revision=None, trust_remote_code=False, dtype=torch.bfloat16, max_seq_len=20480, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=compressed-tensors, enforce_eager=False, kv_cache_dtype=auto,  device_config=cuda, decoding_config=DecodingConfig(backend='auto', disable_fallback=False, disable_any_whitespace=False, disable_additional_properties=False, reasoning_backend=''), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None), seed=0, served_model_name=/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, pooler_config=None, compilation_config={"level":3,"debug_dump_path":"","cache_dir":"/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic/.vllm/torch_compile_cache/8a280991cc","backend":"","custom_ops":[],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"inductor_compile_config":{"enable_auto_functionalized_v2":false},"inductor_passes":{},"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"cudagraph_copy_inputs":false,"full_cuda_graph":false,"max_capture_size":512,"local_cache_dir":"/root/.via/ngc_model_cache/nim_nvidia_cosmos-reason1-7b_1.1-fp8-dynamic/.vllm/torch_compile_cache/8a280991cc/rank_0_0/backbone"}, 
ERROR 12-23 15:20:15 [dump_input.py:76] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=453fd9f1-35c1-465a-b721-446badef0216,prompt_token_ids_len=172,mm_inputs=[{'video_grid_thw': tensor([[ 4, 10,  8]]), 'second_per_grid_ts': tensor([1.], dtype=torch.bfloat16), 'pixel_values_videos': tensor([[-1.7812, -1.7812, -1.7812,  ..., -1.4688, -1.4688, -1.4688],
ERROR 12-23 15:20:15 [dump_input.py:76]         [-1.7812, -1.7812, -1.7812,  ..., -1.4688, -1.4688, -1.4688],
ERROR 12-23 15:20:15 [dump_input.py:76]         [-1.7812, -1.7812, -1.7812,  ..., -1.4688, -1.4688, -1.4688],
ERROR 12-23 15:20:15 [dump_input.py:76]         ...,
ERROR 12-23 15:20:15 [dump_input.py:76]         [-1.7891, -1.7891, -1.7891,  ..., -1.1797, -1.4766, -1.4766],
ERROR 12-23 15:20:15 [dump_input.py:76]         [-1.7891, -1.7891, -1.7891,  ..., -1.4766, -1.4766, -1.4766],
ERROR 12-23 15:20:15 [dump_input.py:76]         [-1.7891, -1.7891, -1.7891,  ..., -1.4766, -1.4766, -1.4766]],
ERROR 12-23 15:20:15 [dump_input.py:76]        dtype=torch.bfloat16)}],mm_hashes=['de68b4624477db76fab93e802e2e6a6553ebbd3a2188863730c5b95b7ef73cfc'],mm_positions=[PlaceholderRange(offset=86, length=80, is_embed=None)],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.1, temperature=0.7, top_p=0.9, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[151643], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=50, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None, extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],),num_computed_tokens=0,lora_request=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[], resumed_from_preemption=[], new_token_ids=[], new_block_ids=[], num_computed_tokens=[]), num_scheduled_tokens={453fd9f1-35c1-465a-b721-446badef0216: 172}, total_num_scheduled_tokens=172, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={453fd9f1-35c1-465a-b721-446badef0216: [0]}, num_common_prefix_blocks=[11], finished_req_ids=[], free_encoder_input_ids=[], structured_output_request_ids={}, grammar_bitmask=null, kv_connector_metadata=null)
ERROR 12-23 15:20:15 [dump_input.py:79] Dumping scheduler stats: SchedulerStats(num_running_reqs=1, num_waiting_reqs=0, kv_cache_usage=0.00031471282454764715, prefix_cache_stats=PrefixCacheStats(reset=False, requests=1, queries=172, hits=0), spec_decoding_stats=None, num_corrupted_reqs=0)
ERROR 12-23 15:20:15 [core.py:588] EngineCore encountered a fatal error.
ERROR 12-23 15:20:15 [core.py:588] Traceback (most recent call last):
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 579, in run_engine_core
ERROR 12-23 15:20:15 [core.py:588]     engine_core.run_busy_loop()
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 606, in run_busy_loop
ERROR 12-23 15:20:15 [core.py:588]     self._process_engine_step()
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 631, in _process_engine_step
ERROR 12-23 15:20:15 [core.py:588]     outputs, model_executed = self.step_fn()
ERROR 12-23 15:20:15 [core.py:588]                               ^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 235, in step
ERROR 12-23 15:20:15 [core.py:588]     model_output = self.execute_model(scheduler_output)
ERROR 12-23 15:20:15 [core.py:588]                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 221, in execute_model
ERROR 12-23 15:20:15 [core.py:588]     raise err
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 212, in execute_model
ERROR 12-23 15:20:15 [core.py:588]     return self.model_executor.execute_model(scheduler_output)
ERROR 12-23 15:20:15 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
ERROR 12-23 15:20:15 [core.py:588]     output = self.collective_rpc("execute_model",
ERROR 12-23 15:20:15 [core.py:588]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
ERROR 12-23 15:20:15 [core.py:588]     answer = run_method(self.driver_worker, method, args, kwargs)
ERROR 12-23 15:20:15 [core.py:588]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2736, in run_method
ERROR 12-23 15:20:15 [core.py:588]     return func(*args, **kwargs)
ERROR 12-23 15:20:15 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 12-23 15:20:15 [core.py:588]     return func(*args, **kwargs)
ERROR 12-23 15:20:15 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 308, in execute_model
ERROR 12-23 15:20:15 [core.py:588]     output = self.model_runner.execute_model(scheduler_output,
ERROR 12-23 15:20:15 [core.py:588]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 12-23 15:20:15 [core.py:588]     return func(*args, **kwargs)
ERROR 12-23 15:20:15 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1321, in execute_model
ERROR 12-23 15:20:15 [core.py:588]     self._execute_mm_encoder(scheduler_output)
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1020, in _execute_mm_encoder
ERROR 12-23 15:20:15 [core.py:588]     curr_group_outputs = self.model.get_multimodal_embeddings(
ERROR 12-23 15:20:15 [core.py:588]                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1086, in get_multimodal_embeddings
ERROR 12-23 15:20:15 [core.py:588]     video_embeddings = self._process_video_input(multimodal_input)
ERROR 12-23 15:20:15 [core.py:588]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1045, in _process_video_input
ERROR 12-23 15:20:15 [core.py:588]     return video_embeds.split(sizes.tolist())
ERROR 12-23 15:20:15 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588]   File "/usr/local/lib/python3.12/dist-packages/torch/_tensor.py", line 1053, in split
ERROR 12-23 15:20:15 [core.py:588]     return torch._VF.split_with_sizes(self, split_size, dim)
ERROR 12-23 15:20:15 [core.py:588]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [core.py:588] RuntimeError: split_with_sizes expects split_sizes have only non-negative entries, but got split_sizes=[-2020506758715646411]
Process EngineCore_0:
Traceback (most recent call last):
ERROR 12-23 15:20:15 [async_llm.py:419] AsyncLLM output_handler failed.
ERROR 12-23 15:20:15 [async_llm.py:419] Traceback (most recent call last):
ERROR 12-23 15:20:15 [async_llm.py:419]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 378, in output_handler
ERROR 12-23 15:20:15 [async_llm.py:419]     outputs = await engine_core.get_output_async()
ERROR 12-23 15:20:15 [async_llm.py:419]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 12-23 15:20:15 [async_llm.py:419]   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 740, in get_output_async
ERROR 12-23 15:20:15 [async_llm.py:419]     raise self._format_exception(outputs) from None
ERROR 12-23 15:20:15 [async_llm.py:419] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 590, in run_engine_core
    raise e
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 579, in run_engine_core
    engine_core.run_busy_loop()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 606, in run_busy_loop
    self._process_engine_step()
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 631, in _process_engine_step
    outputs, model_executed = self.step_fn()
                              ^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 235, in step
    model_output = self.execute_model(scheduler_output)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 221, in execute_model
    raise err
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 212, in execute_model
    return self.model_executor.execute_model(scheduler_output)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/abstract.py", line 87, in execute_model
    output = self.collective_rpc("execute_model",
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/executor/uniproc_executor.py", line 57, in collective_rpc
    answer = run_method(self.driver_worker, method, args, kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/utils/__init__.py", line 2736, in run_method
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_worker.py", line 308, in execute_model
    output = self.model_runner.execute_model(scheduler_output,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1321, in execute_model
    self._execute_mm_encoder(scheduler_output)
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/worker/gpu_model_runner.py", line 1020, in _execute_mm_encoder
    curr_group_outputs = self.model.get_multimodal_embeddings(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1086, in get_multimodal_embeddings
    video_embeddings = self._process_video_input(multimodal_input)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/models/qwen2_5_vl.py", line 1045, in _process_video_input
    return video_embeds.split(sizes.tolist())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/_tensor.py", line 1053, in split
    return torch._VF.split_with_sizes(self, split_size, dim)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: split_with_sizes expects split_sizes have only non-negative entries, but got split_sizes=[-2020506758715646411]
INFO 12-23 15:20:15 [async_llm.py:345] Request 453fd9f1-35c1-465a-b721-446badef0216 failed (engine dead).
[rank0]:[W1223 15:20:16.450285082 ProcessGroupNCCL.cpp:1538] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
Process VlmProcess-1:
Traceback (most recent call last):
  File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
    self.run()
  File "/opt/nvidia/via/via-engine/vlm_pipeline/process_base.py", line 245, in run
    self._warmup()
  File "/opt/nvidia/via/via-engine/vlm_pipeline/vlm_pipeline.py", line 787, in _warmup
    self._model.warmup()
  File "/opt/nvidia/via/via-engine/models/cosmos_reason1/cosmos_reason1_model.py", line 350, in warmup
    ret = ret.result()
          ^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/nvidia/via/via-engine/models/cosmos_reason1/cosmos_reason1_model.py", line 310, in process_async_vllm
    output = asyncio.run_coroutine_threadsafe(generate_async(), self._event_loop).result()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/opt/nvidia/via/via-engine/models/cosmos_reason1/cosmos_reason1_model.py", line 300, in generate_async
    async for output_item in self._llm.generate(
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 326, in generate
    out = q.get_nowait() or await q.get()
                            ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 57, in get
    raise output
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 378, in output_handler
    outputs = await engine_core.get_output_async()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 740, in get_output_async
    raise self._format_exception(outputs) from None
vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

The .env file is:

export NVIDIA_API_KEY=<redacted>
export NGC_API_KEY=<redacted>

#Adjust ports if needed
export FRONTEND_PORT=9100
export BACKEND_PORT=8100

#Change default user and pass if needed 
export GRAPH_DB_USERNAME=neo4j
export GRAPH_DB_PASSWORD=password

#Update paths local paths to config files if needed. If it appears VSS is not using these configurations, then change the relative paths to absolute paths. 
export CA_RAG_CONFIG=./config.yaml
export GUARDRAILS_CONFIG=./guardrails

export ENABLE_AUDIO=false

export RIVA_ASR_SERVER_URI="grpc.nvcf.nvidia.com"
export RIVA_ASR_GRPC_PORT=443
export RIVA_ASR_SERVER_IS_NIM=true
export RIVA_ASR_SERVER_USE_SSL=true
export RIVA_ASR_SERVER_API_KEY=nvapi-*** #FIXME - api key RIVA ASR server
export RIVA_ASR_SERVER_FUNC_ID="d8dd4e9b-fbf5-4fb0-9dba-8cf436c8d965"


export DISABLE_CV_PIPELINE=false
export INSTALL_PROPRIETARY_CODECS=true	# Set to true when enabling CV

#Set VLM to Cosmos-Reason1
export VLM_MODEL_TO_USE=cosmos-reason1
export MODEL_PATH=ngc:nim/nvidia/cosmos-reason1-7b:1.1-fp8-dynamic

#Adjust misc configs if needed
export DISABLE_GUARDRAILS=false
export NVIDIA_VISIBLE_DEVICES=0 #For H100 Deployment
#export NVIDIA_VISIBLE_DEVICES=0,1,2 #For L40S Deployment

[junshengy]

Try this script to clean up memory. I’ve tried it on agx thor and it works.

deploy/scripts/sys_cache_cleaner.sh

Prerequisites (NVIDIA Jetson Thor) - NVIDIA Jetson Thor — Video Search and...

[abdelrahman.hosny]

I left this script already running in another terminal. I stopped it, and ran it again. Then, I ran docker compose up and after some logging, the same error occurs. Is this something that is expected to prevent the server from coming up on :9100 ? If so, how should I debug it? I don’t seem to see the scripts on the same repo.

[junshengy]

@abdelrahman.hosny

export NGC_API_KEY=<redacted>

The reason port 9100 is inaccessible is that the VSS server is not running correctly. Please first correctly set the NGC_API_KEY in your .env file to properly pull the cosmos-reason1 model.

You can try mounting the VSS code directory into a container to try debugging. like this topic.

Alert creation overrides user-defined summarization prompts in VSS UI

It should be included in the next release; please wait for the official announcement for the exact date. As a workaround, you can add the following configuration to your compose.yml file to override the code in the container with your local code for debugging. volumes: - xxxx/video-search-and-summarization/src/vss-engine/src:/opt/nvidia/via/via-engine

This GitHub user tried the same remote_llm_deployment mode and it worked after executing sys_cache_cleaner.sh.

Deployment failure on Jetson Thor (via-server crashes during “Starting model warmup…”)