Unable to start VSS reasoning properly

jefferson.chiu · April 16, 2026, 9:12am

Hardware Platform (GPU model and numbers) AGX-Thor
System Memory 64GB
Ubuntu Version 24.04.3 LTS
NVIDIA GPU Driver Version (valid for GPU only) 580.00
Issue Type questions

Hi, I tried to deploy VSS 3.1 on the said machine and mostly followed the steps in Prerequisites and Quickstart and I was only able to enter the VSS GUI page http://<HOST_IP>:3000/ at most and reasoning couldn’t be done successfully.

VSS UI reasoning attempt

VSS UI reasoning attempt - 2

To dict all the commands I have entered doesn’t seem to be necessary(or does it?), so I’m first to just mention the last few commands.

### Download Sample Data From NGC

mkdir sample_data

ngc registry resource download-version nvidia/vss-developer/dev-profile-sample-data:3.1.0

tar -xf dev-profile-sample-data_v3.1.0/dev-profile-sample-data.tar.gz -C ./sample_data

rm -rf dev-profile-sample-data_v3.1.0

### Download the Deployment Package

# Install Git LFS if not already installed

sudo apt install git-lfs

git lfs install

git clone GitHub - NVIDIA-AI-Blueprints/video-search-and-summarization: Blueprint for Ingesting massive volumes of live or archived videos and extract insights for summarization and interactive Q&A · GitHub

cd video-search-and-summarization

git lfs install

git lfs pull

The last command I entered was as follows. Is this the command that runs VLM locally and LLM remotely?

export NGC_CLI_API_KEY=(MY CLI API KEY)
export NVIDIA_API_KEY=(MY NIM CATALOG API KEY)
export LLM_ENDPOINT_URL=‘https://integrate.api.nvidia.com’

scripts/dev-profile.sh up -p base
-H AGX-THOR
–use-remote-llm
–llm ‘nvidia/nvidia-nemotron-nano-9b-v2’

Spent roughly 30 minutes on pulling all the 12 docker images and then another 20+ minutes to create the containers because there was one container, rtvi-vlm, which wasn’t able to start succesfully and instead got skipped and docker ps showed unhealthy.

Skipped rtvi-vlm

Related containers(one unhealthy)

Also, there’re two places where users are required to enter API keys when I follow the Prerequisites and Quickstart pages. One is ngc config set, and the other one was docker login nvcr.io

The first one(NGC_CLI_API_KEY) whose Key name was set as VSS was created based on the instruction in Prerequisites page:

1. Once logged in, click on your username in the top right corner
2. Select Setup from the dropdown menu
3. Navigate to API Keys under the Keys/Secrets section
4. Click Generate API Key
5. Click Generate Personal Key (available at top-right of the page)
6. Provide a descriptive name for your API key (e.g., “VSS Blueprint Development”)
7. Select NGC Catalog for Key Permissions
8. Click Generate Personal Key
9. Copy the generated API key immediately - you won’t be able to see it again

By the way, the button Generate Personal Key is now under the account settings section.

As for the other API key(NVIDIA_API_KEY), it was auto-generated by clicking the button the Get API Key button in step 1 under the following page.

Would like to know what I should do to fix this.

junshengy · April 17, 2026, 8:13am

This issue seems to be caused by incompatibility between the current VLLM image and Jetpack, but I found a workaround for your reference.

Deploy cosmos-reason2-8b using this method. This method is provided by Jetson AI Lab.

2.Deploying VSS using a remote VLM(You can use the local IP address)

export NGC_CLI_API_KEY="cli api key is used for download model and docker images"
export NVIDIA_API_KEY="api key is used for call llm api"
export VLM_ENDPOINT_URL='http://ip:8010'
export LLM_ENDPOINT_URL='https://integrate.api.nvidia.com'

./scripts/dev-profile.sh up -p base \
  --use-remote-llm \
  --llm "nvidia/nvidia-nemotron-nano-9b-v2" \
  --use-remote-vlm

jefferson.chiu · April 17, 2026, 8:15am

(UPDATE)

I didn’t run the command found here first. Not sure if it has something to do with the problem I’m encountering.

I then ran the following command but still couldn’t run it successfully.

export NGC_CLI_API_KEY=(MY CLI API KEY)

export NGC_API_KEY=$NGC_CLI_API_KEY

export LOCAL_NIM_CACHE=~/.cache/nim

mkdir -p “$LOCAL_NIM_CACHE”

chmod -R a+w “$LOCAL_NIM_CACHE”

docker run -it --rm \
--gpus all \\

--ipc host \\

--shm-size=32GB \\

-e NGC_API_KEY \\

-e NIM_MAX_MODEL_LEN=65536 \\

-v "$LOCAL_NIM_CACHE:/opt/nim/.cache" \\

-u $(id -u) \\

-p 8000:8000 \\

nvcr.io/nim/nvidia/cosmos-reason2-8b:latest

Attached is the log message I received and there’s something I noticed:
“torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device”

nim_cosmos_reason2_error_log.txt (30.8 KB)

junshengy · April 17, 2026, 8:19am

The method mentioned here is for x86 + dGPU deployments. For Thor, please refer to the solution above. I have tried it, and it works on my AGX Thor.

jefferson.chiu · April 17, 2026, 8:19am

Hi, thanks for the reply. I just happened to update the situation right after your post.

Is it exactly the error message I received?

torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device

junshengy · April 17, 2026, 8:26am

Since you’re using an x86+dGPU deployment, the errors appear to be different. You can check on Thor using docker logs rtvi-vlm. From my side, the error is as follows:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/nvidia/rtvi/rtvi/server/rtvi_vlm_server.py", line 2366, in <module>
    server = RTVIServer(args)
             ^^^^^^^^^^^^^^^^
  File "/opt/nvidia/rtvi/rtvi/server/rtvi_vlm_server.py", line 216, in __init__
    raise ServiceException(
common.service_exception.ServiceException: ServiceException - code: InternalServerError message: Failed to load RTVI stream handler - Failed to load VLM on GPU 0
NvRmMemInitNvmap failed: error Permission denied
NvRmMemMgrInit failed: Memory Manager Not supported, line 333
NvRmMemMgrInit failed: error type 196626
libnvrm_gpu.so: NvRmGpuLibOpen failed, error=196625

jefferson.chiu · April 17, 2026, 10:18am

I followed the 2 steps and still encountered some problems.

Firstly, I entered the following command:

sudo docker run -it --rm --runtime=nvidia --network host \
-v $MODEL_PATH:/models/cosmos-reason2-8b:ro \
-v ${HOME}/.cache/vllm:/root/.cache/vllm \
-e MAX_JOBS=1
-e FLASHINFER_JIT_THREADS=1 \
Package vllm · GitHub \
vllm serve /models/cosmos-reason2-8b \
–served-model-name nvidia/cosmos-reason2-8b-fp8 \
–max-model-len 8192
–gpu-memory-utilization 0.5 \
–reasoning-parser qwen3 \
–media-io-kwargs ‘{“video”: {“num_frames”: -1}}’ \
–enable-prefix-caching \
–port 8010

Problem 1: OOM

workaround_1st_error_oom.txt (100.4 KB)

I added 2 more parameters to the previous command and it worked, cosuming roughly 32GB out of 59.90GB:

-e MAX_JOBS=1 \

-e FLASHINFER_JIT_THREADS=1 \

Then I ran the dev-profile.sh command as shown below and encountered another error:

export NGC_CLI_API_KEY=(MY CLI API KEY)
export NVIDIA_API_KEY=(MY CATALOG KEY)
export VLM_ENDPOINT_URL=‘http://0.0.0.0:8010’
export LLM_ENDPOINT_URL=‘https://integrate.api.nvidia.com’

./scripts/dev-profile.sh up -p base \
-H AGX-THOR \
–use-remote-llm \
–llm “nvidia/nvidia-nemotron-nano-9b-v2” \
–use-remote-vlm

Problem 2:

[ERROR] --use-remote-vlm is not accepted for base profile with hardware profile AGX-THOR

junshengy · April 17, 2026, 11:19am

Thank you for sharing. I’m using a 128GB AGX Thor, which has more resources, so I didn’t encounter any errors.

Since remote LLM/VLM is used here, and the HW model doesn’t need to be specified。
deploying VLM already consumes almost all of the video memory. I suspect you might encounter other errors.

jefferson.chiu · April 20, 2026, 1:12am

I set up the remote LLM in one terminal and then started another terminal to enter the commands.

If param -H isn’t given, the error message becomes this way:

Also I saw the description as follows:

--use-remote-vlm Use remote VLM; base URL taken from host env VLM_ENDPOINT_URL.
• Optional for profile=search; not accepted for profile=alerts or base on IGX-THOR or AGX-THOR

I tried replacing base with lvs but still didn’t work.

nvidia@tegra-ubuntu:~/video-search-and-summarization$ ./scripts/dev-profile.sh up -p lvs --use-remote-llm --llm “nvidia/nvidia-nemotron-nano-9b-v2” --use-remote-vlm --vlm “nvidia/nvidia/cosmos-reason2-8b-fp8” -H AGX-THOR
[ERROR] Hardware profile ‘AGX-THOR’ is only valid for profile base or alerts, not ‘lvs’

junshengy · April 20, 2026, 2:34am

junshengy:

export NGC_CLI_API_KEY="cli api key is used for download model and docker images"
export NVIDIA_API_KEY="api key is used for call llm api"
export VLM_ENDPOINT_URL='http://ip:8010'
export LLM_ENDPOINT_URL='https://integrate.api.nvidia.com'

./scripts/dev-profile.sh up -p base \
  --use-remote-llm \
  --llm "nvidia/nvidia-nemotron-nano-9b-v2" \
  --use-remote-vlm

Declare the following environment variables

export SKIP_HARDWARE_CHECK=true

jefferson.chiu · April 20, 2026, 2:37am

I checked dev-profile.sh and figured out that SKIP_HARDWARE_CHECK was mentioned.

I added SKIN_HARDWARE_CHECK=‘true’ to the environment, ran dev-profile.sh again and it worked.

export SKIP_HARDWARE_CHECK=‘true’
./scripts/dev-profile.sh up -p base
–use-remote-llm
–llm “nvidia/nvidia-nemotron-nano-9b-v2”
–use-remote-vlm
–vlm “nvidia/cosmos-reason2-8b-fp8”

But still, there were some other issues regarding during reasoning.

jefferson.chiu · April 20, 2026, 2:54am

After successfully running dev-profile.sh I dragged a test video clip to the VSS GUI and asked questions again, but it still ended up not responding until it reached the iteration limit. Below is the message the APIServer kept tossing out.

Step - 19 - Thought

Updated Plan: 1. [x] Call

Copy code
vst_video_clip
— adjusted sensor_id to ‘sample-sim-traffic’ (valid stream found). 2. [ ] Call

Copy code
video_understanding
— adjust timestamps to match video length (19) and retry (e.g., start=0, end=19). — ### Latest Tool Results

Copy code
video_understanding
result: Tool call failed: [###] {‘message’: ‘The timestamps length(15) should be equal video length (19).’, ‘type’: ‘Internal Server Error’, ‘param’: None, ‘code’: 500} {‘error’: {‘message’: ‘The timestamps length(15) should be equal video length (19).’, ‘type’: ‘Internal Server Error’, ‘param’: None, ‘code’: 500}}

Step - 20 - Error

Error: Recursion limit of 50 reached without hitting a stop condition. You can increase the limit by setting the

Copy code
recursion_limit
config key. For troubleshooting, visit: https://docs.langchain.com/oss/python/langgraph/errors/GRAPH_RECURSION_LIMIT

jefferson.chiu · April 20, 2026, 6:59am

After having tried to do local VLM with remote LLM a few times but ended up getting only failure, I tried using both remote VLM and LLM at the same time.

Command:

export VLM_ENDPOINT_URL=‘https://integrate.api.nvidia.com’

export LLM_ENDPOINT_URL=‘https://integrate.api.nvidia.com’

export SKIP_HARDWARE_CHECK=‘true’

./scripts/dev-profile.sh up -p base \

--use-remote-llm \

--llm “nvidia/nvidia-nemotron-nano-9b-v2” \

--use-remote-vlm \

--vlm “nvidia/nvidia-cosmos-reason2-8b”

Making sure that no unhealthy containers were seen

Still not able to work properly. (502 bad gateway)

junshengy · April 20, 2026, 8:43am

It should be nvidia/cosmos-reason2-8b.

But nim currently does not provide API services for cosmos-reason2-8b. You can use the following command line to check. If you write the model as nvidia/nvidia-cosmos-reason2-8b, you will get another error.

MODEL="nvidia/cosmos-reason2-8b"
curl -sS -w "\nHTTP %{http_code}\n" https://integrate.api.nvidia.com/v1/chat/completions   -H "Authorization: Bearer $NVIDIA_API_KEY"   -H "Content-Type: application/json"   -d "{\"model\":\"$MODEL\",\"messages\":[{\"role\":\"user\",\"content\":\"hi\"}]}"

Try the command line below. The command line you provided doesn’t match the parameters such asmodel name and max-model-len, causing inference failure. I successfully inferred on my 128G AGX-thor.

ngc registry model download-version "nim/nvidia/cosmos-reason2-8b:1208-fp8-static-kv8" \
  --dest ~/.cache/huggingface/hub
export MODEL_PATH="${HOME}/.cache/huggingface/hub/cosmos-reason2-8b_v1208-fp8-static-kv8"

sudo sysctl -w vm.drop_caches=3
sudo docker run -it --rm --runtime=nvidia --network host \
  -e MAX_JOBS=1 \
  -e FLASHINFER_JIT_THREADS=1 \
  -e VLLM_USE_V1=1 \
  -v "$MODEL_PATH":/models/cosmos-reason2-8b:ro \
  -v "${HOME}/.cache/vllm":/root/.cache/vllm \
  ghcr.io/nvidia-ai-iot/vllm:0.14.0-r38.3-arm64-sbsa-cu130-24.04 \
  vllm serve /models/cosmos-reason2-8b \
    --served-model-name nvidia/cosmos-reason2-8b \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.5 \
    --reasoning-parser qwen3 \
    --enable-prefix-caching \
    --limit-mm-per-prompt '{"video": 1, "image": 0}' \
    --media-io-kwargs '{"video": {"num_frames": 32}}' \
    --port 8010

export NGC_CLI_API_KEY=xxxx
export NVIDIA_API_KEY=xxxx
export VLM_ENDPOINT_URL='http://ip:8010'
export LLM_ENDPOINT_URL='https://integrate.api.nvidia.com'
export SKIP_HARDWARE_CHECK=true

./scripts/dev-profile.sh up -p base \
  --use-remote-llm \
  --llm "nvidia/nvidia-nemotron-nano-9b-v2" \
  --use-remote-vlm \
  --vlm "nvidia/cosmos-reason2-8b"

jefferson.chiu · April 21, 2026, 3:26am

Thanks, it finally worked for the first time.

Memory consumption is roughly 36 GB out of 60 GB.

However, I’d like to know if it’s possible to do remote LLM on some other devices as for now, The LLM_ENDPOINT_URL is https://integrate.api.nvidia.com. What I have in mind is to modify it like what we have done to the VLM in this workaround.

Commands entered in 2 different terminals:

mkdir -p ~/.cache/vllm

sudo sysctl -w vm.drop_caches=3

sudo docker run -it --rm --runtime=nvidia --network host \

-e MAX_JOBS=1 \

-e FLASHINFER_JIT_THREADS=1 \

-e VLLM_USE_V1=1 \

-v “$MODEL_PATH”:/models/cosmos-reason2-8b:ro \

-v “${HOME}/.cache/vllm”:/root/.cache/vllm \

Package vllm · GitHub \

vllm serve /models/cosmos-reason2-8b \
--served-model-name nvidia/cosmos-reason2-8b \\

--max-model-len 16384 \\

--gpu-memory-utilization 0.5 \\

--reasoning-parser qwen3 \\

--enable-prefix-caching \\

--limit-mm-per-prompt '{"video": 1, "image": 0}' \\

--media-io-kwargs '{"video": {"num_frames": 32}}' \\

--port 8010

export NGC_CLI_API_KEY=(MY CLI API KEY)

export NVIDIA_API_KEY=(MY NIM API KEY)

export VLM_ENDPOINT_URL=‘http://10.1.9.24:8010’

export LLM_ENDPOINT_URL=‘https://integrate.api.nvidia.com’

export SKIP_HARDWARE_CHECK=‘true’

./scripts/dev-profile.sh up -p base \

--use-remote-llm \

--llm “nvidia/nvidia-nemotron-nano-9b-v2” \

--use-remote-vlm \

--vlm “nvidia/cosmos-reason2-8b”

junshengy · April 21, 2026, 6:55am

As a developer, as long as your API key hasn’t expired, the LLM endpoint can continue to be used, although the speed might be slower.

This is guaranteed by NIM, VSS just a user of NIM.

Topic		Replies	Views
Issue Running VSS on Jetson Thor Visual AI Agent nim , blueprints , cosmos	3	236	December 27, 2025
VSS 2.3.0 Docker remote_llm_deployment Failed to generate TRT-LLM engine Visual AI Agent nim , paligemma , kosmos-2 , llama	4	254	May 23, 2025
VSS FAQ Visual AI Agent	11	917	August 3, 2026
VSS blueprint 2.2.0 - processing, percentage complete is 0.00 forever Visual AI Agent	8	282	March 6, 2025
401 unauthorized access Visual AI Agent nim , llama-31-70b-instruct , llama	11	547	April 28, 2025
VSS local deployment single gpu: Failed to load VIA stream handler - Guardrails / CA-RAG setup failed Visual AI Agent nim , llama-31-8b-instruct , llama	3	264	July 4, 2025
VSS Installation Visual AI Agent	14	482	February 14, 2025
Is there any way to use different VLM models on single VSS instance? Visual AI Agent	2	170	October 20, 2025
VSS blueprint 2.2.0 - ERROR Failed to load VIA stream handler - Failed to generate TRT-LLM engine Visual AI Agent nim , llama-31-70b-instruct , llama	16	680	April 22, 2025
Unable to run Nemotron AGX Thor Dev Kit Jetson Thor generative_ai , nemotron	9	332	March 11, 2026

Unable to start VSS reasoning properly

Related topics