Getting Error while running blueprint-vss demo

[kelvin.lwin]

It runs on 8xL40S since this 3rd configuration was added recently.
That instance is in launchpad so strangely ‘sudo microk8s’ has a lot of issues but running straight with helm and kubectl worked fine.

Also on the 4xH100, when we took out the LLM everything is “running”. Just need to get authorization figured out between local and llama-3.1-70b-instruct Model by Meta | NVIDIA NIM via https://integrate.api.nvidia.com/v1 I believe. Seeing two kinds of errors of connection refused and unauthorized in different trials.

[yuweiw]

Because the 8xL40S can run normally, it can be confirmed that the problem before is caused by the insufficient resources.

@kelvin.lwin

Also on the 4xH100, when we took out the LLM everything is “running”.

The LLM occupies about half of all the resources, so that everything is OK after taking out that. Could you try to configure-the-nims to setup some keys for the model to verify the connection refused and unauthorized issue?

[kelvin.lwin]

But I’m using the same exact NIM nvcr.io/nim/meta/llama-3.1-70b-instruct so not clear to me what I should change according to Configure the NIMs — Video Search and Summarization Agent?

I tried to set NVIDIA_API_KEY but maybe I need to set OPENAI_API_KEY also since it’s using openai-compat.

[aryason]

@kelvin.lwin When you are trying to run on 4xH100, what NIMs are allocated to which GPUS?

[yuweiw]

Hi @kelvin.lwin , could you try to use the configure file and command below to deploy the VSS on 4xH100?
override.yaml (1.4 KB)
Command:

sudo microk8s helm upgrade --install vss-blueprint nvidia-blueprint-vss-2.1.0.tgz --set global.ngcImagePullSecretName=ngc-docker-reg-secret -f override.yaml

[kelvin.lwin]

Yes, @yuweiw this is working now with forced allocation. Thanks for the help!

@aryason I assume you wanted to know to provide override.yaml. But here’s the the current allocation

[yuweiw]

Glad to hear that. If you have any other issues, feel free to file a new topic.

[basha.ghouse]

Thanks for the help :)

[yuweiw]

We have not deployed that on the 8xA100(40G) GPUs at this time. But you can try to deploy on your side and file a new topic when you encounter problems.
Please refer to the note information below.
When running on an L40 / L40S system, the default startup probe timeout might…

[yuweiw]

Sure. Because the forum you selected is not correct, we did not track it in time. You can choose the visual-ai-agent forum to file the VSS related topic later.