VILA1.5 13B somtimes will return empty value

[tim_lin1]

I’m using AGX Orin 64G to try the JPS vlm service.
I call the chat completion api to ask the model to describe the image. But sometime it would return the empty value.

I have tried the llava 1.5 13B, and there was not the same issue.
How can I solve this problem?

[kesong]

Are you follow this guide to ask question: Visual Language Models (VLM) with Jetson Platform Services — Metropolis on Jetson documentation 0.1.0 documentation ?
I don’t reproduce the issue in my side. I added one RTSP stream to VLM and ask question to describe the scene with below. Can you share your reproduce steps? Are you using RTSP camera or NVStreamer to simulate RTSP with local video file? What is the possibility to meet the issue during your test?

curl --location ‘http://0.0.0.0:5010/api/v1/chat/completions’
–header ‘Content-Type: application/json’
–data '{
“messages”: [
{
“role”: “system”,
“content”: “You are a helpful AI assistant.”

    },
    {
        "role": "user",
        "content":[
            {
                "type": "stream",
                "stream":
                {
                    "stream_id": "a782e200-eb48-4d17-a1b9-5ac0696217f7"
                }
            },

            {
                "type":"text",
                "text": "Can you describe the scene?"
            }
        ]
    }
],
"min_tokens": 1,
"max_tokens": 128

}
’

[tim_lin1]

@kesong

“messages”: [
{
“role”: “system”,
“content”: “You are a helpful AI assistant.”

    },
    {
        "role": "user",
        "content":[
            {
                "type": "stream",
                "stream":
                {
                    "stream_id": "a782e200-eb48-4d17-a1b9-5ac0696217f7"
                }
            },

            {
                "type":"text",
                "text": "Can you describe the scene?"
            }
        ]
    }
],
"min_tokens": 1,
"max_tokens": 128

}

I think the problem is the order of the content.
The content I used before is:

"content":[
            {
                "type": "stream",
                "stream":
                {
                    "type":"text",
                    "text": "Can you describe the scene?"
                },
                {
                    "stream_id": "a782e200-eb48-4d17-a1b9-5ac0696217f7"
                }
            }
        ]

The issue have solved after I change the order of the content.
Thank you!

[kesong]

Glad to know the issue is resolved.