Text from a video stream using Live LLaVA

Sesame · April 30, 2024, 12:16am

Hi,

I need help extracting text from video frames. I want to do this in real time.

kayccc · April 30, 2024, 12:50am

Please check LLaMa 2 LLMs w/ NVIDIA Jetson and textgeneration-web-ui - Jetson & Embedded Systems / Jetson Projects - NVIDIA Developer Forums to see if can help.

dusty_nv · May 2, 2024, 2:33pm

Hi @Sesame, perhaps this simplified coding example for Live Llava will be useful for you:

github.com

dusty-nv/NanoLLM/blob/main/nano_llm/vision/example.py

#!/usr/bin/env python3
#
# This multimodal example is a simplified version of the 'Live Llava' demo,
# wherein the same prompt (or set of prompts) is applied to a stream of images.
#
# You can run it like this (these options will replicate the defaults)
#
#    python3 -m nano_llm.vision.example \
#      --model Efficient-Large-Model/VILA-2.7b \
#      --video-input "/data/images/*.jpg" \
#      --prompt "Describe the image." \
#      --prompt "Are there people in the image?"
#
# You can specify multiple prompts (or a text file) to be applied to each image,
# and the video inputs can be sequences of files, camera devices, or network streams.
# For example, `--video-input /dev/video0` will capture from a V4L2 webcam. See here:
#  https://github.com/dusty-nv/jetson-inference/blob/master/docs/aux-streaming.md
#
from nano_llm import NanoLLM, ChatHistory
from nano_llm.utils import ArgParser, load_prompts

This file has been truncated. show original

system · May 16, 2024, 2:33pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
record and run inference at the same time, split video Jetson Nano	26	2936	October 14, 2021
Hello AI World - now supports Python and onboard training with PyTorch Jetson Projects	0	762	July 10, 2019
LLM based Multimodal AI w/ Azure Open AI & NVIDIA Jetson Jetson Projects	0	533	August 22, 2023
Hello AI World - now supports Python and onboard training with PyTorch! Jetson TX2	13	1867	April 30, 2020
Hello AI World - new object detection training and video interfaces Jetson AGX Xavier	2	580	July 16, 2020
Hello AI World - now supports Python and onboard training with PyTorch! Jetson TX1	2	968	April 30, 2020
Hello AI World - new object detection training and video interfaces Jetson Nano	0	344	July 15, 2020
DetectNetV_2: Get inference images with label and confidence along with bounding box embedded on inference images TAO Toolkit extract-transform-load-etl , deep-learning-profiler	2	943	October 12, 2021
how can i use trt-yolo-app to deal with video and camera DeepStream SDK	2	539	September 7, 2022
Generate Text Responses from Visual and Text Inputs with Google's New PaliGemma Model Technical Blog	1	136	May 15, 2024

Text from a video stream using Live LLaVA

Related topics