OpenCV SIFT feature IP camera h264

Hi.
I’ve tried to compile some example on github with SIFT feature. I was using handy with some ip webcam application to send h264 stream via gstreamer. I’m dealing with latency and low refresh. I know Jetson isn’t a powerful computer but is it possible to make approx function with reasonable latency ?

take a look at results : https://youtu.be/R7eXtmXLK2E

import cv2
import numpy as np
import gi
gi.require_version('Gst', '1.0')

def sift_detector(new_image, image_template):
    # Function that compares input image to template
    # It then returns the number of SIFT matches between them
    image1 = cv2.cvtColor(new_image, cv2.COLOR_BGR2GRAY)
    image2 = image_template

    # Create SIFT detector object
    #sift = cv2.SIFT()
    sift = cv2.xfeatures2d.SIFT_create()
    # Obtain the keypoints and descriptors using SIFT
    keypoints_1, descriptors_1 = sift.detectAndCompute(image1, None)
    keypoints_2, descriptors_2 = sift.detectAndCompute(image2, None)

    # Define parameters for our Flann Matcher
    FLANN_INDEX_KDTREE = 0
    index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 3)
    search_params = dict(checks = 100)

    # Create the Flann Matcher object
    flann = cv2.FlannBasedMatcher(index_params, search_params)

    # Obtain matches using K-Nearest Neighbor Method
    # the result 'matchs' is the number of similar matches found in both images
    matches = flann.knnMatch(descriptors_1, descriptors_2, k=2)

    # Store good matches using Lowe's ratio test
    good_matches = []
    for m,n in matches:
        if m.distance < 0.7 * n.distance:
            good_matches.append(m) 
    return len(good_matches)


gst = "rtspsrc location=rtsp://192.168.0.12:8080/video/h264 latency=0 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"
cap = cv2.VideoCapture(gst)
# Load our image template, this is our reference image
image_template = cv2.imread('nivea.jpg', 0) 

while True:
    # Get webcam images
    ret, frame = cap.read()

    # Get height and width of webcam frame
    height, width = frame.shape[:2]

    # Define ROI Box Dimensions
    top_left_x = int (width / 3)
    top_left_y = int ((height / 2) + (height / 4))
    bottom_right_x = int ((width / 3) * 2)
    bottom_right_y = int ((height / 2) - (height / 4))

    # Draw rectangular window for our region of interest   
    cv2.rectangle(frame, (top_left_x,top_left_y), (bottom_right_x,bottom_right_y), 255, 3)

    # Crop window of observation we defined above
    cropped = frame[bottom_right_y:top_left_y , top_left_x:bottom_right_x]

    # Flip frame orientation horizontally
    frame = cv2.flip(frame,1)

    # Get number of SIFT matches
    matches = sift_detector(cropped, image_template)

    # Display status string showing the current no. of matches 
    cv2.putText(frame,str(matches),(450,450), cv2.FONT_HERSHEY_COMPLEX, 2,(0,255,0),1)

    # Our threshold to indicate object deteciton
    # We use 10 since the SIFT detector returns little false positves
    threshold = 10

    # If matches exceed our threshold then object has been detected
    if matches > threshold:
        cv2.rectangle(frame, (top_left_x,top_left_y), (bottom_right_x,bottom_right_y), (0,255,0), 3)
        cv2.putText(frame,'Object Found',(50,50), cv2.FONT_HERSHEY_COMPLEX, 2 ,(0,255,0), 2)
    cv2.imshow('Object Detector using SIFT', frame)
    if cv2.waitKey(1) == 13: #13 is the Enter Key
        break

cap.release()
cv2.destroyAllWindows()

Hi,

gst = "rtspsrc location=rtsp://192.168.0.12:8080/video/h264 latency=0 ! queue ! rtph264depay ! queue ! h264parse ! omxh264dec ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink

The pipeline looks optimal for hooking gstreamer and OpenCV. We don’t support BGR format in hardware engine so inevitably have to use software converter. One more thing you can try is to run ‘sudo jetson_clocks’ to get max performance.

There is similar discussion in
https://devtalk.nvidia.com/default/topic/1064944/

Hi.
Why it was working? , gstreamer failed. See on movie. Today I tried to do it without success.
Is it possible that gstreamer fails but receive and decoding ?
How can I do similar function using sdk ?

Hi,
Please try with uridecodebin in gst-launch-1.0:

$ gst-launch-1.0 uridecodebin uri=rtsp://192.168.0.12:8080/video/h264 ! nvoverlaysink

If it is working, please try

$ gst-launch-1.0 uridecodebin uri=rtsp://192.168.0.12:8080/video/h264 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! fakesink

If this is working, the following pipeline should work:

gst= "uridecodebin uri=rtsp://192.168.0.12:8080/video/h264 ! nvvidconv ! video/x-raw,format=BGRx ! videoconvert ! video/x-raw,format=BGR ! appsink"