Output Tensor in FaceDetectIR

Accelerated Computing Intelligent Video Analytics DeepStream SDK
,
1

• Hardware Platform (Jetson Nano)
• DeepStream Version 5.0.1
• JetPack Version 4.4
• Issue Type( questions)

Hi, i’d like to crop detected face after inferencing. I’m using SSD Parser as an example and my problem is that in
pgie_src_pad_buffer_probe
when I call
l_user = frame_meta.frame_user_meta_list
l_user is None
but tracker and nvosd are working fine they detect face perfectly

my config file is

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
tlt-model-key=tlt_encode
tlt-encoded-model=/opt/nvidia/deepstream/deepstream-5.0/samples/models/tlt_pretrained_models/facedetectir/resnet18_facedetectir_pruned.etlt
labelfile-path=labels_facedetectir.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream-5.0/samples/models/tlt_pretrained_models/facedetectir/facedetectir_int8.txt
model-engine-file=/opt/nvidia/deepstream/deepstream-5.0/samples/models/tlt_pretrained_models/facedetectir/resnet18_facedetectir_pruned.etlt_b1_gpu0_fp16.engine
input-dims=3;240;384;0
uff-input-blob-name=input_1
batch-size=1
process-mode=1
model-color-format=0
network-mode=2
num-detected-classes=1
interval=0
gie-unique-id=1
output-blob-names=output_bbox/BiasAdd;output_cov/Sigmoid
[class-attrs-all]
pre-cluster-threshold=0.2
group-threshold=1
eps=0.2

3

Hi,

If the osd works correctly, you can check the object meta from here:

Thanks.

5

Can you specify how to print all object meta?

6

I reinstalled everything and moved project from /opt/nvidia/deepstream/deepstream-5.0/… to home directory and copied …/deepstream_python_apps/apps/common with engine, labels and configs.
I made this because had problems with permissions.
Now it’s working, I have my roi In ObjectMeta.

7

Hi guys, how is it going?
Could you extend on how to try the facedetectIR with deepstream, please?
Is it just a matter of running the command with proper config? which exactly config?
Does it require to generate/ download the faceIR model somehow?
Thanks

8

Hi, first you can download pruned version FacedetectIR
following README at /opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models than you are checking that everything is ok launching deepstream-app -c deepstream_app_source1_facedetectir.txt .

You’ll have to carefully write paths (don’t write full path like /opt/nvidia..this can brake due to bug) to etlt file, labeles etc. After first launch you’ll get your tlt engine resnet18_facedetectir_pruned.etlt_b1_gpu0_fp16.engine check that your path to engine is correct with new engine (it should be fp16 not int8).
Than I installed python API for deepstream and used deepstream-image-multistreamer example. And corrected program for config, which worked with deepstream-app. Main problem was with permissions I moved my project with dependencies to /home, because i had problems with permissions which prevented me from gaining my output data)

Here is my gstreamer pipeline for FacedetectIR

9

@pilotfdd Thank you very much!
It works up to the point of execution of the step below

 deepstream-app -c deepstream_app_source1_facedetectir.txt

However, given the two downloaded files are

resnet18_facedetectir_pruned.etlt facedetectir_int8.txt

that are the result of executing of the readme command below:

mkdir -p ../../models/tlt_pretrained_models/facedetectir && \
    wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_facedetectir/versions/pruned_v1.0/files/resnet18_facedetectir_pruned.etlt \
    -O ../../models/tlt_pretrained_models/facedetectir/resnet18_facedetectir_pruned.etlt && \
    wget https://api.ngc.nvidia.com/v2/models/nvidia/tlt_facedetectir/versions/pruned_v1.0/files/facedetectir_int8.txt \
    -O ../../models/tlt_pretrained_models/facedetectir/facedetectir_int8.txt

why should the engine file named

resnet18_facedetectir_pruned.etlt_b1_gpu0_fp16.engine

be created?
I can see there are only these files:

/opt/nvidia/deepstream/deepstream-5.0/samples/models/tlt_pretrained_models/facedetectir/resnet18_facedetectir_pruned.etlt
/opt/nvidia/deepstream/deepstream-5.0/samples/models/tlt_pretrained_models/facedetectir/resnet18_facedetectir_pruned.etlt_b1_gpu0_int8.engine

trying the other pipeline with

/opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models$ cp ../../streams/sample_1080p_h264.mp4 test.mp4
 gst-launch-1.0 filesrc location= /opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/test.mp4 ! qtdemux ! h264parse ! nvv4l2decoder ! m.sink_0 nvstreammux name=m batch-size=1 width=1280 height=720 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/config_infer_primary_facedetectir.txt batch-size=1 unique-id=1 ! nvtracker ll-lib-file=/opt/nvidia/deepstream/deepstream-5.0/lib/libnvds_mot_klt.so ! nvmultistreamtiler rows=1 columns=1 width=1280 height=720 ! nvvideoconvert ! nvdsosd ! nvoverlaysink

it works

But it doesn’t seem to detect faces on played videos, does it?
the original example also comes redacted already so that the detection doesn’t seem to happen, right?
It seems to boundebox faces if running

deepstream-app -c /opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/deepstream_app_source1_peoplenet.txt

but doesn’t seem to show detection or draw boxes if running

 deepstream-app -c /opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/deepstream_app_source1_facedetectir.txt

Another complication is that it doesn’t seem to support input from vl4 usb camera?

 deepstream-app -c /opt/nvidia/deepstream/deepstream-5.0/samples/configs/tlt_pretrained_models/deepstream_app_source1_peoplenet.txt 

(deepstream-app:26591): GStreamer-CRITICAL **: 00:28:24.050: passed '0' as denominator for `GstFraction'

(deepstream-app:26591): GStreamer-CRITICAL **: 00:28:24.050: passed '0' as denominator for `GstFraction'
** ERROR: <create_camera_source_bin:160>: Failed to link 'src_elem' (image/jpeg; video/mpeg, mpegversion=(int)4, systemstream=(boolean)false; video/mpeg, mpegversion=(int)2; video/mpegts, systemstream=(boolean)true; video/x-bayer, format=(string){ bggr, gbrg, grbg, rggb }, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-dv, systemstream=(boolean)true; video/x-h263, variant=(string)itu; video/x-h264, stream-format=(string){ byte-stream, avc }, alignment=(string)au; video/x-pwc1, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-pwc2, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-raw, format=(string){ RGB16, BGR, RGB, GRAY8, GRAY16_LE, GRAY16_BE, YVU9, YV12, YUY2, YVYU, UYVY, Y42B, Y41B, YUV9, NV12_64Z32, NV24, NV61, NV16, NV21, NV12, I420, BGRA, BGRx, ARGB, xRGB, BGR15, RGB15 }, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-sonix, width=(int)[ 1, 32768 ], height=(int)[ 1, 32768 ], framerate=(fraction)[ 0/1, 2147483647/1 ]; video/x-vp8; video/x-vp9; video/x-wmv, wmvversion=(int)3, format=(string)WVC1) and 'src_cap_filter1' (video/x-raw, width=(int)0, height=(int)0)
** ERROR: <create_camera_source_bin:215>: create_camera_source_bin failed
** ERROR: <create_pipeline:1296>: create_pipeline failed
** ERROR: <main:636>: Failed to create pipeline
Quitting
App run failed

@pilotfdd However, if you could share the code to save /cut faces from video it might be helpful to try / for reference.
Thank you very much

Any ideas on how can we use the FaceDetectIR/ PeopleNet with test4/test5 apps that have mqqt option so that data gets to AWS from the detection via the MQQT ? Also any idea how to get it to work with usb camera? Probably it doesn’t work with camera/ default sample file doe to wrong resolution? Does FaceDecectIR require input files to be necessarily 384 X 240 ? so far I tried with 1080p inputs that did not work.

10

Maybe you are using other device? Which can work with int8 precision, than you don’t have to care about this.

If i understood correctly, yes. There is no detection, because there is no faces on that video, you can upload your video in mp4 format with faces and everything should work nicely.

Check your configuration for camera resolution carefully with v4l2-ctl -d /dev/video1 --list-formats-ext seems like problem with this. I had problems with webcams, because I’m using MJPEG format. But as I remember, when I’ve made fake cam with v4l2loopback, deepstream-app worked correctly with tracking.

def crop_face(image,obj_meta,confidence):
    rect_params=obj_meta.rect_params
    top=int(rect_params.top)
    left=int(rect_params.left)
    width=int(rect_params.width)
    height=int(rect_params.height)
    image=image[top:height+top,left:width+left]
    return image

Didn’t looked in that direction much.

No the resolutin doesn’t affect it, but aspect ratio seems to, because roi different for different aspect ratios. And i don’t know how to deal with that, yet.

12

@pilotfdd I have managed both peoplenet/facedetectIR to take as input rtsp stream that is generated withe with zed camera

 gst-zed-rtsp-launch -a 192.168.x.x zedsrc ! videoconvert ! 'video/x-raw, format=(string)I420' ! omxh265enc ! rtph265pay name=pay0 pt=96  name=pay0
 ZED RTSP Server
-----------------
 * Stream ready at rtsp://192.168.x.x:8554/zed-stream

running the peoplenet will indicate detected face/person,
the same works with facedetectIR after enabligh the sink
but where to put the cut code for execution? it needs to go to some specific file? config file? else? separate .c/ cpp file? that needs to be built? also to load opencv libraries, also read from deepstream outputs? e.g. with gstreamer rtsp opencv wrapper , then execute the code as a separate application? or it could be implemented without running the deepstream app but with the gstreamer pipeline mentined previously, but the crop/save face will still require to be build in a separate file?

Could you extend where/how to apply the facecut code, please?

13

As I know there a several ways to do that.
First and best to write a gstreamer plugin in C/C++, Python, Vala etc. Like in this article.
Second write a program which will launch your pipeline in C/C++, Python. You can take as example this program like i did, rewrite pipeline, change config file, and instead of draw_bounding_boxes use my crop_face function.

14

@pilotfdd could you share exact steps, please?
so it won’t be the deepstream-meta-test or deepstream-app -c “facedetectIR config file”
but rather deepstream_imagedata-multistream.py to execute?
lines will need to be modified only in this file or in multiple? the pipeline you are reffering to is the gstreamer pipeline probably that runs with deepstream without executing binaries probably?
this code does it without deepstream [ saving cropped faces]

import cv2
import os

classifier = cv2.CascadeClassifier(cv2.data.haarcascades+"haarcascade_frontalface_default.xml")

dirFace = 'cropped_face'

# Create if there is no cropped face directory
if not os.path.exists(dirFace):
    os.mkdir(dirFace)
    print("Directory " , dirFace ,  " Created ")
else:    
    print("Directory " , dirFace ,  " has found.")

webcam = cv2.VideoCapture(0) # Camera 0 according to USB port
# video = cv2.VideoCapture(r"use full windows path") # video path

while (True):
    (f, im) = webcam.read() # f returns only True, False according to video access
    # (f, im) = video.read() # video 

    if f != True:
       break

    # im=cv2.flip(im,1,0) #if you would like to give mirror effect

    # detectfaces 
    faces = classifier.detectMultiScale(
        im, # stream 
        scaleFactor=1.10, # change these parameters to improve your video processing performance
        minNeighbors=20, 
        minSize=(30, 30) # min image detection size
        ) 

    # Draw rectangles around each face
    for (x, y, w, h) in faces:

        cv2.rectangle(im, (x, y), (x + w, y + h),(0,0,255),thickness=2)
        # saving faces according to detected coordinates 
        sub_face = im[y:y+h, x:x+w]
        FaceFileName = "cropped_face/face_" + str(y+x) + ".jpg" # folder path and random name image
        cv2.imwrite(FaceFileName, sub_face)

    # Video Window
    cv2.imshow('Video Stream',im)
    key = cv2.waitKey(1) & 0xFF
    # q for exit
    if key == ord('q'): 
        break
webcam.release()

However, it would be great to try saving cropped faces with TensorRT use also

15

I don’t think I should share all the code but the steps are:
Take deepstream_imagedata-multistream.py program
Change pipeline creation as you need (In my case I used MJPEG cams)
Change line 331 as
pgie.set_property('config-file-path', "config_infer_primary_facedetectir.txt")
I’ve met problems with permissions in /opn/nvidia/deepstream/… so i moved all which files

  • facedetectir_int8.txt

  • labels.txt

  • resnet18_facedetectir_pruned.etlt

  • resnet18_facedetectir_pruned.etlt.1

  • resnet18_facedetectir_pruned.etlt_b1_gpu0_fp16.engine

in another directory at home in directory with python script
And i had to copy common directory with this files

  • bus_call.py
  • FPS.py
  • __init__.py
  • is_aarch_64.py
  • __pycache__
  • utils.py

in same directory. Changed sys in deepstream_imagedata-multistream.py according to this changes. And all destinations in config_infer_primary_facedetectir.txt
If program is running then you will rewrite function where you are getting frame buffer in numpy format.
def tiler_sink_pad_buffer_probe(pad,info,u_data):

16

@pilotfdd
Thank you for sharing!
I will try to follow your steps.
However, for me it turned out less complicated to use plain opencv rather than figure out complications of deepstream implementation

xhost +
docker run -it --rm --net=host --runtime nvidia -e DISPLAY=$DISPLAY --privileged --ipc=host -v /tmp/.X11-unix/:/tmp/.X11-unix/ -v /tmp/argus_socket:/tmp/argus_socket -v /dev/video2:/dev/video2 --cap-add SYS_PTRACE iad.ocir.io/idso6d7wodhe/jetson_nx/opencv541 /bin/bash -c ' cp -r /usr/local/opencv-4.5.1-dev/lib/python3.6/dist-packages/cv2 /usr/lib/python3.6/dist-packages/cv2 && export OPENCV_VERSION=opencv-4.5.1-dev && export LD_LIBRARY_PATH=/usr/local/$OPENCV_VERSION/lib && wget https://raw.githubusercontent.com/AndreV84/Jetson/master/usbcam.py && ls && pwd && python3 usbcam.py'

this one just saves cropped face images from the camera [ given usb camera has index /dev/video2.
e.g. it can be used to save to temporary container [ destroyed after launch]
or could save to mount to a folder [ requires adding a folder e.g.
-v /path/path also cd path &&