openCv + detectNet in python

Charly · June 16, 2020, 7:35pm

Hello, I need your help again !
I’m trying to use netdetect-camera.py with IP Camera read by opencv.
I use opencv 3.2.0 like :

camera = cv2.VideoCapture(“rtsp://admin:xxx@192.168.0.13:554//h264Preview_01_sub”)
image = camera.read()[1]
img = cv2.cvtColor(image, cv2.COLOR_BGR2RGBA).astype(np.float32)
img = jetson.utils.cudaFromNumpy(img)
detections = net.Detect(img, image.shape[1], image.shape[0], opt.overlay)

image is a numpy array with shape (480, 640, 3)
so, i convert the color to RGBA and to float32 and put it in CUDA Memory.
But I got a problem when it’s time to run the network.

detections = net.Detect(img, image.shape[1], image.shape[0], opt.overlay)
Exception: jetson.inference – detectNet.Detect() failed to parse args tuple

there is not more informations to explain what is going wrong.

And I’m looking for python documentation of jetson.utils and jetson.inference but I found nothing.

Thank you again for your help !!!

Have a great day

AastaLLL · June 17, 2020, 3:13am

Hi,

You should be able to feed OpenCV image into jetson_inference like this:

The main difference is that there is no overlay argument in net.Detect:

predictions = net.Detect(input_image, input.shape[1], input.shape[0])

Thanks.

Charly · June 17, 2020, 5:32am

Hello, I don’t see the for overlay.
I will try, thank you !

Charly · June 17, 2020, 10:43am

Hello AastaLLL. Your solution work ! Thank you !
but there is another problem…

The performance is worst.
I use ssd-mobilenet-v2.
before detection I resize the image to 300x300.
But the detection take 115ms. ( 8,6 fps)
In nVidia benchmark ssd-mobilenet-v2 300x300 is given to 39fps

To estimate the time I use

start = time.time()
Detections = net.detect(…)
print((time.time()- start) *1000
It is aprox 115ms.

Thank you !

dusty_nv · June 17, 2020, 3:29pm

Hi @Charly,

Use net.PrintProfilerTimes() for a more accurate breakdown of the timing information. net.Detect() also performs image overlay, which you can disable by passing overlay='none' option to net.Detect().

Also, you may need to run sudo nvpmodel -m 0 and sudo jetson_clocks beforehand, since you are only processing one image and the processor clocks don’t have time to spin up. So these commands will set them to the maximum.

The 39FPS benchmark was with the 37-class Oxford PETS dataset, not the 90-class MS COCO model. But you should still see 20-25FPS with MS COCO SSD-Mobilenet-v2 model on Jetson Nano.

Charly · June 17, 2020, 3:49pm

Hello @dusty_nv.
Thank you for your quick reponse.

I will try your proposition.
In facts I got 10 ip cams. I want to read one ilage of each. And detect if there is a perwon in one of the images. So, can I put a batch of images to net.Detect()?

I Just want to detect persons, maybe cars. Which pre-trained model do you recommanded ? Maybe train a custom model ?

I’m desperately looking for documentation for the python jetson library, maybe with a documentation I will reach my goal without help !

Thank you so much. I very much appreciate your help.

dusty_nv · June 17, 2020, 4:06pm

jetson-inference doesn’t do batched processing. If you have 10 IP cameras, I would recommend looking into DeepStream, it is ideal for multi-camera video analytics.

There are some pre-trained models like this from the Transfer Learning Toolkit: https://developer.nvidia.com/transfer-learning-toolkit
They are compatible with DeepStream.

The jetson-inference documentation is from the tutorial, and the Python API is here: Python: package jetson.inference

Charly · June 17, 2020, 7:33pm

Thank you @dusty_nv, the documentation is exactly what I was looking for. Sorry i didn’t found it alone

the performance still far away from 20-25 FPS.

So I will give you some informations :

sudo nvpmodel -q

NVPM WARN: fan mode is not set!
NV Power Mode: MAXN
0

sudo jetson_clocks --show

SOC family:tegra210  Machine:NVIDIA Jetson Nano Developer Kit
Online CPUs: 0-3
CPU Cluster Switching: Disabled
cpu0: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
cpu1: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
cpu2: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
cpu3: Online=1 Governor=schedutil MinFreq=1479000 MaxFreq=1479000 CurrentFreq=1479000 IdleStates: WFI=0 c7=0 
GPU MinFreq=921600000 MaxFreq=921600000 CurrentFreq=921600000
EMC MinFreq=204000000 MaxFreq=1600000000 CurrentFreq=1600000000 FreqOverride=1
Fan: speed=255
NV Power Mode: MAXN

detectnet-camera.py modified

#!/usr/bin/python
import time
import cv2
from imutils.video import VideoStream
import numpy as np
import jetson.inference
import jetson.utils

import argparse
import sys


parser = argparse.ArgumentParser(description="Locate objects in a live camera stream using an object detection DNN.", 
						   formatter_class=argparse.RawTextHelpFormatter, epilog=jetson.inference.detectNet.Usage())

parser.add_argument("--network", type=str, default="ssd-mobilenet-v2", help="pre-trained model to load (see below for options)")
parser.add_argument("--overlay", type=str, default="box,labels,conf", help="detection overlay flags (e.g. --overlay=box,labels,conf)\nvalid combinations are:  'box', 'labels', 'conf', 'none'")
parser.add_argument("--threshold", type=float, default=0.5, help="minimum detection threshold to use") 
parser.add_argument("--camera", type=str, default="0", help="index of the MIPI CSI camera to use (e.g. CSI camera 0)\nor for VL42 cameras, the /dev/video device to use.\nby default, MIPI CSI camera 0 will be used.")
parser.add_argument("--width", type=int, default=1280, help="desired width of camera stream (default is 1280 pixels)")
parser.add_argument("--height", type=int, default=720, help="desired height of camera stream (default is 720 pixels)")

try:
	opt = parser.parse_known_args()[0]
except:
	print("")
	parser.print_help()
	sys.exit(0)

net = jetson.inference.detectNet(opt.network, sys.argv, opt.threshold)

#camera =  VideoStream("rtsp://admin:xx@192.168.0.13:554//h264Preview_01_sub").start()
camera =  VideoStream("rtmp://192.168.0.13/bcs/channel0_sub.bcs?channel=0&stream=1&user=admin&password=xx").start()
display = jetson.utils.glDisplay()
time.sleep(1)

    while True:
    	image = camera.read()
    	image = cv2.resize(image, (300, 300))
    	img = cv2.cvtColor(image, cv2.COLOR_BGR2RGBA).astype(np.float16)
    	img = jetson.utils.cudaFromNumpy(img)
    	detections = net.Detect(img, image.shape[1], image.shape[0], False)

    	print("detected {:d} objects in image".format(len(detections)))

    	for detection in detections:
    		print(detection)

    	display.RenderOnce(img, 300, 300)

    	display.SetTitle("{:s} | Network {:.0f} ms".format(opt.network, net.GetNetworkTime()))

    	net.PrintProfilerTimes()

and the result :

Terminal output

charly@nano:~/Projects$  cd /home/charly/Projects ; env /usr/bin/python3 /home/charly/.vscode-oss/extensions/ms-python.python-2020.6.88468/pythonFiles/lib/python/debugpy/launcher 38117 -- /home/charly/Projects/video-surv/jetson/camera.py 
jetson.inference.__init__.py
jetson.inference -- initializing Python 3.6 bindings...
jetson.inference -- registering module types...
jetson.inference -- done registering module types
jetson.inference -- done Python 3.6 binding initialization
jetson.utils.__init__.py
jetson.utils -- initializing Python 3.6 bindings...
jetson.utils -- registering module functions...
jetson.utils -- done registering module functions
jetson.utils -- registering module types...
jetson.utils -- done registering module types
jetson.utils -- done Python 3.6 binding initialization
jetson.inference -- PyTensorNet_New()
jetson.inference -- PyDetectNet_Init()
jetson.inference -- detectNet loading network using argv command line params
jetson.inference -- detectNet.__init__() argv[0] = '/home/charly/Projects/video-surv/jetson/camera.py'

detectNet -- loading detection network model from:
          -- prototxt     networks/ped-100/deploy.prototxt
          -- model        networks/ped-100/snapshot_iter_70800.caffemodel
          -- input_blob   'data'
          -- output_cvg   'coverage'
          -- output_bbox  'bboxes'
          -- mean_pixel   0.000000
          -- mean_binary  NULL
          -- class_labels networks/ped-100/class_labels.txt
          -- threshold    0.500000
          -- batch_size   1

[TRT]   TensorRT version 7.1.0
[TRT]   loading NVIDIA plugins...
[TRT]   Plugin creator registration succeeded - ::GridAnchor_TRT
[TRT]   Plugin creator registration succeeded - ::NMS_TRT
[TRT]   Plugin creator registration succeeded - ::Reorg_TRT
[TRT]   Plugin creator registration succeeded - ::Region_TRT
[TRT]   Plugin creator registration succeeded - ::Clip_TRT
[TRT]   Plugin creator registration succeeded - ::LReLU_TRT
[TRT]   Plugin creator registration succeeded - ::PriorBox_TRT
[TRT]   Plugin creator registration succeeded - ::Normalize_TRT
[TRT]   Plugin creator registration succeeded - ::RPROI_TRT
[TRT]   Plugin creator registration succeeded - ::BatchedNMS_TRT
[TRT]   Could not register plugin creator:  ::FlattenConcat_TRT
[TRT]   Plugin creator registration succeeded - ::CropAndResize
[TRT]   Plugin creator registration succeeded - ::DetectionLayer_TRT
[TRT]   Plugin creator registration succeeded - ::Proposal
[TRT]   Plugin creator registration succeeded - ::ProposalLayer_TRT
[TRT]   Plugin creator registration succeeded - ::PyramidROIAlign_TRT
[TRT]   Plugin creator registration succeeded - ::ResizeNearest_TRT
[TRT]   Plugin creator registration succeeded - ::Split
[TRT]   Plugin creator registration succeeded - ::SpecialSlice_TRT
[TRT]   Plugin creator registration succeeded - ::InstanceNormalization_TRT
[TRT]   completed loading NVIDIA plugins.
[TRT]   detected model format - caffe  (extension '.caffemodel')
[TRT]   desired precision specified for GPU: FASTEST
[TRT]   requested fasted precision for device GPU without providing valid calibrator, disabling INT8
[TRT]   native precisions detected for GPU:  FP32, FP16
[TRT]   selecting fastest native precision for GPU:  FP16
[TRT]   attempting to open engine cache file /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel.1.1.GPU.FP16.engine
[TRT]   loading network profile from engine cache... /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel.1.1.GPU.FP16.engine
[TRT]   device GPU, /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel loaded
[TRT]   Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
[TRT]   Deserialize required 3905137 microseconds.
[TRT]   device GPU, CUDA engine context initialized with 3 bindings
[TRT]   binding -- index   0
               -- name    'data'
               -- type    FP32
               -- in/out  INPUT
               -- # dims  3
               -- dim #0  3 (SPATIAL)
               -- dim #1  512 (SPATIAL)
               -- dim #2  1024 (SPATIAL)
[TRT]   binding -- index   1
               -- name    'coverage'
               -- type    FP32
               -- in/out  OUTPUT
               -- # dims  3
               -- dim #0  1 (SPATIAL)
               -- dim #1  32 (SPATIAL)
               -- dim #2  64 (SPATIAL)
[TRT]   binding -- index   2
               -- name    'bboxes'
               -- type    FP32
               -- in/out  OUTPUT
               -- # dims  3
               -- dim #0  4 (SPATIAL)
               -- dim #1  32 (SPATIAL)
               -- dim #2  64 (SPATIAL)
[TRT]   binding to input 0 data  binding index:  0
[TRT]   binding to input 0 data  dims (b=1 c=3 h=512 w=1024) size=6291456
[TRT]   binding to output 0 coverage  binding index:  1
[TRT]   binding to output 0 coverage  dims (b=1 c=1 h=32 w=64) size=8192
[TRT]   binding to output 1 bboxes  binding index:  2
[TRT]   binding to output 1 bboxes  dims (b=1 c=4 h=32 w=64) size=32768
device GPU, /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel initialized.
detectNet -- number object classes:   1
detectNet -- maximum bounding boxes:  2048
detectNet -- loaded 1 class info entries
detectNet -- number of object classes:  1
jetson.utils -- PyDisplay_New()
jetson.utils -- PyDisplay_Init()
[OpenGL] glDisplay -- X screen 0 resolution:  1920x1080
[OpenGL] glDisplay -- display device initialized
jetson.utils -- cudaFromNumpy()  ndarray dim 0 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 1 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 2 = 4
detected 0 objects in image
[OpenGL]   creating 300x300 texture
[cuda]   registered 1440000 byte openGL texture for interop access (300x300)

[TRT]   ----------------------------------------------
[TRT]   Timing Report /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel
[TRT]   ----------------------------------------------
[TRT]   Pre-Process   CPU  0.08500ms  CUDA  2.84552ms
[TRT]   Network       CPU 127.37260ms  CUDA 124.45849ms
[TRT]   Post-Process  CPU  0.33917ms  CUDA  0.49391ms
[TRT]   Total         CPU 127.79677ms  CUDA 127.79791ms
[TRT]   ----------------------------------------------

[TRT]   note -- when processing a single image, run 'sudo jetson_clocks' before
                to disable DVFS for more accurate profiling/timing measurements

jetson.utils -- freeing CUDA mapped memory
jetson.utils -- cudaFromNumpy()  ndarray dim 0 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 1 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 2 = 4
detected 0 objects in image

[TRT]   ----------------------------------------------
[TRT]   Timing Report /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel
[TRT]   ----------------------------------------------
[TRT]   Pre-Process   CPU  0.08797ms  CUDA  3.88193ms
[TRT]   Network       CPU 130.04083ms  CUDA 126.09641ms
[TRT]   Post-Process  CPU  0.33975ms  CUDA  0.49729ms
[TRT]   Total         CPU 130.46855ms  CUDA 130.47562ms
[TRT]   ----------------------------------------------

jetson.utils -- freeing CUDA mapped memory
jetson.utils -- cudaFromNumpy()  ndarray dim 0 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 1 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 2 = 4
detected 0 objects in image

[TRT]   ----------------------------------------------
[TRT]   Timing Report /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel
[TRT]   ----------------------------------------------
[TRT]   Pre-Process   CPU  0.07651ms  CUDA  2.88812ms
[TRT]   Network       CPU 130.97610ms  CUDA 128.01839ms
[TRT]   Post-Process  CPU  0.32193ms  CUDA  0.47964ms
[TRT]   Total         CPU 131.37454ms  CUDA 131.38614ms
[TRT]   ----------------------------------------------

jetson.utils -- freeing CUDA mapped memory
jetson.utils -- cudaFromNumpy()  ndarray dim 0 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 1 = 300
jetson.utils -- cudaFromNumpy()  ndarray dim 2 = 4
detected 0 objects in image

[TRT]   ----------------------------------------------
[TRT]   Timing Report /usr/local/bin/networks/ped-100/snapshot_iter_70800.caffemodel
[TRT]   ----------------------------------------------
[TRT]   Pre-Process   CPU  0.08698ms  CUDA  4.26156ms
[TRT]   Network       CPU 132.05255ms  CUDA 127.73360ms
[TRT]   Post-Process  CPU  0.31933ms  CUDA  0.31958ms
[TRT]   Total         CPU 132.45886ms  CUDA 132.31474ms
[TRT]   ----------------------------------------------

One more detail, i’m running the jetson nano on a tv screen. I don’t think this matter
I will try to take a look to DeepStream but, for the moment I want to make it work. To learn from the beginning.
Convert the image input in float16 or 32 didn’t change anything

I want to reach 20-25 FPS to be abble to get analysis of two images par secondes for each camera. To reduce the chances which some one get through without get detected

thank you for your time helping me.

dusty_nv · June 17, 2020, 10:11pm

I see, it is actually loading the pednet model which is slower than SSD-Mobilenet. pednet is based on an older neural network architecture.

Can you try running your script with --model=ssd-mobilenet-v2 option?

When you run it, beneath detectNet -- loading detection network model from: in the log, make sure it is loading ssd-mobilenet-v2. For some reason, it appears your extra command line settings at the beginning (debugpy/launcher, ect) may be confusing my parsing.

Charly · June 18, 2020, 10:25pm

Hello @dusty_nv

You were right.
The default model is ssd-mobilenet-v2.
I see pednet in the logs and I did not realize the wrong model was loaded.

The whole process takes 40ms. 25 FPS.

Thank you for your help. I hope you don’t waste too much time !

I will go further with pre-trained models with Transfert Learning Toolkit for person detection.

Have a good day, and thanks again !

dusty_nv · June 19, 2020, 2:36pm

No problem, glad you got it working!

Topic		Replies	Views
Using 10 lines of code tutorial, feed the frame into opencv Jetson Xavier NX opencv	13	1594	October 18, 2021
Object detection, Jetson-inference, cant start up Jetson Nano jetson-inference	19	1988	December 8, 2021
Jetson Nano - Limiting the results shown by the DetectNet example. Jetson Nano	8	2293	October 14, 2021
resnet10.caffemodel_b8_fp16.engine is optimized for DeepStream SDK	10	1386	October 12, 2021
Hello AI World - new object detection training and video interfaces Jetson Nano	29	4465	April 20, 2021
Opencv Face Detection Poor Performance with jetson nano Jetson Nano opencv	51	14177	October 14, 2021
Hello AI World - now supports Python and onboard training with PyTorch! Jetson Nano	95	7710	July 18, 2022
Imagenet-camera.cpp libraries not working on jetson nano Jetson Nano	40	4470	October 14, 2021
jetson-inference with OpenCV camera input? Jetson TX2 opencv	14	6205	October 18, 2021
Jetson NANO and USB 5.8G UVC Camera Receiver Jetson Nano	12	1491	October 14, 2021

openCv + detectNet in python

Related topics