DeepStream, Python USB cam sample hangs

Accelerated Computing Intelligent Video Analytics DeepStream SDK
1

Please provide complete information as applicable to your setup.

• Hardware Platform (Jetson / GPU)
Jetson Nano Dev Kit 4GB
• DeepStream Version
5.0.1
• JetPack Version (valid for Jetson only)
4.4
• TensorRT Version
• NVIDIA GPU Driver Version (valid for GPU only)
• Issue Type( questions, new requirements, bugs)
• How to reproduce the issue ? (This is for bugs. Including which sample app is using, the configuration files content, the command line used and other details for reproducing)
• Requirement details( This is for new requirement. Including the module name-for which plugin or for which sample application, the function description)

I’m running the deepstream_test_1_usb.py python sample on my Nano. While I had success to playback and infer an MP4 file already, I cannot make the usb cam sample run.

It starts up promising, but then it hangs and there is also no video window as it was with the MP4 file.

Here is a pointer of my Nano’s desktop while it is trying to infer from /dev/video0

image
image1920×1080 811 KB

2

I have limited the image size to 640 x 480. Now at least kind of a window pops up, but the content is not the camera video. It is some frozen background…

Gees…

4

Can you check your camera configuration? The deepstream_test_1_usb.py is just a sample, some configuration need to be modified if it is not compatible to your device.

5

Thanks for the response. What specific “camera configuration” do you mean?

I can only see this relevant line of code deepstream_python_apps/deepstream_test_1_usb.py at 5cb4cb8be92e079acd07d911d265946580ea81cd · NVIDIA-AI-IOT/deepstream_python_apps · GitHub

For sure this camera supports this setup, if you mean this. I have two other cams, which work fine with the code, just not this particular brand (which is not that uncommon, I think. ELP 1080p)

6

Can you use “v4l2-ctl” tool to list your camera supported formats, resolution and framerates?

7

Here we go:

neil@jetson:~$ v4l2-ctl --list-formats
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'MJPG' (compressed)
	Name        : Motion-JPEG

	Index       : 1
	Type        : Video Capture
	Pixel Format: 'YUYV'
	Name        : YUYV 4:2:2



neil@jetson:~$ gst-device-monitor-1.0 Video/Source
Probing devices...


Device found:

	name  : H264 USB Camera
	class : Video/Source
	caps  : video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)5/1;
	        video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)5/1;
	        video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)10/1;
	        video/x-raw, format=(string)YUY2, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-raw, format=(string)YUY2, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-raw, format=(string)YUY2, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-raw, format=(string)YUY2, width=(int)320, height=(int)180, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        image/jpeg, width=(int)320, height=(int)180, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	properties:
		udev-probed = true
		device.bus_path = platform-70090000.xusb-usb-0:2.1:1.0
		sysfs.path = /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/video4linux/video0
		device.bus = usb
		device.subsystem = video4linux
		device.vendor.id = 32e4
		device.vendor.name = "USB\\x20Camera\\x20Manufacture"
		device.product.id = 9422
		device.product.name = "H264\ USB\ Camera"
		device.serial = USB_Camera_Manufacture_H264_USB_Camera_2020052801
		device.capabilities = :capture:
		device.api = v4l2
		device.path = /dev/video0
		v4l2.device.driver = uvcvideo
		v4l2.device.card = "H264\ USB\ Camera"
		v4l2.device.bus_info = usb-70090000.xusb-2.1
		v4l2.device.version = 264588 (0x0004098c)
		v4l2.device.capabilities = 2216689665 (0x84200001)
		v4l2.device.device_caps = 69206017 (0x04200001)
	gst-launch-1.0 v4l2src ! ...


Device found:

	name  : H264 USB Camera
	class : Video/Source
	caps  : video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)640, height=(int)360, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)320, height=(int)240, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-h264, stream-format=(string)byte-stream, alignment=(string)au, width=(int)320, height=(int)180, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	properties:
		udev-probed = true
		device.bus_path = platform-70090000.xusb-usb-0:2.1:1.0
		sysfs.path = /sys/devices/70090000.xusb/usb1/1-2/1-2.1/1-2.1:1.0/video4linux/video1
		device.bus = usb
		device.subsystem = video4linux
		device.vendor.id = 32e4
		device.vendor.name = "USB\\x20Camera\\x20Manufacture"
		device.product.id = 9422
		device.product.name = "H264\ USB\ Camera"
		device.serial = USB_Camera_Manufacture_H264_USB_Camera_2020052801
		device.capabilities = :capture:
		device.api = v4l2
		device.path = /dev/video1
		v4l2.device.driver = uvcvideo
		v4l2.device.card = "H264\ USB\ Camera"
		v4l2.device.bus_info = usb-70090000.xusb-2.1
		v4l2.device.version = 264588 (0x0004098c)
		v4l2.device.capabilities = 2216689665 (0x84200001)
		v4l2.device.device_caps = 69206017 (0x04200001)
	gst-launch-1.0 v4l2src device=/dev/video1 ! ...

neil@jetson:~$
8

Since your camera support several different resolutions with the same 30/1 framerate, so the pipeline should specify the caps.

For example as the following pipeline.
gst-launch-1.0 v4l2src device=“/dev/video0” ! ‘video/x-raw, width=640,height=480,format=YUY2’ ! videoconvert ! nvvideoconvert ! ‘video/x-raw(memory:NVMM), width=640,height=480,format=NV12’ ! m.sink_0 nvstreammux name=m width=640 height=480 batch-size=1 ! fakesink

9

GST claims to see a syntax problem:

 gst-launch-1.0 v4l2src device="/dev/video0" ! ‘video/x-raw, width=640,height=480,format=YUY2’ ! videoconvert ! nvvideoconvert ! ‘video/x-raw(memory:NVMM), width=640,height=480,format=NV12’ ! m.sink_0 nvstreammux name=m width=640 height=480 batch-size=1 ! fakesink
bash: syntax error near unexpected token `('

But I of course also already tried to specify width and height and format to no avail. It simply just hangs after start and doesn’t capture. The window opens, but it just shows a frozen background.

EDIT: To be more specific: Here is my last attempt:

caps_v4l2src.set_property('caps', Gst.Caps.from_string("video/x-raw, framerate=30/1, width=640, height=480, format=YUY2"))

Thanks

10

Please change the quotation marks to the normal one. When you copy it from the web page, the web page changed its format. You can search the gstreamer syntax in internet. Please try with “gst-launch-1.0” tool to find the correct pipeline first and then change your code according to the correct pipeline. We don’t have your camera, so we can not try it for you.
The last comment is to remove the framerate in caps.

11

OK, thanks for being that patient.

Result now

gst-launch-1.0 v4l2src device="/dev/video0" ! "video/x-raw, width=640,height=480,format=YUY2" ! videoconvert ! nvvideoconvert ! "video/x-raw(memory:NVMM), width=640,height=480,format=NV12" ! m.sink_0 nvstreammux name=m width=640 height=480 batch-size=1 ! fakesink

Setting pipeline to PAUSED ...

Pipeline is live and does not need PREROLL ...

Setting pipeline to PLAYING ...

New clock: GstSystemClock

Not sure, if that means it works. I have replaced the last sink by `autovideosink´, but no video window

12

Shortening the pipe and forwarding to xvimagesink after the first videoconvert shows the same hanging video window…

13

Surely, it will never work with ‘autovideosink’. It is a deepstream pipeline, you need to use deepstream plugins. We use ‘nvegltransform’ and ‘nveglglessink’ in sample, please do not change it and use the pileine in the sample.

14

OK, thanks. Sorry for being that dumb :)

15

I’ll blame the camera… No problem

16

I can confirm, after ‘nvegltransform’ and ‘nveglglessink’ the same hang

17

Did run the “autovideosink” pipe with the same camera on my Ubuntu 18.04, GST 1.17.2. Works w/o any problem

18

But I was having so many issues with so many USB cams on the Jetson Nano during the past two weeks, Amazon will surely fire me, since I was purchasing and returning at least 5 different brands of cams during this time… I’m not surprised, that it doesn’t work, but als not very amused :)

19

Anyway, tomorrow comes the 6th attempt. From one sample I have over here I know, it works. So don’t bother. Thanks for your assistance.

20

Maybe you could give me a hint, how to change the combination of these two elements in the sample app, if I would make use of the MJPEG input?

caps_v4l2src.set_property('caps', Gst.Caps.from_string("video/x-raw, framerate=30/1, width=640, height=480, format=YUY2"))
caps_vidconvsrc.set_property('caps', Gst.Caps.from_string("video/x-raw(memory:NVMM)"))

I suppose, videoconvsrc should now be able to cope with JPEG (?)

21

For MJPEG format, the whole pipeline will change, it is no use to just modify the caps.

And MJPEG can containing many different color formats, we only support YUV420 format MJPEG with current hardware video decoder. So if the MJPEG contains other format data, you can just use software video decoder.

A sample pipeline with software decoder is(please set correct parameters for your own camera):

gst-launch-1.0 v4l2src device=/dev/video0 ! ‘image/jpeg,width=1280,height=720,framerate=30/1’ ! jpegdec ! videoconvert ! nvvideoconvert ! ‘video/x-raw(memory:NVMM),fromat=NV12’ ! m.sink_0 nvstreammux name=m batch-size=1 width=640 height=480 ! nvinfer config-file-path=/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary.txt ! fakesink