28. [DSx_All_App] How to connect a USB camera in DeepStream?
28.1 Query the device number by the v4l2-ctl tool
E.g.
$ sudo apt install v4l-utils && v4l2-ctl --list-devices
28.2 Query the supported formats and capabilities of the camera by v4l2-ctl tool.
E.g. Query the camera’s formats and capabilities whose device number is 2.
$ v4l2-ctl -d /dev/video2 --list-formats-ext
The information may be displayed in the following format
ioctl: VIDIOC_ENUM_FMT
Type: Video Capture
octl: VIDIOC_ENUM_FMT
Type: Video Capture
[0]: 'YUYV' (YUYV 4:2:2)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
......
[1]: 'H264' (H.264, compressed)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
......
[2]: 'MJPG' (Motion-JPEG, compressed)
Size: Discrete 640x480
Interval: Discrete 0.033s (30.000 fps)
Interval: Discrete 0.042s (24.000 fps)
Interval: Discrete 0.050s (20.000 fps)
Interval: Discrete 0.067s (15.000 fps)
Interval: Discrete 0.100s (10.000 fps)
Interval: Discrete 0.133s (7.500 fps)
Interval: Discrete 0.200s (5.000 fps)
......
28.3 Choose one format and capabilities from the step 2 query results.
The capsfilter after v4l2src is necessary if your camera can output different formats videos or different resolution/framerate videos with the same format. set the corresponding capsfilter properties to let the camera output the chosen format and capabilities. use gst-launch to construct a working pipeline to test the function.
E.g. The following pipeline lets the camera output video at resolution 640x480, format YUY2 , fps30.
$ gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! mux.sink_0 nvstreammux name=mux width=1280 height=720 batch-size=1 ! fakesink
28.3.1 If the camera can output compressed video formats such as “MJPG”, “H264”,…etc, and you choose to use the compressed format in the DeepStream(GStreamer) pipeline. You need to add the corresponding video decoder after v4l2src.
E.g. The following pipeline lets the camera output video at resolution 640x480, format jpeg, fps30.
For Jetson
$ gst-launch-1.0 v4l2src device=/dev/video2 ! 'image/jpeg, width=640, height=480, framerate=30/1' ! nvv4l2decoder mjpeg=true ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! mux.sink_0 nvstreammux name=mux width=1280 height=720 batch-size=1 ! fakesink
For dGPU
$ gst-launch-1.0 v4l2src device=/dev/video2 ! 'image/jpeg, width=640, height=480, framerate=30/1' ! nvv4l2decoder ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! mux.sink_0 nvstreammux name=mux width=1280 height=720 batch-size=1 ! fakesink
E.g. The following pipeline lets the camera output video at resolution 640x480, format h264, fps30.
$ gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-h264, format=avc, width=640, height=480, framerate=30/1' ! nvv4l2decoder ! fakesink
28.3.2 If all formats and capabilities are not supported by nvvideoconvert, there will be a “linking failed” error. you can add a videoconvert before nvvideoconvert to convert the raw data to the format nvvideoconvert can support.
E.g. The following pipeline lets the camera output video at resolution 640x480, format YUY2 , fps30. videoconvert is used to convert YUY2 to NV12 which is supported by nvvideoconvert.
gst-launch-1.0 v4l2src device=/dev/video2 ! 'video/x-raw, format=YUY2, width=640, height=480, framerate=30/1' ! videoconvert ! 'video/x-raw, format=NV12' ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! fakesink
28.3.3 using uridecodebin to connect the camera.
uridecodebin is a Gstreamer bin including v4l2src and decoder. Capsfilter can’t be added manually after v4l2src, so the negotiated format and capabilities are unpredictable. we only recommend using v4l2src to connect USB cameras.
E.g. The following pipeline uses uridecodebin to connect the camera, the camera 's output format and capabilities are unknown.
gst-launch-1.0 uridecodebin uri=v4l2:///dev/video2 ! nvvideoconvert ! 'video/x-raw(memory:NVMM),format=NV12' ! mux.sink_0 nvstreammux name=mux width=1280 height=720 batch-size=1 ! fakesink