Jetson Nano CSI Raspberry Pi Camera V2 upside down video when run an example with deepstream-app

Hi guys, I’m trying to run deepstream-app with the example “source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano.txt” on Jetson Nano with “deepstream-4.0_4.0-1_arm64.deb”, using a Raspberry Pi V2 camera NoIr, but the image appears face-up. anybody can help me?

I modified the config file:
[source0]
enable = 1
#Type - 1 = CameraV4L2 2 = URI 3 = MultiURI 4 = RTSP 5 = Camera (CSI) (Jetson only)
type = 5
intra-decode-enable = 1
gpu-id = 0
camera-id = 0
camera-width = 1920
camera-height = 1280
camera-fps-n = 30
camera-fps-d = 1
camera-csi-sensor-id = 0
drop-frame-interval = 0

but, when i run, the image appears upside down. On the other hand, when executing the jetson inference example on the same computer
$ ~/git/jetson-inference/build/aarch64/bin $ ./camera-viewer
the image appears ok and in the pipeline show
“nvvidconv flip-method = 2”. Then the question is:

How can I indicate the orientation of the camera in the configuration file to run deepstream-app?

JetPack Version: 4.2.2

Thanks.

Version JetPack 4.2.2

Hi,
It does not support ‘flip-method’ property in nvvideoconvert. Please modify create_camera_source_bin() to run

nvarguscamerasrc bufapi-version=0 ! nvvidconv flip-method=2 ! video/x-raw,forma=NV12 ! nvvideoconvert ! video/x-raw(memory:NVMM),format=NV12

Hello, I am just learning about Jetson Nano DK, I tried several ways to program as indicated to solve the problem of turning the image, but I could not move forward. Is there any way to see the string generated by the create_camera_source_bin program? That is, there is some method to query / emulate the gstreamer command generated by a program.
The attached contains the source code modified, but, when execute them the mesage is:

<li></li>
./deepstream-app -c ../../../../samples/configs/deepstream-app/source8_1080p_dec_infer-resnet_tracker_tiled_display_fp16_nano-csiCam-test01.txt --gst-debug=3 2>&1 | tee debug.txt 
0:00:00.242025044 10454   0x557ee9c290 WARN                     omx gstomx.c:2826:plugin_init: Failed to load configuration file: Valid key file could not be found in search dirs (searched in: /home/mherrera/.config:/etc/xdg/xdg-unity:/etc/xdg as per GST_OMX_CONFIG_DIR environment variable, the xdg user config directory (or XDG_CONFIG_HOME) and the system config directory (or XDG_CONFIG_DIRS)

(deepstream-app:10454): GLib-GObject-WARNING **: 02:31:43.653: cannot register existing type 'GstInterpolationMethod'

(deepstream-app:10454): GLib-GObject-CRITICAL **: 02:31:43.654: g_param_spec_enum: assertion 'G_TYPE_IS_ENUM (enum_type)' failed

(deepstream-app:10454): GLib-GObject-CRITICAL **: 02:31:43.654: validate_pspec_to_install: assertion 'G_IS_PARAM_SPEC (pspec)' failed
0:00:11.051427092 10454   0x557ef64850 FIXME                default gstutils.c:3981:gst_pad_create_stream_id_internal:<src_elem:src> Creating random stream-id, consider implementing a deterministic way of creating a stream-id
Creating LL OSD context new

Runtime commands:
	h: Print this help
	q: Quit

	p: Pause
	r: Resume

NOTE: To expand a source in the 2D tiled display and view object details, left-click on the source.
      To go back to the tiled display, right-click anywhere on the window.

**PERF: FPS 0 (Avg)	
**PERF: 0.00 (0.00)	
** INFO: <bus_callback:163>: Pipeline ready

** INFO: <bus_callback:149>: Pipeline running

Creating LL OSD context new
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 3264 x 2464 FR = 21.000000 fps Duration = 47619048 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 3264 x 1848 FR = 28.000001 fps Duration = 35714284 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1920 x 1080 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: 1280 x 720 FR = 120.000005 fps Duration = 8333333 ; Analog Gain range min 1.000000, max 10.625000; Exposure Range min 13000, max 683709000;

GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 2 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
   Frame Rate = 29.999999 
GST_ARGUS: PowerService: requested_clock_Hz=627200000
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: nvbuffer Payload Type not supported
NvBufferGetParams failed for src_dmabuf_fd
nvbuffer_transform Failed
gst_nvvconv_transform: NvBufferTransform Failed 
0:00:11.977186035 10454   0x557ef64850 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<src_elem> error: Internal data stream error.
0:00:11.977212963 10454   0x557ef64850 WARN                 basesrc gstbasesrc.c:3055:gst_base_src_loop:<src_elem> error: streaming stopped, reason error (-5)
ERROR from src_elem: Internal data stream error.
Debug info: gstbasesrc.c(3055): gst_base_src_loop (): /GstPipeline:pipeline/GstBin:multi_src_bin/GstBin:src_sub_bin0/GstNvArgusCameraSrc:src_elem:
streaming stopped, reason error (-5)
Quitting
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
App run failed
GST_ARGUS: 
PowerServiceHwVic::cleanupResources
mherrera@JN01EXPL:/opt/nvidia/deepstream/deepstream-4.0/sources/apps/sample_apps/deepstream-app$ 
<li></li>

Thanks
deepstream_source_bin.c (23.7 KB)

Hi,
It should be ‘bufapi-version=FALSE’.

g_object_set (G_OBJECT (bin->src_elem), "bufapi-version", <b>FALSE</b>, NULL);

Hello Dane:
When update the code with

g_object_set (G_OBJECT (bin->src_elem), "bufapi-version", FALSE, NULL);

the message is

ERROR from src_bin_muxer: Input buffer number of surfaces (0) must be equal to mux->num_surfaces_per_frame (1)

For reference, i’m attached the output file and deepstream_source_bin.c.

Obs: I tried different code updates to the deepstream_source_bin.c without success. Finlay, i back to the original code and update only with the previous recommendation and the error message is the same.

Obs2: I restarted with a new installation of the operating system (and Deepstream SDK 4.0.1, that is, the tests I mentioned in this post are with 4.0.1

Regards,

deepstream_source_bin.c (23.8 KB)
20191004-deepstream-app-RBPV2Cam.log (2.79 KB)

Hi,
Please use NvBufSurface APIs. Below is a patch for reference:
sources\apps\apps-common\src\deepstream_source_bin.c:

@@ -22,6 +22,8 @@
 
 #include <string.h>
 
+#include "nvbufsurface.h"
+#include "nvbufsurftransform.h"
 #include "gstnvdsmeta.h"
 #include "deepstream_common.h"
 #include "deepstream_sources.h"
@@ -62,6 +64,64 @@ set_camera_v4l2_params (NvDsSourceConfig * config, NvDsSrcBin * bin)
   return TRUE;
 }
 
+static GstPadProbeReturn
+nvargus_src_pad_buffer_probe (GstPad * pad, GstPadProbeInfo * info,
+    gpointer u_data)
+{
+    GstBuffer *buf = (GstBuffer *) info->data;
+    GstMapInfo outmap = GST_MAP_INFO_INIT;
+    gst_buffer_map (buf, &outmap, GST_MAP_WRITE);
+    NvBufSurface*  surface = (NvBufSurface *)outmap.data;
+
+    NvBufSurfTransformRect src_rect, dst_rect;
+    src_rect.top   = 0;
+    src_rect.left  = 0;
+    src_rect.width = (guint) surface->surfaceList[0].width;
+    src_rect.height= (guint) surface->surfaceList[0].height;
+
+    dst_rect.top   = 0;
+    dst_rect.left  = 0;
+    dst_rect.width = (guint) surface->surfaceList[0].width;
+    dst_rect.height= (guint) surface->surfaceList[0].height;
+
+    NvBufSurface *dst_surface = NULL;
+    NvBufSurfaceCreateParams nvbufsurface_create_params;
+
+    nvbufsurface_create_params.gpuId  = surface->gpuId;
+    nvbufsurface_create_params.width  = (gint) surface->surfaceList[0].width;
+    nvbufsurface_create_params.height = (gint) surface->surfaceList[0].height;
+    nvbufsurface_create_params.size = 0;
+    nvbufsurface_create_params.colorFormat = surface->surfaceList[0].colorFormat;
+    nvbufsurface_create_params.layout = surface->surfaceList[0].layout;
+    nvbufsurface_create_params.memType = surface->memType;
+
+    NvBufSurfaceCreate(&dst_surface,1,&nvbufsurface_create_params);
+
+    NvBufSurfTransformParams nvbufsurface_params;
+    nvbufsurface_params.src_rect = &src_rect;
+    nvbufsurface_params.dst_rect = &dst_rect;
+    nvbufsurface_params.transform_flag =  0;
+    nvbufsurface_params.transform_filter = NvBufSurfTransformInter_Default;
+
+    NvBufSurfTransformConfigParams transform_config_params;
+    NvBufSurfTransform_Error err;
+
+    transform_config_params.compute_mode = NvBufSurfTransformCompute_Default;
+    transform_config_params.gpu_id = surface->gpuId;
+    transform_config_params.cuda_stream = NULL;
+    err = NvBufSurfTransformSetSessionParams (&transform_config_params);
+    // copy to dst_surface
+    err = NvBufSurfTransform (surface, dst_surface, &nvbufsurface_params);
+    // rototate 180 degree to original surface
+    nvbufsurface_params.transform_flag =  NVBUFSURF_TRANSFORM_FLIP;
+    nvbufsurface_params.transform_flip = NvBufSurfTransform_Rotate180;
+    err = NvBufSurfTransform (dst_surface, surface, &nvbufsurface_params);
+    NvBufSurfaceDestroy(dst_surface);
+
+    gst_buffer_unmap (buf, &outmap);
+    return GST_PAD_PROBE_OK;
+}
+
 static gboolean
 create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)
 {
@@ -144,12 +204,16 @@ create_camera_source_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)
     NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter, "src");
 
   } else {
+    GstPad *src_pad;
 
     g_object_set (G_OBJECT (bin->cap_filter), "caps", caps, NULL);
 
     gst_bin_add_many (GST_BIN (bin->bin), bin->src_elem, bin->cap_filter, NULL);
 
     NVGSTDS_LINK_ELEMENT (bin->src_elem, bin->cap_filter);
+    src_pad = gst_element_get_static_pad (bin->src_elem, "src");
+    gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER,
+        nvargus_src_pad_buffer_probe, NULL, NULL);
 
     NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter, "src");
   }

sources\apps\sample_apps\deepstream-app\Makefile

@@ -24,6 +24,7 @@ APP:= deepstream-app
 
 TARGET_DEVICE = $(shell gcc -dumpmachine | cut -f1 -d -)
 
+CUDA_VER:=10.0
 NVDS_VERSION:=4.0
 
 LIB_INSTALL_DIR?=/opt/nvidia/deepstream/deepstream-$(NVDS_VERSION)/lib/
@@ -41,10 +42,10 @@ PKGS:= gstreamer-1.0 gstreamer-video-1.0 x11
 
 OBJS:= $(SRCS:.c=.o)
 
-CFLAGS+= -I../../apps-common/includes -I../../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4
+CFLAGS+= -I../../apps-common/includes -I../../../includes -DDS_VERSION_MINOR=0 -DDS_VERSION_MAJOR=4 -I /usr/local/cuda-$(CUDA_VER)/include
 
 LIBS+= -L$(LIB_INSTALL_DIR) -lnvdsgst_meta -lnvds_meta -lnvdsgst_helper -lnvds_utils -lm \
-       -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,$(LIB_INSTALL_DIR)
+       -lgstrtspserver-1.0 -lgstrtp-1.0 -Wl,-rpath,$(LIB_INSTALL_DIR) -lnvbufsurface -lnvbufsurftransform
 
 CFLAGS+= `pkg-config --cflags $(PKGS)`
3 Likes

hi @DaneLLL
I am trying achieve same thing, i.e. rotate video stream 180 degrees, for an RTSP source. It works, but only partially. Some frames of the RTSP are rotated, some are not, and it seems random (see attached output video (882.6 KB) ).

My setup:
Jetson Xavier NX
Jetpack 4.4 DP
Deepstream 5.0.0 DP

I’ve applied the suggested patch to sources\apps\apps-common\src\deepstream_source_bin.c:

} else{
NVGSTDS_LINK_ELEMENT (bin->cap_filter, bin->nvvidconv);
NVGSTDS_LINK_ELEMENT (bin->nvvidconv, bin->cap_filter1);

/// added here, line 888
GstPad *src_pad;
src_pad = gst_element_get_static_pad (bin->nvvidconv, "src");
gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER,
    nvargus_src_pad_buffer_probe, NULL, NULL);
///

NVGSTDS_BIN_ADD_GHOST_PAD (bin->bin, bin->cap_filter1, "src");

The rest follows exactly like yours.

This is the deepstream config file:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1
#gie-kitti-output-dir=streamscl

[tiled-display]
enable=0
rows=4
columns=2
width=1280
height=720
gpu-id=0
nvbuf-memory-type=0

[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=rtsp://192.168.2.216:8554/test
#num-sources=8
#drop-frame-interval=2
latency=200
gpu-id=0
# (0): memtype_device   - Memory type Device
# (1): memtype_pinned   - Memory type Host Pinned
# (2): memtype_unified  - Memory type Unified
cudadec-memtype=0

[sink0]
enable=1
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265
codec=1
#encoder type 0=Hardware 1=Software
enc-type=0
sync=0
#iframeinterval=10
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=testf.mp4
source-id=0

[streammux]
gpu-id=0
##Boolean property to inform muxer that sources are live
live-source=1
batch-size=8
##time out in usec, to wait after the first buffer is available
##to push the batch even if the complete batch is not formed
batched-push-timeout=40000
## Set muxer output width and height
width=1920
height=1080
##Enable to maintain aspect ratio wrt source, and allow black borders, works
##along with width, height properties
enable-padding=0
nvbuf-memory-type=0
## If set to TRUE, system timestamp will be attached as ntp timestamp
## If set to FALSE, ntp timestamp from rtspsrc, if available, will be attached
## attach-sys-ts-as-ntp=1

Any help is appreciated!

Hi,
Looks like you add prob function to source pad of nvvideoconvert, not nvarguscamerasrc. Please make sure you apply this:

+    src_pad = gst_element_get_static_pad (bin->src_elem, "src");
+    gst_pad_add_probe (src_pad, GST_PAD_PROBE_TYPE_BUFFER,
+        nvargus_src_pad_buffer_probe, NULL, NULL);

Hi @DaneLLL
Thanks for your quick response.

I suspect my previous speculation, that some frames are rotated and some are not, was flawed. So, I further tested it with

nvbufsurface_params.transform_flip = NvBufSurfTransform_Rotate90;

and indeed, my suspicion is proven true. The rotation does work, but it seems like the same frame is being rotated a few times and pushed downstream. So instead of having a 90 degree rotated stream, I have some frames rotated 90, 180, 270, 360 degrees, in that particular order. See this output video (1.0 MB) .

Please note that the above video happens when I try to patch sources\apps\apps-common\src\deepstream_source_bin.c in either:

src_pad = gst_element_get_static_pad (bin->cap_filter, "src");

or:

src_pad = gst_element_get_static_pad (bin->nvvidconv, "src");

or:

src_pad = gst_element_get_static_pad (bin->cap_filter1, "src");

.
.
I did try to prob to source pad of src_elem

src_pad = gst_element_get_static_pad (bin->src_elem, "src");

but the video is not rotated at all. Kindly note that this is for RTSP input, if it helps. Meaning, the patch is within this function:

static gboolean
create_rtsp_src_bin (NvDsSourceConfig * config, NvDsSrcBin * bin)
{
   ....
}

A bit clueless on how to proceed at the moment…

Hi,
For clearness, please make a new post with a patch so that we can try to reproduce it by running deepstream-app with rtspsrc. This topic is about nvarguscamerasrc and the patch is well verified.

Initially thinking the patch could be carried over for RTSP, apparently a new post is needed. Kindly follow up in this new post.

Does this solution involv modifying the existing sources\apps\apps-common\src\deepstream_source_bin.c and then modifying sources\apps\sample_apps\deepstream-app\Makefile and then running make again?

@DaneLLL

Hi,
Yes, after applying the patch, you would need to make again to rebuild deepstream-app.