Problem to using with hardware jpeg lib from nvidia via opencv.

  1. I compiled OpenCV 3.2.0 with hardware jpegencode from nvidia.
--     ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
--     JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
--     WEBP:                        /usr/lib/aarch64-linux-gnu/libwebp.so (ver encoder: 0x0202)
--     PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
--     TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 - 4.0.6)
--     JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)

After installed the lib. I could imread(“test.jpg”) with opencv and imshow() the image in windows.
For jpeg file read and show with opencv works fine.

while(1) {
        Mat frame = imread("/home/ubuntu/tegra_multimedia_api/samples/13_camera_v4l2_jpeg/output0.jpg");

        printf("%d %d", frame.cols, frame.rows);
        if (frame.data)
            printf("good");

        imshow("test", frame);
        waitKey(30);
    }

But I can not open the camera with VideoCapture(0) to preivew.

{
    VideoCapture capture;
    Mat image;
    capture.open(0);
    if(capture.isOpened())
    {
        cout << "Capture is opened" << endl;
        for(;;)
        {
            capture >> image;
            cout << "====%d\n" << image.cols<< endl;
            if(image.empty())
                break;
        //  drawText(image);
            imshow("Sample", image);
            if(waitKey(10) >= 0)
                break;
        }
    }
    else
    {
        cout << "No capture" << endl;
        image = Mat::zeros(480, 640, CV_8UC1);
        //drawText(image);
        imshow("Sample", image);
        waitKey(0);
    }
    return 0;
}

The log from console.

[ 4095.166564]  pixelformat : YUV UYVY 2X8 4:2:2 tegra_core_get_format_by_fourcc, drivers/media/platform/tegra/camera/core.c, 381
[ 4095.178050]  pixelformat :YUV UYVY 2X8 4:2:2 code:0x00002006 __tegra_channel_set_format, drivers/media/platform/tegra/camera/channel.c, 1397
[ 4095.190752] ov5640 30-003c: camera_common_try_fmt: size 1920 x 1080
[ 4095.197021] ov5640 30-003c: camera_common_try_fmt: w: 1920, h:1080, maxframerate: 30, s_data->mode: 0, code: 0x00002006
[ 4095.208548] ov5640 30-003c: camera_common_s_fmt(0x00002006) size 1920 x 1080, colorfmt:1498831189
[ 4095.229817]  vb2_streamon, drivers/media/v4l2-core/videobuf2-core.c, 1708
[ 4095.236810]  ===============   bypass  : 0 ===============================================
[ 4095.236810]
[ 4095.246874]  port_num: 0 drivers/media/platform/tegra/csi/csi.c, tegra_csi_start_streaming, 253
[ 4095.255826] vi vi: csi2_write:port 0 offset 0x00000218 val:0x00000000
[ 4095.262765] vi vi: csi2_read:port 0 offset 0x000000d0
[ 4095.267910] vi vi: csi2_write:port 0 offset 0x000000d0 val:0x00000001
[ 4095.290597] (NULL device *): camera_common_dpd_disable: csi 0
[ 4095.296500] ov5640 30-003c: ov5640_s_stream++
[ 4095.390247] ov5640 30-003c: ov5640_write_table: ret = 0
[ 4095.396231] ov5640 30-003c: ov5640_set_gain: gain 0100 val: 0010
[ 4095.420501] ov5640 30-003c: ov5640_write_table: ret = 0
[ 4095.425730] ov5640 30-003c: ov5640_s_stream--
[ 4095.490532] smmu_dump_pagetable(): fault_address=0x0000000080352000 pa=0x0000000000000000 bytes=1000 #pte=533 in L2
[ 4095.501004] mc-err: (18) csw_viw: EMEM decode error on PDE or PTE entry
[ 4095.507628] mc-err:   status = 0x60010072; addr = 0x80352000
[ 4095.513333] mc-err:   secure: no, access-type: write, SMMU fault: nr-nw-s
[ 4095.523864] smmu_dump_pagetable(): fault_address=0x0000000080552000 pa=0x0000000000000000 bytes=1000 #pte=676 in L2
[ 4095.534337] mc-err: (18) csw_viw: EMEM decode error on PDE or PTE entry
[ 4095.540987] mc-err:   status = 0x60010072; addr = 0x80552000
[ 4095.546648] mc-err:   secure: no, access-type: write, SMMU fault: nr-nw-s
[ 4095.557195] smmu_dump_pagetable(): fault_address=0x0000000080752000 pa=0x0000000000000000 bytes=1000 #pte=676 in L2
[ 4095.567676] mc-err: (18) csw_viw: EMEM decode error on PDE or PTE entry
[ 4095.574339] mc-err:   status = 0x60010072; addr = 0x80752000
[ 4095.578812] pgd = ffffffc081768000
[ 4095.578816] [0078fa42] *pgd=0000000125d6b003, *pmd=00000001361f1003, *pte=0000000000000000
[ 4095.578866] pgd = ffffffc081768000
[ 4095.578872] [00775002] *pgd=0000000125d6b003, *pmd=00000001361f1003
[ 4095.578905] Library at 0x7f87befcb4: 0x7f87a67000 /usr/local/lib/libopencv_imgproc.so.3.2.0
[ 4095.578909] Library at 0x7f88b46710: 0x7f88aee000 /usr/local/lib/libopencv_core.so.3.2.0
[ 4095.578911] vdso base = 0x7f88f78000
[ 4095.578944] Library at 0x7f87befcb4: 0x7f87a67000 /usr/local/lib/libopencv_imgproc.so.3.2.0
[ 4095.578947] Library at 0x7f88b46710: 0x7f88aee000 /usr/local/lib/libopencv_core.so.3.2.0
[ 4095.578948] vdso base = 0x7f88f78000
[ 4095.641358] mc-err:   secure: no, access-type: write, SMMU fault: nr-nw-s
[ 4095.657202] smmu_dump_pagetable(): fault_address=0x0000000080552000 pa=0x0000000000000000 bytes=1000 #pte=676 in L2
[ 4095.667728] mc-err: (18) csw_viw: EMEM decode error on PDE or PTE entry
[ 4095.674416] mc-err:   status = 0x60010072; addr = 0x80552000
[ 4095.680077] mc-err:   secure: no, access-type: write, SMMU fault: nr-nw-s
[ 4095.695074]  port_num: 0 drivers/media/platform/tegra/csi/csi.c, tegra_csi_stop_streaming, 282
[ 4095.709626] (NULL device *): camera_common_dpd_enable: csi 0
[ 4095.715339] ov5640 30-003c: ov5640_s_stream++
[ 4095.737626] ov5640 30-003c: ov5640_write_table: ret = 0
[ 4095.744485] (NULL device *): camera_common_dpd_enable: csi 0
[ 4095.750158] ov5640 30-003c: ov5640_power_off: power off
[ 4095.777994] ov5640 30-003c: camera_common_mclk_disable: disable MCLK

The error log from shell termainal.

ubuntu@tegra-ubuntu:~/tegra_multimedia_api/samples/13_camera_v4l2_jpeg/build/build$ ./camera
Capture is opened
Segmentation fault
ubuntu@tegra-ubuntu:~/tegra_multimedia_api/samples/13_camera_v4l2_jpeg/build/build$

I could preview with

gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=300 ! 'video/x-raw,width=1920,height=1080,format=(string)UYVY' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=(string)NV12' ! nvoverlaysink

Also I could get a jpeg with

gst-launch-1.0 v4l2src device="dev/video0" num-buffers=1 ! "video/x-raw, width=1920, height=1080, format=I420" ! nvjpegenc ! filesink location=test.jpg -e

The whole opencv configure.

-- General configuration for OpenCV 3.2.0-dev =====================================
--   Version control:               3.2.0-1-g7dd3723-dirty
--
--   Extra modules:
--     Location (extra):            /media/ubuntu/91df5a31-05fd-4885-b29b-d7e625107ac5/opencv/opencv_contrib/modules
--     Version control (extra):     3.2.0
--
--   Platform:
--     Timestamp:                   2017-01-06T03:38:25Z
--     Host:                        Linux 3.10.96-tegra aarch64
--     CMake:                       3.5.1
--     CMake generator:             Unix Makefiles
--     CMake build tool:            /usr/bin/make
--     Configuration:               RELEASE
--
--   C/C++:
--     Built as dynamic libs?:      YES
--     C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
--     C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
--     C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
--     C Compiler:                  /usr/bin/cc
--     C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
--     C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
--     Linker flags (Release):
--     Linker flags (Debug):
--     ccache:                      NO
--     Precompiled headers:         YES
--     Extra dependencies:          /usr/lib/aarch64-linux-gnu/libwebp.so /usr/lib/aarch64-linux-gnu/libjasper.so gtk-x11-2.0 gdk-x11-2.0 pangocairo-1.0 atk-1.0 cairo gdk_pixbuf-2.0 gio-2.0 pangoft2-1.0 pango-1.0 gobject-2.0 glib-2.0 fontconfig freetype gthread-2.0 gtkglext-x11-1.0 gdkglext-x11-1.0 GLU GL Xmu Xt SM ICE pangox-1.0 X11 gmodule-2.0 gstbase-1.0 gstreamer-1.0 gstvideo-1.0 gstapp-1.0 gstriff-1.0 gstpbutils-1.0 dc1394 avcodec-ffmpeg avformat-ffmpeg avutil-ffmpeg swscale-ffmpeg /usr/lib/aarch64-linux-gnu/hdf5/serial/lib/libhdf5.so /usr/lib/aarch64-linux-gnu/libpthread.so /usr/lib/aarch64-linux-gnu/libsz.so /usr/lib/aarch64-linux-gnu/libdl.so /usr/lib/aarch64-linux-gnu/libm.so vtkRenderingOpenGL vtkImagingHybrid vtkIOImage vtkCommonDataModel vtkCommonMath vtkCommonCore vtksys vtkCommonMisc vtkCommonSystem vtkCommonTransforms vtkCommonExecutionModel vtkDICOMParser vtkIOCore /usr/lib/aarch64-linux-gnu/libz.so vtkmetaio /usr/lib/aarch64-linux-gnu/libjpeg.so /usr/lib/aarch64-linux-gnu/libpng.so /usr/lib/aarch64-linux-gnu/libtiff.so vtkImagingCore vtkRenderingCore vtkCommonColor vtkFiltersExtraction vtkFiltersCore vtkFiltersGeneral vtkCommonComputationalGeometry vtkFiltersStatistics vtkImagingFourier vtkalglib vtkFiltersGeometry vtkFiltersSources vtkInteractionStyle vtkRenderingLOD vtkFiltersModeling vtkIOPLY vtkIOGeometry /usr/lib/aarch64-linux-gnu/libjsoncpp.so vtkFiltersTexture vtkRenderingFreeType /usr/lib/aarch64-linux-gnu/libfreetype.so vtkftgl vtkIOExport vtkRenderingAnnotation vtkImagingColor vtkRenderingContext2D vtkRenderingGL2PS vtkRenderingContextOpenGL /usr/lib/libgl2ps.so vtkRenderingLabel dl m pthread rt /usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so cudart nppc nppi npps cufft -L/usr/local/cuda-8.0/lib64
--     3rdparty dependencies:       IlmImf tegra_hal
--
--   OpenCV modules:
--     To be built:                 cudev core cudaarithm flann hdf imgproc ml reg surface_matching video viz cudabgsegm cudafilters cudaimgproc cudawarping freetype fuzzy imgcodecs photo shape videoio cudacodec highgui objdetect plot ts xobjdetect xphoto bgsegm bioinspired dpm face features2d line_descriptor saliency text calib3d ccalib cudafeatures2d cudalegacy cudaobjdetect cudaoptflow cudastereo datasets rgbd stereo superres videostab xfeatures2d ximgproc aruco optflow phase_unwrapping stitching structured_light java python2 python3
--     Disabled:                    world contrib_world
--     Disabled by dependency:      tracking
--     Unavailable:                 cnn_3dobj cvv dnn matlab sfm
--
--   GUI:
--     QT:                          NO
--     GTK+ 2.x:                    YES (ver 2.24.30)
--     GThread :                    YES (ver 2.48.1)
--     GtkGlExt:                    YES (ver 1.2.0)
--     OpenGL support:              YES (/usr/lib/aarch64-linux-gnu/libGLU.so /usr/lib/aarch64-linux-gnu/libGL.so)
--     VTK support:                 YES (ver 6.2.0)
--
--   Media I/O:
--     ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
--     JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
--     WEBP:                        /usr/lib/aarch64-linux-gnu/libwebp.so (ver encoder: 0x0202)
--     PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
--     TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 - 4.0.6)
--     JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
--     OpenEXR:                     build (ver 1.7.1)
--     GDAL:                        NO
--     GDCM:                        NO
--
--   Video I/O:
--     DC1394 1.x:                  NO
--     DC1394 2.x:                  YES (ver 2.2.4)
--     FFMPEG:                      YES
--       avcodec:                   YES (ver 56.60.100)
--       avformat:                  YES (ver 56.40.101)
--       avutil:                    YES (ver 54.31.100)
--       swscale:                   YES (ver 3.1.101)
--       avresample:                NO
--     GStreamer:
--       base:                      YES (ver 1.8.2)
--       video:                     YES (ver 1.8.2)
--       app:                       YES (ver 1.8.2)
--       riff:                      YES (ver 1.8.2)
--       pbutils:                   YES (ver 1.8.2)
--     OpenNI:                      NO
--     OpenNI PrimeSensor Modules:  NO
--     OpenNI2:                     NO
--     PvAPI:                       NO
--     GigEVisionSDK:               NO
--     Aravis SDK:                  NO
--     UniCap:                      NO
--     UniCap ucil:                 NO
--     V4L/V4L2:                    NO/YES
--     XIMEA:                       NO
--     Xine:                        NO
--     gPhoto2:                     NO
--
--   Parallel framework:            pthreads
--
--   Other third-party libraries:
--     Use IPP:                     NO
--     Use VA:                      NO
--     Use Intel VA-API/OpenCL:     NO
--     Use Lapack:                  NO
--     Use Eigen:                   YES (ver 3.2.92)
--     Use Cuda:                    YES (ver 8.0)
--     Use OpenCL:                  YES
--     Use OpenVX:                  NO
--     Use custom HAL:              YES (carotene (ver 0.0.1))
--
--   NVIDIA CUDA
--     Use CUFFT:                   YES
--     Use CUBLAS:                  NO
--     USE NVCUVID:                 NO
--     NVIDIA GPU arch:             53
--     NVIDIA PTX archs:
--     Use fast math:               NO
--
--   OpenCL:                        <Dynamic loading of OpenCL library>
--     Include path:                /media/ubuntu/91df5a31-05fd-4885-b29b-d7e625107ac5/opencv/opencv/3rdparty/include/opencl/1.2
--     Use AMDFFT:                  NO
--     Use AMDBLAS:                 NO
--
--   Python 2:
--     Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
--     Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
--     numpy:                       /usr/lib/python2.7/dist-packages/numpy/core/include (ver 1.11.0)
--     packages path:               lib/python2.7/dist-packages
--
--   Python 3:
--     Interpreter:                 /usr/bin/python3 (ver 3.5.2)
--     Libraries:                   /usr/lib/aarch64-linux-gnu/libpython3.5m.so (ver 3.5.2)
--     numpy:                       /usr/local/lib/python3.5/dist-packages/numpy/core/include (ver 1.11.3)
--     packages path:               lib/python3.5/dist-packages
--
--   Python (for build):            /usr/bin/python3
--
--   Java:
--     ant:                         /usr/bin/ant (ver 1.9.6)
--     JNI:                         /usr/lib/jvm/java-8-oracle/include /usr/lib/jvm/java-8-oracle/include/linux /usr/lib/jvm/java-8-oracle/include
--     Java wrappers:               YES
--     Java tests:                  YES
--
--   Matlab:                        Matlab not found or implicitly disabled
--
--   Documentation:
--     Doxygen:                     NO
--
--   Tests and samples:
--     Tests:                       YES
--     Performance tests:           YES
--     C/C++ Examples:              YES
--
--   Install path:                  /usr/local
--
--   cvconfig.h is in:              /media/ubuntu/91df5a31-05fd-4885-b29b-d7e625107ac5/opencv/opencv/test_example
-- -----------------------------------------------------------------
--
-- Configuring done

I test the tegra_multimedia_api sample 13. It works fine. I think the jpeg decoder work fine. but can not be used by opencv. refer to topic.
https://devtalk.nvidia.com/default/topic/984850/jetson-tx1/how-to-convert-yuv-to-jpg-using-jpeg-encoder-hardware-/

Questions:

How could I got it work with opencv?

Hello, Allen:
Do you build up a new camera sensor by ‘ov5640’?
what’s the output of that sensor? Why you think that error comes from jpeg lib?

From you GST pipeline, the sensor outputs YUV data, right?
you can also try GST pipeline in opencv if that sensor works well in GST.

br
Chenjian

Hi, jachen

Yes I got YUV data from ov5640.

When I compile with /usr/lib/aarch64-linux-gnu/libjpeg.so, I can not get fram from opencv with VideoCapture. Maybe caused by something else.

Could I have a efficient way to convert the UYVY frame to RGB24 by the hardware?

Hello, Allen:
From your code (below ‘But I can not open the camera with VideoCapture(0) to preivew.’), I cannot see anything related to libjpeg. Please check that again.

As to format conversion, you can either try GST pluin ‘nvvidconv’ or refer to code in tegra_multimedia_api/samples/07_video_convert (installed from Jetpack by selecting ‘Multimedia API package’).

br
ChenJian

Hi, Jachen

Sorry to response late.
Thanks for your suggestion.

I’ll try.

Thanks.