Jetpack 4.2 + Nano + IMX219 - 3 out of every 4 frames is blank?

With Jetpack 4.2 + Nano + an IMX219 camera (Buy a Raspberry Pi Camera Module 2 NoIR – Raspberry Pi)

When we use libargus, the IMX219 works straight out of the box. Yay.

However when we use V4L2 directly, 3 out of every 4 frames are black.

i.e. if we use the demo:
tegra_multimedia_api\samples\v4l2cuda

with:

static const char *     dev_name        = "/dev/video0";
static io_method        io              = IO_METHOD_USERPTR;
static int              fd              = -1;
struct buffer *         buffers         = NULL;
static unsigned int     n_buffers       = 0;
static unsigned int     width           = 1920;
static unsigned int     height          = 1080;
static unsigned int     count           = 20;
static unsigned char *  cuda_out_buffer = NULL;
static bool             cuda_zero_copy = true;
static const char *     file_name       = "out.ppm";
static unsigned int     pixel_format    = V4L2_PIX_FMT_SRGGB10;
static unsigned int     field           = V4L2_FIELD_NONE;

and simply average the pixel values in process_image instead of doing CUDA work to see the black frames:

static void
process_image                   (void *           p)
{
    printf ("CUDA format conversion on frame %p\n", p);
#if 0
    gpuConvertYUYVtoRGB ((unsigned char *) p, cuda_out_buffer, width, height);

    /* Save image. */
    if (count == 0) {
        FILE *fp = fopen (file_name, "wb");
        fprintf (fp, "P6\n%u %u\n255\n", width, height);
        fwrite (cuda_out_buffer, 1, width * height * 3, fp);
        fclose (fp);
    }
#endif
    
    long long total = 0;
    char* p2 = (char*)p;
    for(int i = 0; i < width*height; ++i) {
    	uint16_t val = *((uint16_t*)(&p2[i*2]));
    	total += val;
    }
    
    printf("avg = %2.2f\n", (static_cast<double>(total))/(width*height));
}

we see:

CUDA format conversion on frame 0x7fb2868000
avg = 118.77
CUDA format conversion on frame 0x7fb2473000
avg = 0.00
CUDA format conversion on frame 0x7fb207e000
avg = 0.00
CUDA format conversion on frame 0x7fb1c89000
avg = 0.00
CUDA format conversion on frame 0x7fb2868000
avg = 118.72
CUDA format conversion on frame 0x7fb2473000
avg = 0.00
CUDA format conversion on frame 0x7fb207e000
avg = 0.00
CUDA format conversion on frame 0x7fb1c89000
avg = 0.00
CUDA format conversion on frame 0x7fb2868000
avg = 118.75
CUDA format conversion on frame 0x7fb2473000
avg = 0.00
CUDA format conversion on frame 0x7fb207e000
avg = 0.00
CUDA format conversion on frame 0x7fb1c89000
avg = 0.00
CUDA format conversion on frame 0x7fb2868000
avg = 118.95
CUDA format conversion on frame 0x7fb2473000
avg = 0.00
CUDA format conversion on frame 0x7fb207e000
avg = 0.00
CUDA format conversion on frame 0x7fb1c89000
avg = 0.00
CUDA format conversion on frame 0x7fb2868000
avg = 128.40
CUDA format conversion on frame 0x7fb2473000
avg = 0.00
CUDA format conversion on frame 0x7fb207e000
avg = 0.00
CUDA format conversion on frame 0x7fb1c89000
avg = 0.00

which can be confirmed if we save out the images that indeed 3 out of every 4 images are black. Using IO_METHOD_MMAP produces the same result, but runs slower.

I feel like some of the default settings are either wrong or libargus is doing something else underneath.

Full code

#include <stdint.h>
/*
 * Copyright (c) 2015, NVIDIA CORPORATION. All rights reserved.
 *
 * Permission is hereby granted, free of charge, to any person obtaining a
 * copy of this software and associated documentation files (the "Software"),
 * to deal in the Software without restriction, including without limitation
 * the rights to use, copy, modify, merge, publish, distribute, sublicense,
 * and/or sell copies of the Software, and to permit persons to whom the
 * Software is furnished to do so, subject to the following conditions:
 *
 * The above copyright notice and this permission notice shall be included in
 * all copies or substantial portions of the Software.
 *
 * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL
 * THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
 * FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
 * DEALINGS IN THE SOFTWARE.
 */

/*
 *  V4L2 video capture example
 */

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <assert.h>

#include <getopt.h>             /* getopt_long() */

#include <fcntl.h>              /* low-level i/o */
#include <unistd.h>
#include <errno.h>
#include <malloc.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <sys/time.h>
#include <sys/mman.h>
#include <sys/ioctl.h>

#include <asm/types.h>          /* for videodev2.h */

#include <linux/videodev2.h>

#include <cuda_runtime.h>
#include "yuv2rgb.cuh"

#define CLEAR(x) memset (&(x), 0, sizeof (x))
#define ARRAY_SIZE(a)   (sizeof(a)/sizeof((a)[0]))

typedef enum {
    IO_METHOD_READ,
    IO_METHOD_MMAP,
    IO_METHOD_USERPTR,
} io_method;

struct buffer {
    void *                  start;
    size_t                  length;
};

static const char *     dev_name        = "/dev/video0";
static io_method        io              = IO_METHOD_USERPTR;
static int              fd              = -1;
struct buffer *         buffers         = NULL;
static unsigned int     n_buffers       = 0;
static unsigned int     width           = 1920;
static unsigned int     height          = 1080;
static unsigned int     count           = 20;
static unsigned char *  cuda_out_buffer = NULL;
static bool             cuda_zero_copy = true;
static const char *     file_name       = "out.ppm";
static unsigned int     pixel_format    = V4L2_PIX_FMT_SRGGB10;
static unsigned int     field           = V4L2_FIELD_NONE;

static void
errno_exit                      (const char *           s)
{
    fprintf (stderr, "%s error %d, %s\n",
            s, errno, strerror (errno));

    exit (EXIT_FAILURE);
}

static int
xioctl                          (int                    fd,
                                 int                    request,
                                 void *                 arg)
{
    int r;

    do r = ioctl (fd, request, arg);
    while (-1 == r && EINTR == errno);

    return r;
}

static void
process_image                   (void *           p)
{
    printf ("CUDA format conversion on frame %p\n", p);
#if 0
    gpuConvertYUYVtoRGB ((unsigned char *) p, cuda_out_buffer, width, height);

    /* Save image. */
    if (count == 0) {
        FILE *fp = fopen (file_name, "wb");
        fprintf (fp, "P6\n%u %u\n255\n", width, height);
        fwrite (cuda_out_buffer, 1, width * height * 3, fp);
        fclose (fp);
    }
#endif
    
    long long total = 0;
    char* p2 = (char*)p;
    for(int i = 0; i < width*height; ++i) {
    	uint16_t val = *((uint16_t*)(&p2[i*2]));
    	//printf("%d ", val);
    	total += val;
    }
    
    printf("avg = %2.2f\n", (static_cast<double>(total))/(width*height));
}

static int
read_frame                      (void)
{
    struct v4l2_buffer buf;
    unsigned int i;

    switch (io) {
        case IO_METHOD_READ:
            if (-1 == read (fd, buffers[0].start, buffers[0].length)) {
                switch (errno) {
                    case EAGAIN:
                        return 0;

                    case EIO:
                        /* Could ignore EIO, see spec. */

                        /* fall through */

                    default:
                        errno_exit ("read");
                }
            }

            process_image (buffers[0].start);

            break;

        case IO_METHOD_MMAP:
            CLEAR (buf);

            buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
            buf.memory = V4L2_MEMORY_MMAP;

            if (-1 == xioctl (fd, VIDIOC_DQBUF, &buf)) {
                switch (errno) {
                    case EAGAIN:
                        return 0;

                    case EIO:
                        /* Could ignore EIO, see spec. */

                        /* fall through */

                    default:
                        errno_exit ("VIDIOC_DQBUF");
                }
            }

            assert (buf.index < n_buffers);

            process_image (buffers[buf.index].start);

            if (-1 == xioctl (fd, VIDIOC_QBUF, &buf))
                errno_exit ("VIDIOC_QBUF");

            break;

        case IO_METHOD_USERPTR:
            CLEAR (buf);

            buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
            buf.memory = V4L2_MEMORY_USERPTR;

            if (-1 == xioctl (fd, VIDIOC_DQBUF, &buf)) {
                switch (errno) {
                    case EAGAIN:
                        return 0;

                    case EIO:
                        /* Could ignore EIO, see spec. */

                        /* fall through */

                    default:
                        errno_exit ("VIDIOC_DQBUF");
                }
            }

            for (i = 0; i < n_buffers; ++i)
                if (buf.m.userptr == (unsigned long) buffers[i].start
                        && buf.length == buffers[i].length)
                    break;

            assert (i < n_buffers);

            process_image ((void *) buf.m.userptr);

            if (-1 == xioctl (fd, VIDIOC_QBUF, &buf))
                errno_exit ("VIDIOC_QBUF");

            break;
    }

    return 1;
}

static void
mainloop                        (void)
{
    while (count-- > 0) {
        for (;;) {
            fd_set fds;
            struct timeval tv;
            int r;

            FD_ZERO (&fds);
            FD_SET (fd, &fds);

            /* Timeout. */
            tv.tv_sec = 2;
            tv.tv_usec = 0;

            r = select (fd + 1, &fds, NULL, NULL, &tv);

            if (-1 == r) {
                if (EINTR == errno)
                    continue;

                errno_exit ("select");
            }

            if (0 == r) {
                fprintf (stderr, "select timeout\n");
                exit (EXIT_FAILURE);
            }

            if (read_frame ())
                break;

            /* EAGAIN - continue select loop. */
        }
    }
}

static void
stop_capturing                  (void)
{
    enum v4l2_buf_type type;

    switch (io) {
        case IO_METHOD_READ:
            /* Nothing to do. */
            break;

        case IO_METHOD_MMAP:
        case IO_METHOD_USERPTR:
            type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

            if (-1 == xioctl (fd, VIDIOC_STREAMOFF, &type))
                errno_exit ("VIDIOC_STREAMOFF");

            break;
    }
}

static void
start_capturing                 (void)
{
    unsigned int i;
    enum v4l2_buf_type type;

    switch (io) {
        case IO_METHOD_READ:
            /* Nothing to do. */
            break;

        case IO_METHOD_MMAP:
            for (i = 0; i < n_buffers; ++i) {
                struct v4l2_buffer buf;

                CLEAR (buf);

                buf.type        = V4L2_BUF_TYPE_VIDEO_CAPTURE;
                buf.memory      = V4L2_MEMORY_MMAP;
                buf.index       = i;

                if (-1 == xioctl (fd, VIDIOC_QBUF, &buf))
                    errno_exit ("VIDIOC_QBUF");
            }

            type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

            if (-1 == xioctl (fd, VIDIOC_STREAMON, &type))
                errno_exit ("VIDIOC_STREAMON");

            break;

        case IO_METHOD_USERPTR:
            for (i = 0; i < n_buffers; ++i) {
                struct v4l2_buffer buf;

                CLEAR (buf);

                buf.type        = V4L2_BUF_TYPE_VIDEO_CAPTURE;
                buf.memory      = V4L2_MEMORY_USERPTR;
                buf.index       = i;
                buf.m.userptr   = (unsigned long) buffers[i].start;
                buf.length      = buffers[i].length;

                if (-1 == xioctl (fd, VIDIOC_QBUF, &buf))
                    errno_exit ("VIDIOC_QBUF");
            }

            type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

            if (-1 == xioctl (fd, VIDIOC_STREAMON, &type))
                errno_exit ("VIDIOC_STREAMON");

            break;
    }
}

static void
uninit_device                   (void)
{
    unsigned int i;

    switch (io) {
        case IO_METHOD_READ:
            free (buffers[0].start);
            break;

        case IO_METHOD_MMAP:
            for (i = 0; i < n_buffers; ++i)
                if (-1 == munmap (buffers[i].start, buffers[i].length))
                    errno_exit ("munmap");
            break;

        case IO_METHOD_USERPTR:
            for (i = 0; i < n_buffers; ++i) {
                if (cuda_zero_copy) {
                    cudaFree (buffers[i].start);
                } else {
                    free (buffers[i].start);
                }
            }
            break;
    }

    free (buffers);

    if (cuda_zero_copy) {
        cudaFree (cuda_out_buffer);
    }
}

static void
init_read                       (unsigned int           buffer_size)
{
    buffers = (struct buffer *) calloc (1, sizeof (*buffers));

    if (!buffers) {
        fprintf (stderr, "Out of memory\n");
        exit (EXIT_FAILURE);
    }

    buffers[0].length = buffer_size;
    buffers[0].start = malloc (buffer_size);

    if (!buffers[0].start) {
        fprintf (stderr, "Out of memory\n");
        exit (EXIT_FAILURE);
    }
}

static void
init_mmap                       (void)
{
    struct v4l2_requestbuffers req;

    CLEAR (req);

    req.count               = 4;
    req.type                = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    req.memory              = V4L2_MEMORY_MMAP;

    if (-1 == xioctl (fd, VIDIOC_REQBUFS, &req)) {
        if (EINVAL == errno) {
            fprintf (stderr, "%s does not support "
                    "memory mapping\n", dev_name);
            exit (EXIT_FAILURE);
        } else {
            errno_exit ("VIDIOC_REQBUFS");
        }
    }

    if (req.count < 2) {
        fprintf (stderr, "Insufficient buffer memory on %s\n",
                dev_name);
        exit (EXIT_FAILURE);
    }

    buffers = (struct buffer *) calloc (req.count, sizeof (*buffers));

    if (!buffers) {
        fprintf (stderr, "Out of memory\n");
        exit (EXIT_FAILURE);
    }

    for (n_buffers = 0; n_buffers < req.count; ++n_buffers) {
        struct v4l2_buffer buf;

        CLEAR (buf);

        buf.type        = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        buf.memory      = V4L2_MEMORY_MMAP;
        buf.index       = n_buffers;

        if (-1 == xioctl (fd, VIDIOC_QUERYBUF, &buf))
            errno_exit ("VIDIOC_QUERYBUF");

        buffers[n_buffers].length = buf.length;
        buffers[n_buffers].start =
            mmap (NULL /* start anywhere */,
                    buf.length,
                    PROT_READ | PROT_WRITE /* required */,
                    MAP_SHARED /* recommended */,
                    fd, buf.m.offset);

        if (MAP_FAILED == buffers[n_buffers].start)
            errno_exit ("mmap");
    }
}

static void
init_userp                      (unsigned int           buffer_size)
{
    struct v4l2_requestbuffers req;
    unsigned int page_size;

    page_size = getpagesize ();
    buffer_size = (buffer_size + page_size - 1) & ~(page_size - 1);

    CLEAR (req);

    req.count               = 4;
    req.type                = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    req.memory              = V4L2_MEMORY_USERPTR;

    if (-1 == xioctl (fd, VIDIOC_REQBUFS, &req)) {
        if (EINVAL == errno) {
            fprintf (stderr, "%s does not support "
                    "user pointer i/o\n", dev_name);
            exit (EXIT_FAILURE);
        } else {
            errno_exit ("VIDIOC_REQBUFS");
        }
    }

    buffers = (struct buffer *) calloc (4, sizeof (*buffers));

    if (!buffers) {
        fprintf (stderr, "Out of memory\n");
        exit (EXIT_FAILURE);
    }

    for (n_buffers = 0; n_buffers < 4; ++n_buffers) {
        buffers[n_buffers].length = buffer_size;
        if (cuda_zero_copy) {
            cudaMallocManaged (&buffers[n_buffers].start, buffer_size, cudaMemAttachGlobal);
        } else {
            buffers[n_buffers].start = memalign (/* boundary */ page_size,
                    buffer_size);
        }

        if (!buffers[n_buffers].start) {
            fprintf (stderr, "Out of memory\n");
            exit (EXIT_FAILURE);
        }
    }
}

static void
init_device                     (void)
{
    struct v4l2_capability cap;
    struct v4l2_cropcap cropcap;
    struct v4l2_crop crop;
    struct v4l2_format fmt;
    unsigned int min;

    if (-1 == xioctl (fd, VIDIOC_QUERYCAP, &cap)) {
        if (EINVAL == errno) {
            fprintf (stderr, "%s is no V4L2 device\n",
                    dev_name);
            exit (EXIT_FAILURE);
        } else {
            errno_exit ("VIDIOC_QUERYCAP");
        }
    }

    if (!(cap.capabilities & V4L2_CAP_VIDEO_CAPTURE)) {
        fprintf (stderr, "%s is no video capture device\n",
                dev_name);
        exit (EXIT_FAILURE);
    }

    switch (io) {
        case IO_METHOD_READ:
            if (!(cap.capabilities & V4L2_CAP_READWRITE)) {
                fprintf (stderr, "%s does not support read i/o\n",
                        dev_name);
                exit (EXIT_FAILURE);
            }

            break;

        case IO_METHOD_MMAP:
        case IO_METHOD_USERPTR:
            if (!(cap.capabilities & V4L2_CAP_STREAMING)) {
                fprintf (stderr, "%s does not support streaming i/o\n",
                        dev_name);
                exit (EXIT_FAILURE);
            }

            break;
    }


    /* Select video input, video standard and tune here. */


    CLEAR (cropcap);

    cropcap.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;

    if (0 == xioctl (fd, VIDIOC_CROPCAP, &cropcap)) {
        crop.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
        crop.c = cropcap.defrect; /* reset to default */

        if (-1 == xioctl (fd, VIDIOC_S_CROP, &crop)) {
            switch (errno) {
                case EINVAL:
                    /* Cropping not supported. */
                    break;
                default:
                    /* Errors ignored. */
                    break;
            }
        }
    } else {
        /* Errors ignored. */
    }


    CLEAR (fmt);

    fmt.type                = V4L2_BUF_TYPE_VIDEO_CAPTURE;
    fmt.fmt.pix.width       = width;
    fmt.fmt.pix.height      = height;
    fmt.fmt.pix.pixelformat = pixel_format;
    fmt.fmt.pix.field       = field;

    if (-1 == xioctl (fd, VIDIOC_S_FMT, &fmt))
        errno_exit ("VIDIOC_S_FMT");

    /* Note VIDIOC_S_FMT may change width and height. */

    /* Buggy driver paranoia. */
    min = fmt.fmt.pix.width * 2;
    if (fmt.fmt.pix.bytesperline < min)
        fmt.fmt.pix.bytesperline = min;
    min = fmt.fmt.pix.bytesperline * fmt.fmt.pix.height;
    if (fmt.fmt.pix.sizeimage < min)
        fmt.fmt.pix.sizeimage = min;

    switch (io) {
        case IO_METHOD_READ:
            init_read (fmt.fmt.pix.sizeimage);
            break;

        case IO_METHOD_MMAP:
            init_mmap ();
            break;

        case IO_METHOD_USERPTR:
            init_userp (fmt.fmt.pix.sizeimage);
            break;
    }
}

static void
close_device                    (void)
{
    if (-1 == close (fd))
        errno_exit ("close");

    fd = -1;
}

static void
open_device                     (void)
{
    struct stat st;

    if (-1 == stat (dev_name, &st)) {
        fprintf (stderr, "Cannot identify '%s': %d, %s\n",
                dev_name, errno, strerror (errno));
        exit (EXIT_FAILURE);
    }

    if (!S_ISCHR (st.st_mode)) {
        fprintf (stderr, "%s is no device\n", dev_name);
        exit (EXIT_FAILURE);
    }

    fd = open (dev_name, O_RDWR /* required */ | O_NONBLOCK, 0);

    if (-1 == fd) {
        fprintf (stderr, "Cannot open '%s': %d, %s\n",
                dev_name, errno, strerror (errno));
        exit (EXIT_FAILURE);
    }
}

static void
init_cuda                       (void)
{
    /* Check unified memory support. */
    if (cuda_zero_copy) {
        cudaDeviceProp devProp;
        cudaGetDeviceProperties (&devProp, 0);
        if (!devProp.managedMemory) {
            printf ("CUDA device does not support managed memory.\n");
            cuda_zero_copy = false;
        }
    }

    /* Allocate output buffer. */
    size_t size = width * height * 3;
    if (cuda_zero_copy) {
        cudaMallocManaged (&cuda_out_buffer, size, cudaMemAttachGlobal);
    } else {
        cuda_out_buffer = (unsigned char *) malloc (size);
    }

    cudaDeviceSynchronize ();
}

static void
usage                           (FILE *                 fp,
                                 int                    argc,
                                 char **                argv)
{
    fprintf (fp,
            "Usage: %s [options]\n\n"
            "Options:\n"
            "-c | --count N       Frame count (default: %u)\n"
            "-d | --device name   Video device name (default: %s)\n"
            "-f | --format        Capture input pixel format (default: UYVY)\n"
            "-h | --help          Print this message\n"
            "-m | --mmap          Use memory mapped buffers\n"
            "-o | --output        Output file name (default: %s)\n"
            "-s | --size WxH      Frame size (default: %ux%u)\n"
            "-u | --userp         Use application allocated buffers\n"
            "-z | --zcopy         Use zero copy CUDA memory\n"
            "Experimental options:\n"
            "-r | --read          Use read() calls\n"
            "-F | --field         Capture field (default: INTERLACED)\n"
            "",
            argv[0], count, dev_name, file_name, width, height);
}

static const char short_options [] = "c:d:f:F:hmo:rs:uz";

static const struct option
long_options [] = {
    { "count",      required_argument,      NULL,           'c' },
    { "device",     required_argument,      NULL,           'd' },
    { "format",     required_argument,      NULL,           'f' },
    { "field",      required_argument,      NULL,           'F' },
    { "help",       no_argument,            NULL,           'h' },
    { "mmap",       no_argument,            NULL,           'm' },
    { "output",     required_argument,      NULL,           'o' },
    { "read",       no_argument,            NULL,           'r' },
    { "size",       required_argument,      NULL,           's' },
    { "userp",      no_argument,            NULL,           'u' },
    { "zcopy",      no_argument,            NULL,           'z' },
    { 0, 0, 0, 0 }
};

static struct {
    const char *name;
    unsigned int fourcc;
} pixel_formats[] = {
    { "RGB332", V4L2_PIX_FMT_RGB332 },
    { "RGB555", V4L2_PIX_FMT_RGB555 },
    { "RGB565", V4L2_PIX_FMT_RGB565 },
    { "RGB555X", V4L2_PIX_FMT_RGB555X },
    { "RGB565X", V4L2_PIX_FMT_RGB565X },
    { "BGR24", V4L2_PIX_FMT_BGR24 },
    { "RGB24", V4L2_PIX_FMT_RGB24 },
    { "BGR32", V4L2_PIX_FMT_BGR32 },
    { "RGB32", V4L2_PIX_FMT_RGB32 },
    { "Y8", V4L2_PIX_FMT_GREY },
    { "Y10", V4L2_PIX_FMT_Y10 },
    { "Y12", V4L2_PIX_FMT_Y12 },
    { "Y16", V4L2_PIX_FMT_Y16 },
    { "UYVY", V4L2_PIX_FMT_UYVY },
    { "VYUY", V4L2_PIX_FMT_VYUY },
    { "YUYV", V4L2_PIX_FMT_YUYV },
    { "YVYU", V4L2_PIX_FMT_YVYU },
    { "NV12", V4L2_PIX_FMT_NV12 },
    { "NV21", V4L2_PIX_FMT_NV21 },
    { "NV16", V4L2_PIX_FMT_NV16 },
    { "NV61", V4L2_PIX_FMT_NV61 },
    { "NV24", V4L2_PIX_FMT_NV24 },
    { "NV42", V4L2_PIX_FMT_NV42 },
    { "SBGGR8", V4L2_PIX_FMT_SBGGR8 },
    { "SGBRG8", V4L2_PIX_FMT_SGBRG8 },
    { "SGRBG8", V4L2_PIX_FMT_SGRBG8 },
    { "SRGGB8", V4L2_PIX_FMT_SRGGB8 },
    { "SBGGR10_DPCM8", V4L2_PIX_FMT_SBGGR10DPCM8 },
    { "SGBRG10_DPCM8", V4L2_PIX_FMT_SGBRG10DPCM8 },
    { "SGRBG10_DPCM8", V4L2_PIX_FMT_SGRBG10DPCM8 },
    { "SRGGB10_DPCM8", V4L2_PIX_FMT_SRGGB10DPCM8 },
    { "SBGGR10", V4L2_PIX_FMT_SBGGR10 },
    { "SGBRG10", V4L2_PIX_FMT_SGBRG10 },
    { "SGRBG10", V4L2_PIX_FMT_SGRBG10 },
    { "SRGGB10", V4L2_PIX_FMT_SRGGB10 },
    { "SBGGR12", V4L2_PIX_FMT_SBGGR12 },
    { "SGBRG12", V4L2_PIX_FMT_SGBRG12 },
    { "SGRBG12", V4L2_PIX_FMT_SGRBG12 },
    { "SRGGB12", V4L2_PIX_FMT_SRGGB12 },
    { "DV", V4L2_PIX_FMT_DV },
    { "MJPEG", V4L2_PIX_FMT_MJPEG },
    { "MPEG", V4L2_PIX_FMT_MPEG },
};

static unsigned int v4l2_format_code(const char *name)
{
    unsigned int i;

    for (i = 0; i < ARRAY_SIZE(pixel_formats); ++i) {
        if (strcasecmp(pixel_formats[i].name, name) == 0)
            return pixel_formats[i].fourcc;
    }

    return 0;
}

static struct {
    const char *name;
    unsigned int field;
} fields[] = {
    { "ANY", V4L2_FIELD_ANY },
    { "NONE", V4L2_FIELD_NONE },
    { "TOP", V4L2_FIELD_TOP },
    { "BOTTOM", V4L2_FIELD_BOTTOM },
    { "INTERLACED", V4L2_FIELD_INTERLACED },
    { "SEQ_TB", V4L2_FIELD_SEQ_TB },
    { "SEQ_BT", V4L2_FIELD_SEQ_BT },
    { "ALTERNATE", V4L2_FIELD_ALTERNATE },
    { "INTERLACED_TB", V4L2_FIELD_INTERLACED_TB },
    { "INTERLACED_BT", V4L2_FIELD_INTERLACED_BT },
};

static unsigned int v4l2_field_code(const char *name)
{
    unsigned int i;

    for (i = 0; i < ARRAY_SIZE(fields); ++i) {
        if (strcasecmp(fields[i].name, name) == 0)
            return fields[i].field;
    }

    return -1;
}

int
main                            (int                    argc,
                                 char **                argv)
{
    for (;;) {
        int index;
        int c;

        c = getopt_long (argc, argv,
                short_options, long_options,
                &index);

        if (-1 == c)
            break;

        switch (c) {
            case 0: /* getopt_long() flag */
                break;

            case 'c':
                count = atoi (optarg);
                break;

            case 'd':
                dev_name = optarg;
                break;

            case 'f':
                pixel_format = v4l2_format_code(optarg);
                if (pixel_format == 0) {
                    printf("Unsupported video format '%s'\n", optarg);
                    pixel_format = V4L2_PIX_FMT_UYVY;
                }
                break;

            case 'F':
                field = v4l2_field_code(optarg);
                if ((int)field < 0) {
                    printf("Unsupported field '%s'\n", optarg);
                    field = V4L2_FIELD_INTERLACED;
                }
                break;

            case 'h':
                usage (stdout, argc, argv);
                exit (EXIT_SUCCESS);

            case 'm':
                io = IO_METHOD_MMAP;
                break;

            case 'o':
                file_name = optarg;
                break;

            case 'r':
                io = IO_METHOD_READ;
                break;

            case 's':
                width = atoi (strtok (optarg, "x"));
                height = atoi (strtok (NULL, "x"));
                break;

            case 'u':
                io = IO_METHOD_USERPTR;
                break;

            case 'z':
                cuda_zero_copy = true;
                break;

            default:
                usage (stderr, argc, argv);
                exit (EXIT_FAILURE);
        }
    }

    open_device ();

    init_device ();

    init_cuda ();

    start_capturing ();

    mainloop ();

    stop_capturing ();

    uninit_device ();

    close_device ();

    exit (EXIT_SUCCESS);

    return 0;
}

Thoughts?

Thanks

hello RS64,

could you please dump the raw files with v4l2 standard controls,

$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=2592,height=1944,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=5 --stream-to=test.raw

you may use the 3rdparty tools, such as 7yuv to analysis the raw files.
thanks

Thanks Jerry,

We see the same results. The .raw file produced is available in the .zip below if you would like to verify.

Using our own RG10 → RGB conversion below we see frames 0 and 4 have data, but frames 1,2,and 3 are black:

For completeness, the .raw file, the output .bmp files from the application below, the source code and a debug build are included in this .zip for anyone that wants to use it.

https://drive.google.com/open?id=1E2rFrObSuuU7PXPXylPhC9n8Bc8qsBc1

We run:

RGGB12_to_RGB8.exe c:\temp\test_2592_1944_5_frame.raw 1944 5

where:

RGB12_to_RGB8 <image_file.raw> <heightPx> <framecnt>

will calculate the stride (width) for us (which does so correctly seeing 2592 x 1944 raw frames).

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <string>
#include <fstream>
#include <opencv2/core/core.hpp>
#include <opencv2/opencv.hpp>

using namespace cv;
using namespace std;

int main(int argc, char* argv[])
{
	if(argc != 4) {
		printf("Usage: RGB12_to_RGB8 <image_file.raw> <heightPx> <framecnt> \r\n\r\n");
		return -1;
	}
	
	FILE* pFile = nullptr;
	
	std::ifstream file(argv[1], std::ios::binary | std::ios::ate);
	std::streamsize fileSizeBytes = file.tellg();
	if( fileSizeBytes <= 0 ) {
		printf("File %s has size %d bytes", argv[1], (int)fileSizeBytes);
		return 1;
	}
	file.seekg( 0, std::ios::beg );

	int heightPx = stoi(argv[2]);
	int frameCount = stoi(argv[3]);

	// calculate stride based on image height and file size
	int widthPx = (int)fileSizeBytes / 2 / heightPx / frameCount;
	
	printf("Convert RGGB 12 'RG12' to RGB8, width: %d height: %d\r\n", widthPx, heightPx);

	char* buffer = (char*)calloc(fileSizeBytes, 1);
	if (file.read(buffer, fileSizeBytes)) {
		for (int i = 0; i < frameCount; ++i) {
			uint16_t* p = (uint16_t*)(&buffer[i * widthPx * heightPx * 2]);

			long long total = 0;
			double average = 0;
			uint16_t max = 0;

			for(int j = 0; j < widthPx * heightPx; ++j) {
				uint16_t val = p[j];
				total += val;
				if( val > max ) {
					max = val;
				}
			}

			average = static_cast<double>(total) / (widthPx * heightPx);
			printf("Frame %d max: %d total: %lld average: %2.2f\n", i, max, total, average);

			cv::Mat inputMat_16UC1 = cv::Mat(heightPx, widthPx, CV_16UC1, p);
			cv::Mat convertMat_16UC3 = cv::Mat( heightPx, widthPx, CV_16UC3);
			cv::cvtColor(inputMat_16UC1, convertMat_16UC3, cv::COLOR_BayerRG2RGB);
			
			cv::Mat resultMat_8UC3 = cv::Mat(heightPx, widthPx, CV_8UC3);
			convertMat_16UC3.convertTo( resultMat_8UC3, convertMat_16UC3.type(), 1.0f); // use MSBs
			
			std::string outFileName = "output_" + std::to_string(i) + ".bmp";
			imwrite(outFileName, resultMat_8UC3 );
		}
		file.close();
	} else {
		printf("Could not open file: %s\r\n", argv[1]);
	}
	free(buffer);
	return 0;
}

We find the max pixel value, sum the total of all pixel values and find the average for each frame. As you can see frames 1,2,3 are black with no data

Output:

Convert RGGB 12 'RG12' to RGB8, width: 2592 height: 1944
Frame 0 max: 1023 total: 443900050 average: 88.10
Frame 1 max: 0 total: 0 average: 0.00
Frame 2 max: 0 total: 0 average: 0.00
Frame 3 max: 0 total: 0 average: 0.00
Frame 4 max: 1023 total: 436242729 average: 86.58

This was run on two separate Nano’s with 2 different IMX219 (V2.1) cameras, the issue persists on both setups

hello RS64,

by checking with 3rdparty tools, I’ve seen there are hex values for each pixels.
however, I cannot analysis your raw files as graphic mode by setting BayerRGGB/2592x1944.

could you please share your v4l2 capture commands?
thanks

Hi Jerry, I am using your command line exactly as you’ve stated it (and just renaming the output file afterwards).

What tools you are using to view SRRGB images? The raw bytes looks correct when I process from RG10 → RGB them in OpenCV as per the code above:

cv::Mat inputMat_16UC1 = cv::Mat(heightPx, widthPx, CV_16UC1, p);
cv::Mat convertMat_16UC3 = cv::Mat( heightPx, widthPx, CV_16UC3);
cv::cvtColor(inputMat_16UC1, convertMat_16UC3, cv::COLOR_BayerRG2RGB);
			
cv::Mat resultMat_8UC3 = cv::Mat(heightPx, widthPx, CV_8UC3);
convertMat_16UC3.convertTo( resultMat_8UC3, convertMat_16UC3.type(), 1.0f); // use MSBs

At 2592x1944 (and also 1920x1080) frames 0 and 4 appear correct when we look at the .bmp, which is confirmed by the raw byte analysis. Frames 1,2,3 are black.

To test further, yesturday I’ve updated the Nano to R32.2.

Interestingly, the issue is not present when using 3264x2464 resolution on R32.2 (but is on 32.1). However the issue remains for 2592 x 1944 and 1920 x 1080 on both both R32.1 and R32.2

Both the good (3264x2464) capture and bad (2592x1944) raw and bitmap converted images can be found here from the R32.2 test, using your command line:

https://drive.google.com/open?id=1WXk1qCtMYKsEkRDuagMAMH0RqwSXYO8Z

Given libargus works wtih the camera at the resolutions in question, it would be interesting to know exactly what settings/controls it is using with V4L2

hello RS64,

since libargus and v4l2src were going through different path. please refer to the Camera Architecture Stack.
this might be potential bugs in VI kernel driver, suggest you check the sensor capability and analysis the raw files.

could you please share all the sensor supported formats with the following command,
for example,

$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
	Index       : 0
	Type        : Video Capture
	Pixel Format: 'BG10'
	Name        : 10-bit Bayer BGBG/GRGR
		Size: Discrete 2592x1944
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 2592x1458
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.008s (120.000 fps)

I’m using this 3rdparty tool, 7yuv to analysis raw files.
thanks

nvidia@nano:~$ v4l2-ctl -d /dev/video0 --list-formats-ext
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'RG10'
        Name        : 10-bit Bayer RGRG/GBGB
                Size: Discrete 3264x2464
                        Interval: Discrete 0.048s (21.000 fps)
                Size: Discrete 3264x1848
                        Interval: Discrete 0.036s (28.000 fps)
                Size: Discrete 1920x1080
                        Interval: Discrete 0.033s (30.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)
                Size: Discrete 1280x720
                        Interval: Discrete 0.017s (60.000 fps)

Is your output for the IMX 219? I’m guessing it is something else. 2592x1944 is not listed, although the behavior is the same if we use 1920x1080

hello RS64,

that’s right, I’m using different sensor to list sensor formats for your reference.
however, by checking raw dumps with several sensor modules, I cannot reproduce blank frame issue from our site.

had you also confirm your raw files?
suggest you might also contact with your sensor vendor for more details.
thanks

Do you have an IMX219 laying around by chance?

I have confirmed the raw files, they look correct, for the frames we do get. This feels like a driver issue, which from /kernel/nvidia/drivers/media/i2c/imx219.c looks like it was written by nvidia.

Give this a try with the IMX219 if you can and let me know if your results do not match.

Thanks!

hello RS64,

I don’t have imx219 sensor modules on hand for validation,
suggest you share raw files with below resolutions for checking.
for example,

3264x2464:
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=3264,height=2464,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=5 --stream-to=imx219_3264x2464.raw

1920x1080:
$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=5 --stream-to=imx219_1920x1080.raw

Hi Jerry, as requested:

nvidia@nano-nvidia:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=3264,height=2464,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=5 --stream-to=imx219_3264x2464.raw
<<<<<
nvidia@nano-nvidia:~$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=5 --stream-to=imx219_1920x1080.raw
<<<<<

Both files can be found here:

https://drive.google.com/open?id=1HsI2t4Zd96YOx2ZBuuZdc2tSDjJKNqEv

If we open imx219_3264x2464.raw, we see image data throughout the file. This works correctly and is what we expect.

If we open the imx219_1920x1080.raw file and look at bytes without worry about external or 3rd party apps, looking at each start byte:

Frame 0 = (1920x1080x2) x 0 = 0 = 0x0 - We see frame data
Frame 1 = (1920x1080x2) x 1 = 4147200 = 0x3F4800 - We see all 0’s
Frame 2 = (1920x1080x2) x 2 = 8294400 = 0x7E9000 - We see all 0’s
Frame 3 = (1920x1080x2) x 3 = 12441600 = 0xBDD800 - We see all 0’s
Frame 4 = (1920x1080x2) x 4 = 16588800 = 0xFD2000 - We see frame data

hello RS64,

by checking the raw dumps, there should be some issue with 1920x1080 sensor modes.
suggest you working with full-resolution(3264x2464) mode as short-term solution, you might also contact with your sensor vendor to report the issue.
we would check internally about the issue,
thanks

Thanks Jerry,

It appears the driver is written by Nvidia (stock driver that comes with the Jetson Nano), although things seem to work fine in libargus. This makes me believe this is not a hardware issue, rather driver related or something on the V4L2 side of things.

hello RS64,

FYI,
sine I don’t have Raspberry Pi camera board on hand, I had verified imx219 drivers with other camera board on other platforms. (Jetson-TX2)
it turns out I cannot reproduce three out of every four-frames were blank issue with our reference camera board for 1920x1080 sensor mode.
suggest you also deliver this to your sensor vendor for investigation.
thanks