Low latency camera and software drivers?

[u]What approach is recommended to achieve low camera latency with Jetson Nano?
[/u]
For vision based robot driving I would like that the camera latency approaches 10ms.

I used the examples at https://www.jetsonhacks.com/2019/04/02/jetson-nano-raspberry-pi-camera/ which are based on gstreamer and IX219 camera.

The latency I measured is in 100-200ms range (taking image of stopwatch and a window showing the recorded stopwatch).

I am under assumption that CSI cameras can reach better latency than USB cameras. As reference I know that Raspi based vision system using OV5647 or similar with CSI interface (https://limelightvision.io) approaches 25ms total latency (incl opencv based object recognition).

I could not identify options in gst-inspect-1.0 nvarguscamerasrc that lower latency significantly.

I tested with:

cap = cv2.VideoCapture(gstreamer_pipeline(), cv2.CAP_GSTREAMER)

def gstreamer_pipeline(
    capture_width=1920, capture_height=1080,
    display_width=1280, display_height=720,
    framerate=30, exposure_time= 5, # ms
    flip_method=0):

    exposure_time = exposure_time * 1000000 #ms to ns
    exp_time_str = '"' + str(exposure_time) + ' ' + str(exposure_time) + '"'

    return (
        'nvarguscamerasrc '
        'name="NanoCam" '
        'do-timestamp=true '
        'timeout=0 '                               # 0 - 2147483647
        'blocksize=-1 '                            # block size in bytes
        'num-buffers=-1 '                          # -1..2147483647 (-1=ulimited) num buf before sending EOS
        'sensor-mode=-1 '                          # -1..255, IX279 0(3264x2464,21fps),1 (3264x1848,28),2(1080p.30),3(720p,60),4(=720p,120)
        'tnr-strength=-1 '                         # -1..1
        'tnr-mode=1 '                              # 0,1,2
#        'ee-mode=0'                               # 0,1,2
#        'ee-strength=-1 '                         # -1..1
        'aeantibanding=1 '                         # 0..3, off,auto,50,60Hz
        'bufapi-version=false '                    # new buffer api
        'maxperf=true '                            # max performance
        'silent=true '                             # verbose output
        'saturation=1 '                            # 0..2
        'wbmode=1 '                                # white balance mode, 0..9 0=off 1=auto
        'awblock=false '                           # auto white balance lock
        'aelock=true '                             # auto exposure lock
        'exposurecompensation=0 '                  # -2..2
        'exposuretimerange=%s '                    # "13000 683709000"
        'gainrange="1.0 10.625" '                  # "1.0 10.625"
        'ispdigitalgainrange="1 8" '             # "1 8"
        #
        '! video/x-raw(memory:NVMM), '
        'width=(int)%d, '
        'height=(int)%d, '
        'format=(string)NV12, '
        'framerate=(fraction)%d/1 '
        #
        '! nvvidconv flip-method=%d '
        '! video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx '
        #
        '! videoconvert '
        '! video/x-raw, format=(string)BGR '
        #
        '! appsink'
        % (
            exp_time_str,
            capture_width,
            capture_height,
            framerate,
            flip_method,
            display_width,
            display_height,
        )
)

@utzinger
Sorry to tell current best latency with ISP is about 4 frames. You may have select the high frame rate sensor mode like 120 fps sensor mode to get better performance.

@ShaneCCC

We are also facing similar problem. Is there a way to disable ISP to get single frame latency

Thanks

There’s no support to bypass ISP for argus interface. But v4l2(v4l2src) is the interface exclude the ISP, and if bayer sensor need to debayer from software by yourself.