Adquisition issue with Jetson Orin NX 16GB carried board J4012 and IMX477 camera

Hello,

I am experiencing a strange image acquisition issue in my project involving a Jetson Orin NX with an IMX477 camera connected. My goal is to capture images at a specific framerate and save them into a folder for use in other parts of my project.

The acquisition is performed using the following GStreamer pipeline:

def __gstreamer_pipeline__(
    sensor_id=0,
    width=1920,
    height=1080,
    framerate=30,
    flip_method=0,
):
    return (
        "nvarguscamerasrc sensor-id=%d sensor-mode=0 ! "
        "video/x-raw(memory:NVMM), width=(int)%d, height=(int)%d, framerate=(fraction)%d/1 ! "
        "nvvidconv flip-method=%d ! "
        "video/x-raw, width=(int)%d, height=(int)%d, format=(string)BGRx ! "
        "videoconvert ! "
        "video/x-raw, format=(string)BGR ! appsink"
        % (
            sensor_id,
            width,
            height,
            framerate,
            flip_method,
            width,
            height,
        )
    )

video_capture = cv2.VideoCapture(__gstreamer_pipeline__, cv2.CAP_GSTREAMER)
_, frame = video_capture.read()

However, the images I captured have the following issue: Image

As shown, the camera retains an older frame while it is building a new one. This occurs every time there is movement in the stream, making the issue most noticeable under those conditions.

I have already followed the configuration guidelines provided here:https://wiki.seeedstudio.com/J401_carrierboard_Hardware_Interfaces_Usage/#supported-cameras-are-as-follows but the issue is still there.

I have also tried to chage some parameters in the pipeline including: --exposuretimerange --gainrange --enc-bitrate --hw-enc-path in the nvarguscamerasrc element of the pipeline but it didn’t work.

What it is worse is that I have another Jetson Orin NX, identical to this one and connected to the same type of camera, which runs the exact same code without any issues. This leads me to believe that the problem relies on some hardware configuration I haven’t made.

I have been stuck on this issue for two weeks now and would greatly appreciate any guidance that could help resolve it. If you need any additional information to clarify my situation, please feel free to ask.

Thanks in advance!

What’s the version?
Could you confirm by argus_camera?

I am running JetPack 5.1.2 [L4T 35.4.1] and Ubuntu 20.04 and the nvarguscamera version is 1.0.0

jetson@jetson-154:~$ gst-inspect-1.0 nvarguscamerasrc
Factory Details:
  Rank                     primary (256)
  Long-name                NvArgusCameraSrc
  Klass                    Video/Capture
  Description              nVidia ARGUS Camera Source
  Author                   Viranjan Pagar <vpagar@nvidia.com>, Amit Pandya <apandya@nvidia.com>

Plugin Details:
  Name                     nvarguscamerasrc
  Description              nVidia ARGUS Source Component
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
  Version                  1.0.0
  License                  Proprietary
  Source module            nvarguscamerasrc
  Binary package           NvARGUSCameraSrc
  Origin URL               http://nvidia.com/

Please confirm by nvgstcapture-1.0 if able to reproduce the problem.

Hi, sorry for the delay in answering. I didn’t have access to the hardware to check what you were asking. Now I have everything again.

Running nvgstcapture-1.0 pops up a black window and returns the following log, where I see some errors:

jetson@jetson-152:~$ nvgstcapture-1.0 sensor-id=0
Encoder null, cannot set bitrate!
Encoder Profile = High
Supported resolutions in case of ARGUS Camera
  (2) : 640x480
  (3) : 1280x720
  (4) : 1920x1080
  (5) : 2104x1560
  (6) : 2592x1944
  (7) : 2616x1472
  (8) : 3840x2160
  (9) : 3896x2192
  (10): 4208x3120
  (11): 5632x3168
  (12): 5632x4224

Runtime ARGUS Camera Commands:

  Help : 'h'
  Quit : 'q'
  Set Capture Mode:
      mo:<val>
          (1): image
          (2): video
  Get Capture Mode:
      gmo
  Set sensor orientation:
      so:<val>
          (0): none
          (1): Rotate counter-clockwise 90 degrees
          (2): Rotate 180 degrees
          (3): Rotate clockwise 90 degrees
  Get sensor orientation:
      gso
  Set sensor mode:
      smo:<val> e.g., smo:1
  Get sensor mode:
      gsmo
  Set Whitebalance Mode:
      wb:<val>
          (0): off
          (1): auto
          (2): incandescent
          (3): fluorescent
          (4): warm-fluorescent
          (5): daylight
          (6): cloudy-daylight
          (7): twilight
          (8): shade
          (9): manual
  Get Whitebalance Mode:
      gwb
  Set Saturation (0 to 2):
      st:<val> e.g., st:1.25
  Get Saturation:
      gst
  Set Exposure Compensation (-2 to 2):
      ec:<val> e.g., ec:-2
  Get Exposure Compensation:
      gec
  Set Auto Whitebalance Lock:
      awbl:<val> e.g., awbl:0
  Get Auto Whitebalance Lock:
      awbl
  Set Auto Exposure Lock:
      ael:<val> e.g., ael:0
  Get Auto Exposure Lock:
      gael
  Set TNR Mode:
      tnrm:<val> e.g., tnrm:1
          (0): OFF
          (1): FAST
          (2): HIGH QUALITY
  Get TNR Mode:
      gtnrm
  Set TNR Strength (-1 to 1):
      tnrs:<val> e.g., tnrs:0.5
  Get TNR Strength:
      gtnrs
  Set EE Mode:
      eem:<val> e.g., eem:1
          (0): OFF
          (1): FAST
          (2): HIGH QUALITY
  Get EE Mode:
      geem
  Set EE Strength (-1 to 1):
      ees:<val> e.g., ees:0.5
  Get EE Strength:
      gees
  Set Auto Exposure Anti-Banding (0 to 3):
      aeab:<val> e.g., aeab:2
          (0): OFF
          (1): MODE AUTO
          (2): MODE 50HZ
          (3): MODE 60HZ
  Get Auto Exposure Anti-Banding:
      gaeab
  Set Gain Range:
      gr:<val><space><val> e.g., gr:1 16
  Get Gain Range:
      ggr
  Set Exposure Time Range:
      etr:<val><space><val> e.g., etr:34000 35000
  Get Exposure Time Range:
      getr
  Set ISP Digital Gain Range:
      dgr:<val><space><val> e.g., dgr:2 152
  Get ISP Digital Gain Range:
      gdgr
  Capture: enter 'j' OR
           followed by a timer (e.g., jx5000, capture after 5 seconds) OR
           followed by multishot count (e.g., j:6, capture 6 images)
           timer/multihot values are optional, capture defaults to single shot with timer=0s
  Start Recording : enter '1'
  Stop Recording  : enter '0'
  Video snapshot  : enter '2' (While recording video)
  Get Preview Resolution:
      gpcr
  Get Image Capture Resolution:
      gicr
  Get Video Capture Resolution:
      gvcr


Runtime encoder configuration options:

  Set Encoding Bit-rate(in bytes):
      br:<val> e.g., br:4000000
  Get Encoding Bit-rate(in bytes):
      gbr
  Set Encoding Profile(only for H.264):
      ep:<val> e.g., ep:1
          (0): Baseline
          (1): Main
          (2): High
  Get Encoding Profile(only for H.264):
      gep
  Force IDR Frame on video Encoder(only for H.264):
      Enter 'f'


bitrate = 4000000
Encoder Profile = High
Encoder control-rate = 1

Using winsys: x11
Opening in BLOCKING MODE
** Message: 13:59:03.681: <main:4734> iterating capture loop ....
NvMMLiteOpen : Block : BlockType = 4
===== NvVideo: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
GST_ARGUS: Creating output stream
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function initialize(), line 89)
(Argus) Error BadParameter:  (propagating from src/eglstream/FrameConsumerImpl.cpp, function create(), line 44)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadInitialize:320 Failed to create FrameConsumer
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, threadFunction:241 (propagating)
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, waitRunning:203 Invalid thread state 3
Error generated. /dvs/git/dirty/git-master_linux/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:806 (propagating)
^C** Message: 13:59:19.172: <_intr_handler:4309> User Interrupted..

Terminating the camera pipeline ...
** Message: 13:59:19.361: <main:4744> Capture completed
** Message: 13:59:19.381: <main:4795> Camera application will now exit

Despite this, when I try to start capturing with the pipeline I posted at the beginning, I am able to get the video stream but with the original issue.

Please confirm the sensor HW and driver by v4l2-ctl command.

https://docs.nvidia.com/jetson/archives/r36.3/DeveloperGuide/SD/CameraDevelopment/SensorSoftwareDriverProgramming.html#to-run-a-v4l2-ctl-test

I am not sure if I understand what you are asking, but I tried one of the tests from the link you provided. This was the output:

jetson@jetson-152:~$ v4l2-compliance -d /dev/video0
v4l2-compliance SHA: not available, 64 bits

Compliance test for tegra-video device /dev/video0:

Driver Info:
        Driver name      : tegra-video
        Card type        : vi-output, imx477 9-001a
        Bus info         : platform:tegra-capture-vi:0
        Driver version   : 5.10.120
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : tegra-camrtc-ca
        Model            : NVIDIA Tegra Video Input Device
        Serial           : 
        Bus info         : 
        Media version    : 5.10.120
        Hardware revision: 0x00000003 (3)
        Driver version   : 5.10.120
Interface Info:
        ID               : 0x0300000b
        Type             : V4L Video
Entity Info:
        ID               : 0x00000009 (9)
        Name             : vi-output, imx477 9-001a
        Function         : V4L2 I/O
        Pad 0x0100000a   : 0: Sink
          Link 0x0200000f: from remote pad 0x1000003 of entity '13e40000.host1x:nvcsi@15a00000-': Data, Enabled

Required ioctls:
        test MC information (see 'Media Driver Info' above): OK
        test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
        test second /dev/video0 open: OK
        test VIDIOC_QUERYCAP: OK
        test VIDIOC_G/S_PRIORITY: OK
        test for unlimited opens: OK

Debug ioctls:
        test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
        test VIDIOC_LOG_STATUS: OK

Input ioctls:
        test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
        test VIDIOC_ENUMAUDIO: OK (Not Supported)
        test VIDIOC_G/S/ENUMINPUT: OK
        test VIDIOC_G/S_AUDIO: OK (Not Supported)
        Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
        test VIDIOC_G/S_MODULATOR: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_ENUMAUDOUT: OK (Not Supported)
        test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDOUT: OK (Not Supported)
        Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
        test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
        test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
        test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
        test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls (Input 0):
        test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
        test VIDIOC_QUERYCTRL: OK
        test VIDIOC_G/S_CTRL: OK
        test VIDIOC_G/S/TRY_EXT_CTRLS: OK
        test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
        test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
        Standard Controls: 1 Private Controls: 20

Format ioctls (Input 0):
        test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
                fail: v4l2-test-formats.cpp(1280): ret && node->has_frmintervals
        test VIDIOC_G/S_PARM: FAIL
        test VIDIOC_G_FBUF: OK (Not Supported)
        test VIDIOC_G_FMT: OK
        test VIDIOC_TRY_FMT: OK
        test VIDIOC_S_FMT: OK
        test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
        test Cropping: OK (Not Supported)
        test Composing: OK (Not Supported)
        test Scaling: OK (Not Supported)

Codec ioctls (Input 0):
        test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
        test VIDIOC_G_ENC_INDEX: OK (Not Supported)
        test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls (Input 0):
                fail: v4l2-test-buffers.cpp(715): q.create_bufs(node, 1, &fmt) != EINVAL
        test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
        test VIDIOC_EXPBUF: OK
        test Requests: OK (Not Supported)

Total for tegra-video device /dev/video0: 45, Succeeded: 43, Failed: 2, Warnings: 0

I can observe that almost all the tests are okay, except for two errors at the end. However, I ran the same command on the other Jetson Orin NX I have, where the camera works, and the same error appeared. So, I assume that this log is not related to my issue.

Please check below command.

v4l2-ctl --set-fmt-video=width=1920,height=1080,pixelformat=RG12 --stream-mmap --set-ctrl=sensor_mode=0 --stream-count=100 -d /dev/video0