FPGA / NVCSI / RCE / VI fail to recover video stream after (fast) camera disconnect + reconnect

Robotics & Edge Computing Jetson Systems Jetson Xavier NX
,
1

I am working on a video processing pipeline for the Jetson Xavier NX as part of a larger device. It receives the video signal via an MIPI TX CSI-2 interface, but when the camera signal is briefly interrupted, e.g., by (physically) disconnecting and reconnecting the camera, the signal does not reach the GPU any more. Even rebooting the GPU from the console does not fix it, only turning the power off and on works.

I tried to investigate this and learned that rebooting the GPU does not also reboot the NVCSI / VI because they are separate hardware, is that correct? Is there a way to do this from the GPU?

This low-level and hardware architecture is currently pretty much beyond my understanding, but I believe it looks something like this:

FPGA → CSI-2 → NVCSI → RCE → VI → my video processing pipeline (V4L2 → DMABUF, VIC, CUDA, EGL).

Note that the error seems not to be caused by my pipeline, since I can also confirm it by just using V4L2 from the terminal, e.g. via v4l2-ctl --stream-mmap --stream-count=1 -d /dev/video0. Running dmesg then shows (the errors only appear once V4L2 tries to access the video stream, otherwise the system seems to fail “silently”, i.e., so far, I could not find any signs of it before trying to access the video stream):

...
[  494.826473] [fpga] fpga_power_on called
[  494.826512] [fpga] fpga_power_on set mclk to 24000000 Hz
[  497.376067] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  497.376103] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  497.376167] (NULL device *): vi_capture_control_message: NULL VI channel received
[  497.376185] t194-nvcsi 13e10000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  497.376223] (NULL device *): vi_capture_control_message: NULL VI channel received
[  497.376656] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel
[  497.432060] [RCE] ERROR: camera-ip/vi5/vi5.c:3913 [vi5_update_channel_match_data] "match configuration is already in use by by channel 35 cannot set channel match configuration"
[  497.432071] [RCE] ERROR: services/capture/capture-scheduler.c:3024 [capture_scheduler_thread] "Error processing event 0x100003"
[  499.921875] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  499.921911] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  499.921963] (NULL device *): vi_capture_control_message: NULL VI channel received
[  499.921980] t194-nvcsi 13e10000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=0, csi_port=0
[  499.921995] (NULL device *): vi_capture_control_message: NULL VI channel received
[  499.922033] t194-nvcsi 13e10000.host1x:nvcsi@15a00000: csi5_stream_open: VI channel not found for stream- 0 vc- 0
...

So the error seems to occur somewhere in the FPGA / NVCSI / RCE / VI section. So far, I wasn’t able to narrow it down any further.

Some additional information that may be helpful

media-ctl -p provides:

Media controller API version 5.10.192

Media device information
------------------------
driver          tegra-camrtc-ca
model           NVIDIA Tegra Video Input Device
serial
bus info
hw revision     0x3
driver version  5.10.192

Device topology
- entity 1: 13e10000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev0
        pad0: Sink
                <- "fpga 9-0032":0 [ENABLED]
        pad1: Source
                -> "vi-output, fpga 9-0032":0 [ENABLED]

- entity 4: 13e10000.host1x:nvcsi@15a00000- (2 pads, 2 links)
            type V4L2 subdev subtype Unknown flags 0
            device node name /dev/v4l-subdev1
        pad0: Sink
                <- "fpga 10-0036":0 [ENABLED]
        pad1: Source
                -> "vi-output, fpga 10-0036":0 [ENABLED]

- entity 7: fpga 9-0032 (1 pad, 1 link)
            type V4L2 subdev subtype Sensor flags 0
            device node name /dev/v4l-subdev2
        pad0: Source
                [fmt:UYVY10_1X20/1920x1080 field:none colorspace:srgb]
                -> "13e10000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 9: vi-output, fpga 9-0032 (1 pad, 1 link)
            type Node subtype V4L flags 0
            device node name /dev/video0
        pad0: Sink
                <- "13e10000.host1x:nvcsi@15a00000-":1 [ENABLED]

- entity 23: fpga 10-0036 (1 pad, 1 link)
             type V4L2 subdev subtype Sensor flags 0
             device node name /dev/v4l-subdev3
        pad0: Source
                [fmt:UYVY10_1X20/1920x1080 field:none colorspace:srgb]
                -> "13e10000.host1x:nvcsi@15a00000-":0 [ENABLED]

- entity 25: vi-output, fpga 10-0036 (1 pad, 1 link)
             type Node subtype V4L flags 0
             device node name /dev/video1
        pad0: Sink
                <- "13e10000.host1x:nvcsi@15a00000-":1 [ENABLED]

v4l2-ctl --all -d /dev/video0 provides:

Driver Info:
        Driver name      : tegra-video
        Card type        : vi-output, fpga 9-0032
        Bus info         : platform:tegra-capture-vi:0
        Driver version   : 5.10.192
        Capabilities     : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps      : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format
Media Driver Info:
        Driver name      : tegra-camrtc-ca
        Model            : NVIDIA Tegra Video Input Device
        Serial           :
        Bus info         :
        Media version    : 5.10.192
        Hardware revision: 0x00000003 (3)
        Driver version   : 5.10.192
Interface Info:
        ID               : 0x0300000b
        Type             : V4L Video
Entity Info:
        ID               : 0x00000009 (9)
        Name             : vi-output, fpga 9-0032
        Function         : V4L2 I/O
        Pad 0x0100000a   : 0: Sink
          Link 0x0200000f: from remote pad 0x1000003 of entity '13e10000.host1x:nvcsi@15a00000-': Data, Enabled
Priority: 2
Video input : 0 (Camera 0: ok)
Format Video Capture:
        Width/Height      : 1920/1080
        Pixel Format      : 'UY20' (UYVY 4:2:2 10 bit)
        Field             : None
        Bytes per Line    : 3840
        Size Image        : 4147200
        Colorspace        : sRGB
        Transfer Function : Default (maps to sRGB)
        YCbCr/HSV Encoding: Default (maps to ITU-R 601)
        Quantization      : Default (maps to Full Range)
        Flags             :

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                     hdr_enable 0x009a2004 (intmenu): min=0 max=1 default=0 value=0
                                0: 0 (0x0)
                                1: 1 (0x1)
                           gain 0x009a2009 (int64)  : min=16 max=171 step=1 default=16 value=16 flags=slider
                       exposure 0x009a200a (int64)  : min=13 max=683710 step=1 default=2495 value=13 flags=slider
                     frame_rate 0x009a200b (int64)  : min=2000000 max=60000000 step=1 default=60000000 value=2000000 flags=slider
                 exposure_short 0x009a200c (int64)  : min=13 max=683709 step=1 default=2495 value=13 flags=slider
           sensor_configuration 0x009a2032 (u32)    : min=0 max=4294967295 step=1 default=0 [22] flags=read-only, volatile, has-payload
         sensor_mode_i2c_packet 0x009a2033 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
      sensor_control_i2c_packet 0x009a2034 (u32)    : min=0 max=4294967295 step=1 default=0 [1026] flags=read-only, volatile, has-payload
                    bypass_mode 0x009a2064 (intmenu): min=0 max=1 default=0 value=0
                                0: 0 (0x0)
                                1: 1 (0x1)
                override_enable 0x009a2065 (intmenu): min=0 max=1 default=0 value=0
                                0: 0 (0x0)
                                1: 1 (0x1)
                   height_align 0x009a2066 (int)    : min=1 max=16 step=1 default=1 value=1
                     size_align 0x009a2067 (intmenu): min=0 max=2 default=0 value=0
                                0: 1 (0x1)
                                1: 65536 (0x10000)
                                2: 131072 (0x20000)
               write_isp_format 0x009a2068 (int)    : min=1 max=1 step=1 default=1 value=1
       sensor_signal_properties 0x009a2069 (u32)    : min=0 max=4294967295 step=1 default=0 [30][18] flags=read-only, has-payload
        sensor_image_properties 0x009a206a (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
      sensor_control_properties 0x009a206b (u32)    : min=0 max=4294967295 step=1 default=0 [30][36] flags=read-only, has-payload
              sensor_dv_timings 0x009a206c (u32)    : min=0 max=4294967295 step=1 default=0 [30][16] flags=read-only, has-payload
               low_latency_mode 0x009a206d (bool)   : default=0 value=0
               preferred_stride 0x009a206e (int)    : min=0 max=65535 step=1 default=0 value=0
                   sensor_modes 0x009a2082 (int)    : min=0 max=30 step=1 default=30 value=4 flags=read-only

v4l2-ctl -d /dev/v4l-subdev2 --all provides:

Camera Controls

                     group_hold 0x009a2003 (bool)   : default=0 value=0 flags=execute-on-write
                     hdr_enable 0x009a2004 (intmenu): min=0 max=1 default=0 value=0
                                0: 0 (0x0)
                                1: 1 (0x1)
                           gain 0x009a2009 (int64)  : min=16 max=171 step=1 default=16 value=16 flags=slider
                       exposure 0x009a200a (int64)  : min=13 max=683710 step=1 default=2495 value=13 flags=slider
                     frame_rate 0x009a200b (int64)  : min=2000000 max=60000000 step=1 default=60000000 value=2000000 flags=slider
                 exposure_short 0x009a200c (int64)  : min=13 max=683709 step=1 default=2495 value=13 flags=slider
3

hello mapf,

may I know which Jetpack release version you’re working with?

just an FYI, this scenario is not a default supported camera use-case.

according to VI logs..
it’s attempting to reset the capture channel, but there’s NULL VI channel received in the end.
it means a video node register to linux kernel has gone.

4

Hi Jerry, thank you!

I am using Jetpack version 5.1.3. Will that be an issue?

Ah that’s fair. Good to know. Basically, we would like to make the system as ruggedized as possible, so it should be able to handle if a camera disconnects or has a loose connection (or even hot-swapping of a camera).

I see. Can this be recovered?

I just did some more testing and found that when I disconnect the camera, I get the following on repeat:

[  185.135897] [fpga] fpga_power_on called
[  185.135910] [fpga] fpga_power_on set mclk to 24000000 Hz
[  187.649427] tegra-camrtc-capture-vi tegra-capture-vi: uncorr_err: request timed out after 2500 ms
[  187.649458] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: attempting to reset the capture channel
[  187.649513] (NULL device *): vi_capture_control_message: NULL VI channel received
[  187.649533] t194-nvcsi 13e10000.host1x:nvcsi@15a00000: csi5_stream_close: Error in closing stream_id=4, csi_port=4
[  187.649548] (NULL device *): vi_capture_control_message: NULL VI channel received
[  187.650089] tegra-camrtc-capture-vi tegra-capture-vi: err_rec: successfully reset the capture channel

And if I then reconnect the camera after the first iteration, the signal always seems to be recovered fine. But when I reconnect the camera earlier, the signal recovery fails, and I get the following message once:

[  423.654873] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 64

The previous messages then repeat as before. So it seems these 2500ms are important.

5

Ok. I did some more testing and I revert part my previous message. There seem to be very specific timings during the disconnect-phase when it is ok to reconnect the camera and when it’s not. I.e., I can actually reconnect the camera before the first error message cycle, and instead get this message:

[  713.938359] tegra-camrtc-capture-vi tegra-capture-vi: corr_err: discarding frame 0, flags: 0, err_data 64

And the signal is recovered. Still, if I reconnect too fast however, the signal recovery seems to reliably fail.

Similarly, I can reconnect the camera after the first cycle and the signal recovery will still fail. Seemingly reliably if I reconnect the camera just before or after the error messages are sent. So I wonder what is happening under the hood during this recovery process, because it seems that the recovery loop that is running has certain periods/sections when it’s ok to reconnect the camera and when it’s not.

Another thing that’s possibly worth mentioning: I can also cause the system to fail “silently” by disconnecting and reconnecting the camera after a fresh restart but before starting the video processing application. In that case, the error messages only start to appear once the application is started and tries to access the video stream.

6

hello mapf,

it happened when the FPGA CSI signal disappears or becomes invalid, VI has timed out (i.e. 2500ms by default), VI error recovery mechanism has called.
you’ll also see corr_err and uncorr_err status, which means a recoverable bad-frame status, and later unrecoverable.
CSI recovery tries to stop and restart the NVCSI stream, however, CSI close/open still needs a valid VI channel handle to send control messages to RCE. that’s why reconnect timing has different state, and that’s why NULL VI channel received to damage recovery mechanism.

anyways..
here’re possible workarounds..

  1. Avoid fast CSI restart from the FPGA.
    add debounce on the FPGA/camera side. after signal loss, do not restart CSI immediately; wait until the source is stable for longer than the VI timeout/recovery window, for example 3-5 seconds, then restart clean CSI output.
  2. Keep CSI alive with dummy frames
    when the upstream camera is loose/disconnected, have the FPGA continue sending a valid CSI-2 stream, such as blank/test-pattern frames. that should prevents NVCSI/VI/RCE from entering timeout recovery at all.
7

Hi Jerry,

thanks a lot for the suggestions! They should be feasible.

I was wondering though, would increasing/decreasing the VI timeout period help somehow?

8

hello mapf,

maybe, but that’s not address the root cause.
such 2500ms VI timeout controls when vi_capture_status() gives up waiting.

let’s try adding timeout value to 5-sec or more, if the failure window moves or disappears for short interruptions, then timeout tuning can be a workaround.
however, if it still wedges once recovery begins, the real fix is to avoid entering recovery. such as.. debounce FPGA CSI restart, keep sending dummy valid frames, or stop V4L2 before the signal is removed.

9

Hi Jerry,

thank you! I will give this a try.

11

Hi, so we considered potential solutions but realized that sending a dummy frame in case the signal is lost may not solve the issue if it is caused by hanging MIPI cores on the FPGA and/or Jetson due to incomplete/interrupted frames. So we were wondering if there is a way to restart the MIPI core on the Jetson since a simple reboot of the GPU does not seem to affect it.

12

hello mapf,

it’s by default to have auto-suspend enabled.
please give it a try to disable RCE suspend (rtcpu auto-suspend) for testing.
# echo -1 > /sys/devices/platform/*rtcpu/power/autosuspend_delay_ms
# echo on > /sys/devices/platform/*rtcpu/power/control

13

Hi,

thank you, I will test this. In the meantime, we found that the MIPI core on the FPGA side fails to recover after a video signal is lost/reconnected and may require a hard reset. This might already be the root cause.