Gstreamer works at 4K but freeze the system at 2K

Robotics & Edge Computing Jetson & Embedded Systems Jetson Nano
24

@pietro.assenza

I just have a few follow-up questions regarding device tree and overall configuration.

From Auvidea JN30 TRM I could see that CSI connectors J10 and J5 are supporting 4 CSI lanes. Also, I2C communication on those connectors should be connected to CAM_I2C_SDA/CLK (215/213, i2c@546c0000 node in dtree) .

From your device tree, imx415 node is under i2c@7000c000 which you described as scl/sda gpios of I2C0_SDA/SCL (185,157). This I2C interface is connected to Auvidea JN30 connector J19 that supports only 2 CSI lanes(CSI-E).
Did you make modifications on the board?

If not, I think you should move imx415 node to i2c@546c0000 (the same node where you have nvtc358743 described).

Why did you move i2c@7000c000 node into host1x?

As I see the configuration, you should have sensors on J10 (CAM_I2C, CSI-A/B, port 0, serial_a) and J5 (CAM_I2C, CSI-C/D, port 2, serial_c). Is this correct?

25

@d.kristofic

Actually is J10 (I2C device 6, CSI-A/B, port 0, serial_a) and J5 (I2C device 0, CSI-C/D, port 2, serial_c);
there is no modification on the board.

I think there is an error in Auvidea user manual: please check the image at page 6, where you see that J5 and J10 are connected to different I2C, while in tables at page 8 and 9 they are linked to the same one.

BTW with our configuration of pair I2C/Port on the J5 we have gstreamer capturing 4K properly, so I’m confident the configuration is correct.
Could you share the significant part of your DTS, for comparison?

Thank you

26

@ShaneCCC
I tested with set_mode_delay_ms=“1000”, no changes I can see.
This is the log of nvargus-daemon:

=== gst-launch-1.0[6489]: Connection established (7F83CB01D0)Thread 1 getting next capture
Thread 1 is waiting
Thread 2 getting next capture
Thread 2 is waiting
Thread 3 getting next capture
Thread 3 is waiting
Thread 4 getting next capture
Thread 4 is waiting
Thread 5 getting next capture
Thread 5 is waiting
Thread 6 getting next capture
Thread 6 is waiting
Thread 7 getting next capture
Thread 7 is waiting
Thread 8 getting next capture
Thread 8 is waiting
Thread 9 getting next capture
Thread 9 is waiting
Thread 10 getting next capture
Thread 10 is waiting
Thread 11 getting next capture
Thread 11 is waiting
Thread 12 getting next capture
Thread 12 is waiting
Starting services...
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread IspHw statsComplete start
Worker thread IspHw frameComplete start
Worker thread CaptureScheduler checkFramePending start
Worker thread CaptureScheduler frameStart start
Worker thread V4L2CaptureScheduler checkCaptureComplete start
Worker thread V4L2CaptureScheduler issueCaptures start
Worker thread PS handleRequests start
getInstance: s_instance(0x7f7c77c290)
getInstance: s_instance(0x7f7c77c290)
subscribe: SensorType(gyroscope)
subscribe: create SensorType(gyroscope) sensor(0x7f7c77d640)
subscribe: SensorType(accelerometer)
subscribe: create SensorType(accelerometer) sensor(0x7f7c77d7e0)
AC plugin not present: dlopen "acplugin.so", acplugin.so: cannot open shared object file: No such file or directory
Services are started
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
=== gst-launch-1.0[6489]: CameraProvider initialized (0x7f7c82d220)sourceRegistry[0] assigned

ispRegistry[0] assigned

Using Source GUID 0
Worker thread ViCsiHw frameComplete start
Worker thread ViCsiHw frameStart start
Using ISP A
NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=2, m_value=4294967295, m_timeout_ms=1000

LSC: LSC surface is not based on full res!
AC plugin not present: dlopen "acplugin.so", acplugin.so: cannot open shared object file: No such file or directory
No library found, disabling AC plugin.
Worker thread CaptureDispatcher start
Session::updatePerfHints()
updatePerfHints(): guid=0ll, CaptureIntent=1
updatePerfHints(): NvCameraCoreUseCase_Preview
NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=1, m_value=30, m_timeout_ms=1000

NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=15, m_value=0, m_timeout_ms=1000

No output buffers for 2
InstructionList:
  + GraphSettings
      | SensorMode: 1944x1096 BayerS16GBRG 30.0fps
      | output 0: 1944x1096 BL U8_V8_ER 420SP
      | inputClipRect 0: (0.00,0.00, 1.00,1.00)

  + Instruction List
      | id: 0
      +  0: CCDataSetupStage
          | EstimatedIspOutLatencyFrames: 5
          | NumConcurrentCaptures: 1
          | UnprocessedYuvBufferMask: 0
      +  1: ACSynchronizeStage
      +  2: AeAfApplyStage
      +  3: AcPluginStage
          | operation: opApply
      +  4: AcMergeStage
          | IspIn: [1944, 1096]
          | IspOut0: [1944, 1096]
          | IspOut1: [0, 0]
          | IspOut2: [0, 0]
      +  5: TempBufferAcquireStage
          | Buffer Index: 2
          | BufferRequirements: 1944x1 Pitch NonColor8
      +  6: TempBufferAcquireStage
          | Buffer Index: 3
          | BufferRequirements: 524288x1 Pitch NonColor8
      +  7: SensorISPCaptureStage
          | Source GUID: 0
          | Output A Buffer: 0
          | SensorMetadata Buffer: 2
          | Output IspStats Buffer: 3
      +  8: StatsBufferAcquireStage
          | Buffer Index: 1
          | BufferRequirements: 640x362 Pitch Y8_ER 420
      +  9: BlitStage
          | Input Buffer: 0
          | Output Buffer: 1
          | Filter: Nearest
          | Transform: None
          | Src Rect: Not used
          | Dst Rect: Not used
      + 10: StatsUpdateStage
          | Output Meta Buffer: 2
          | Output IspStats Buffer: 3
      + 11: BufferReturnStage
          | Output A Buffer: 2
      + 12: BufferReturnStage
          | Output A Buffer: 3
      + 13: AcPluginStage
          | operation: opAnalyze
      + 14: AfAnalysisStage
      + 15: MonitorStage
      + 16: ExifStage
      + 17: MakerNoteStage
      + 18: BufferReturnStage
          | Output A Buffer: 0
      + 19: MetadataReturnStage
      + 20: PerfStatsStage

Created fiber 0x7f2c000b20 for CC 101 globalID 101 session 0
Session::updatePerfHints()
updatePerfHints(): guid=0ll, CaptureIntent=1
updatePerfHints(): NvCameraCoreUseCase_Preview
Thread 1 is working on CC 101 session 0 globalID 101 step 0
CC 101 session 0 completed step 0 in fiber 0x7f2c000b20
cc 101(1) session 0 runCount=0 runIspOut=0, latest ccId=0
CC 101 session 0 completed step 1 in fiber 0x7f2c000b20
NV AE and AfApply algorithms are active.
NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=1, m_value=30, m_timeout_ms=1000

NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=15, m_value=0, m_timeout_ms=1000

CC 101 session 0 completed step 2 in fiber 0x7f2c000b20
CC 101 session 0 completed step 3 in fiber 0x7f2c000b20
CC 101 session 0 completed step 4 in fiber 0x7f2c000b20
CC 101 session 0 completed step 5 in fiber 0x7f2c000b20
Created fiber 0x7f2c000dd0 for CC 102 globalID 102 session 0Session::updatePerfHints()
Thread 2 is working on CC 102 session 0 globalID 102 step 0
CC 102 session 0 completed step 0 in fiber 0x7f2c000dd0
cc 102(2) session 0 runCount=1 runIspOut=0, latest ccId=0
CC 102 session 0 completed step 1 in fiber 0x7f2c000dd0
NV AE and AfApply algorithms are active.
CC 102 session 0 completed step 2 in fiber 0x7f2c000dd0
CC 102 session 0 completed step 3 in fiber 0x7f2c000dd0
CC 102 session 0 completed step 4 in fiber 0x7f2c000dd0
CC 102 session 0 completed step 5 in fiber 0x7f2c000dd0
updatePerfHints(): guid=0ll, CaptureIntent=1

updatePerfHints(): NvCameraCoreUseCase_Preview
NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=1, m_value=30, m_timeout_ms=1000

NvPHSSendThroughputHints: sensorId=0, m_usecase=4, m_type=15, m_value=0, m_timeout_ms=1000

CC 101 session 0 completed step 6 in fiber 0x7f2c000b20
CC 101 session 0 processing step 7 in fiber 0x7f2c000b20
FiberScheduler: cc 101, session 0, fiber 0x7f2c000b20 in progress...

Thread 1 getting next capture
Thread 1 is waiting
Thread 3 is waiting
CC 102 session 0 completed step 6 in fiber 0x7f2c000dd0
CC 102 session 0 processing step 7 in fiber 0x7f2c000dd0
FiberScheduler: cc 102, session 0, fiber 0x7f2c000dd0 in progress...

Thread 2 getting next capture
Thread 2 is waiting
Thread 4 is waiting
PowerServiceHw:addRequest: table size: before: 0, after:1
        request table for ISP 0:
        req[0]: output1 width=0, output1 height=0, output1 Bpp=0
        req[0]: output2 width=0, output2 height=0, output2 Bpp=0
        req[0]: input width=1944, input height=1096, input Bpp=12
        req[0]: output width=130, output height=74, output Bpp=16
        req[0]: input fps=30
        req[0]: guID=0, stage type=SensorIspCapture
        req[0]: clock=0 Hz, iso bw=54151 KB/s, non-iso bw=0 KB/s
        req[450]: timeout=0
PowerServiceHwIsp:setLaBw: m_bwVal_Iso=54151 and m_bwVal_NonIso=0 KBpS

PowerServiceCore:setCameraBw: totalIsoBw=54151
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
Created fiber 0x7f2c000f70 for CC 103 globalID 103 session 0
Thread 5 is working on CC 103 session 0 globalID 103 step 0
Fiber 0x7f2c000f70 is aborting in CC 103 Session 0
FiberScheduler: cc 103 session 0, fiber 0x7f2c000f70 aborted

FiberScheduler: fiber 0x7f2c000f70 exiting
Thread 5 getting next capture
Thread 7 is waiting
Thread 5 is waiting
Thread 6 is waiting
PowerServiceHw:addRequest: table size: before: 1, after:2
        request table for ISP 0:
        req[0]: output1 width=0, output1 height=0, output1 Bpp=0
        req[0]: output2 width=0, output2 height=0, output2 Bpp=0
        req[0]: input width=1944, input height=1096, input Bpp=12
        req[0]: output width=130, output height=74, output Bpp=16
        req[0]: input fps=30
        req[0]: guID=0, stage type=SensorIspCapture
        req[0]: clock=0 Hz, iso bw=54151 KB/s, non-iso bw=0 KB/s
        req[388]: timeout=0
        req[1]: output1 width=0, output1 height=0, output1 Bpp=0
        req[1]: output2 width=0, output2 height=0, output2 Bpp=0
        req[1]: input width=1944, input height=1096, input Bpp=12
        req[1]: output width=1944, output height=1096, output Bpp=12
        req[1]: input fps=30
        req[1]: guID=0, stage type=SensorIspCapture
        req[1]: clock=0 Hz, iso bw=595438 KB/s, non-iso bw=0 KB/s
        req[450]: timeout=0
PowerServiceHw:updateRequests: table size: before: 2, after:2
        request table for ISP 0:
        req[0]: output1 width=0, output1 height=0, output1 Bpp=0
        req[0]: output2 width=0, output2 height=0, output2 Bpp=0
        req[0]: input width=1944, input height=1096, input Bpp=12
        req[0]: output width=130, output height=74, output Bpp=16
        req[0]: input fps=30
        req[0]: guID=0, stage type=SensorIspCapture
        req[0]: clock=0 Hz, iso bw=54151 KB/s, non-iso bw=0 KB/s
        req[385]: timeout=0
        req[1]: output1 width=0, output1 height=0, output1 Bpp=0
        req[1]: output2 width=0, output2 height=0, output2 Bpp=0
        req[1]: input width=1944, input height=1096, input Bpp=12
        req[1]: output width=1944, output height=1096, output Bpp=12
        req[1]: input fps=30
        req[1]: guID=0, stage type=SensorIspCapture
        req[1]: clock=0 Hz, iso bw=595438 KB/s, non-iso bw=0 KB/s
        req[447]: timeout=0
PowerServiceHwIsp:setLaBw: m_bwVal_Iso=595438 and m_bwVal_NonIso=0 KBpS

PowerServiceCore:setCameraBw: totalIsoBw=595438
SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 478)
packet_write_wait:

attached the “dmesg -n 8” on serial with lot of log: start capture, freeze (around 240), watchdog, dump and finally reboot.
dmesg-gst-2k-long.txt (37.3 KB)

27

@pietro.assenza
Yes, i can see it.

I think there is an error in Auvidea user manual: please check the image at page 6, where you see that J5 and J10 are connected to different I2C, while in tables at page 8 and 9 they are linked to the same one.

Attached you will find dtree for your configuration(not tested).

Could you share the significant part of your DTS, for comparison?

auvidea.txt (265.2 KB)

28

@sme21ab
after hw analysis of your schematics, it seams that hw is not the issue for configuration bellow 1.5Gbps/lane

Note: we tested the system on our custom carrier board, not Auvidea JN30B.

Could you send the modified driver/regmap of imx415 image sensor to check if this could be the issue?
Do you have any progress so far?

Thanks

29

Hello @d.kristofic
thanks for your review; yes, we are having good result right now.

By comparing with your DTS I confirmed that our configuration is correct, so I went back to code inspection and I found a big mistake: while the driver is configured for 2K, it still sends 4K settings to the sensor.
Result is a weird 2K frame, sort of cropped one, and while V4L still manage to handle it gstremer+nvargus doesn’t and freezes; I still don’t know why nvargus doesn’t discard it or refuse to sync, but fixed it gstreamer doesn’t freeze anymore.

Right now we have a gstreamer streaming 2K30fps, it seems stable and I’m confident to soon push the frame-rate up to 60 as required.
Hope to be near to the solution.

2 Likes