Xavier AGX Synchronized Capture with 4 Cameras

I have tried scaling down all the way to 692 × 520, but still have the same issue: all 4 files have the image from camera #0.

This is the complete output of the execution:

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree

---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again
CAM: serial no file already exists, skips storing againClipHelper

allowIspClipping: true

maxIspDownscale: 4.0:1 4096

maxIspOutWidth: 6144,4096

ispIn: (4056 x 3040)

PRU enabled: false, interleaved input: (0 x 0)

postProcessingSize: (1014 x 760)

postIspClip: (0.00,0.00, 1.00,1.00)

ispOut[0]: (1014 x 760)

ispClip[0]: (0.00,0.00, 1.00,1.00)

ispOut[1]: (0 x 0)

ispClip[1]: (0.00,0.00, 1.00,1.00)

out[0] 692x520 req (0.00,0.00, 1.00,1.00) final (0.00,0.00, 1.00,1.00) isp from isp[0]

StageGroup 0x7f48000d00 parent=(nil) 4056x3040 (1 exposure) obufMask=f finalMask=0

stages[0] = 35 SensorCaptureStage(in = 12, outA= 7, outB = 12, outThumb = 12, outMeta = 8, outStats = 12) routed

StageGroup 0x7f480018d0 parent=0x7f48000d00 1014x760 (1 exposure) obufMask=f finalMask=f

stages[0] = 27 MemoryToISPCaptureStage(in = 7, outA= 5, outB = 12, outThumb = 4, outMeta = 12, outStats = 6) routed

m_bufStates[0] = 0 attached output done readOrder=0 writeOrder=2 group=0x7f48000d00 fbs=none

692x520 BL U8_V8_ER 420SP

m_bufStates[1] = 1 attached output done readOrder=0 writeOrder=2 group=0x7f48000d00 fbs=none

692x520 BL U8_V8_ER 420SP

m_bufStates[2] = 2 attached output done readOrder=0 writeOrder=2 group=0x7f48000d00 fbs=none

692x520 BL U8_V8_ER 420SP

m_bufStates[3] = 3 attached output done readOrder=0 writeOrder=2 group=0x7f48000d00 fbs=none

692x520 BL U8_V8_ER 420SP

m_bufStates[4] = 4 attached readOrder=0 writeOrder=2 group=0x7f480018d0 AF fbs=none

640x480 Pitch U8_V8_ER 420SP

m_bufStates[5] = 5 attached readOrder=0 writeOrder=2 group=0x7f480018d0 fbs=isp0

1014x760 BL U8_V8_ER 420SP

m_bufStates[6] = 6 readOrder=0 writeOrder=2 group=0x7f480018d0 fbs=none

524288x1 Pitch NonColor8

m_bufStates[7] = 7 readOrder=1 writeOrder=1 group=0x7f48000d00 fbs=none

4056x3040 Pitch BayerS16RGGB

m_bufStates[8] = 8 readOrder=0 writeOrder=1 group=0x7f48000d00 fbs=none

4056x1 Pitch NonColor8

GraphHelper blit pixel count=4521920 != ClipHelper blit pixel count=1130480

(Argus) Objects still active during exit: [CameraProvider (0x558f16cf00): refs: 1, cref: 0]

Argus Version: 0.97.3 (single-process)

Camera count: 4

DONE!

Could you attached the binary and the source code to verify.

Also, I’m not rendering any previews, please refer to the code attached in my initial post.

With gstreamer, I have been able to stream from all 4 cameras concurrently without issues, although not at 4k full resolution:

gst-launch-1.0
nvarguscamerasrc sensor-id=2 ! “video/x-raw(memory:NVMM),width=960,height=540,format=(string)NV12,framerate=30/1” ! c.sink_0
nvarguscamerasrc sensor-id=3 ! “video/x-raw(memory:NVMM),width=960,height=540,format=(string)NV12,framerate=30/1” ! c.sink_1
nvarguscamerasrc sensor-id=0 ! “video/x-raw(memory:NVMM),width=960,height=540,format=(string)NV12,framerate=30/1” ! c.sink_2
nvarguscamerasrc sensor-id=1 ! “video/x-raw(memory:NVMM),width=960,height=540,format=(string)NV12,framerate=30/1” ! c.sink_3
nvcompositor name=c
sink_0::width=960 sink_0::height=540 sink_0::xpos=0 sink_0::ypos=0
sink_1::width=960 sink_1::height=540 sink_1::xpos=960 sink_1::ypos=0
sink_2::width=960 sink_2::height=540 sink_2::xpos=0 sink_2::ypos=540
sink_3::width=960 sink_3::height=540 sink_3::xpos=960 sink_3::ypos=540
! nvoverlaysink
-e

@ShaneCCC this is the 3rd time I’m saying it, the code is attached to my first post in this thread. :)

Thanks for your confirm. Could you able to attached the binary here.

Thanks

Sure, see attached.

There are two binaries:
sync4cams - four cameras in session: doesn’t work, all four outputs are from camera #0
sync2cams - two cameras in session: seems OK

Cameras are configured to mode 5, which is 692 × 520 resolution on my sensors.

sync2cams (86.1 KB) sync4cams (85.7 KB)

Can’t run these two binary on r32.4.3
What’s the BSP version? cat /etc/nv_tegra_release

Also listed in my first post… ;)

You can also very easily compile the source from my first post with:

c++ -I/usr/src/jetson_multimedia_api/include -L/usr/lib/aarch64-linux-gnu/tegra -o sync4cams sync4cams.cpp -lnvargus

Hi andy36
Current version only can support to 3 cameras for single session.
Please confirm it 3 cameras worked.

If I put #0, #1, and #2 in a session, it fails with an error message about unsupported output buffer format (see below) for some resolutions, but works OK for others.

E.g., 4056 × 3040 works OK, but 3840x2160 or 692x520 both fail.

This is output when it fails:

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree

---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

CAM: serial no file already exists, skips storing again
CAM: serial no file already exists, skips storing again
SCF: Error NotSupported: Output buffer format not supported: 692x520 BL U8_V8_ER 420SP (in src/components/GraphHelper.cpp, function findBlitSource(), line 1449)

SCF: Error NotSupported: (propagating from src/components/GraphHelper.cpp, function addScalingBlit(), line 2025)

SCF: Error NotSupported: (propagating from src/components/GraphHelper.cpp, function addOutputScalingBlits(), line 1984)

SCF: Error NotSupported: (propagating from src/components/CaptureSetupEngineImpl.cpp, function genInstructionsCoordinatedCamera(), line 1626)

SCF: Error NotSupported: (propagating from src/components/CaptureSetupEngineImpl.cpp, function doGetInstructions(), line 2211)

SCF: Error NotSupported: (propagating from src/components/CaptureSetupEngine.cpp, function getInstructionList(), line 300)

SCF: Error NotSupported: (propagating from src/components/CaptureSetupEngine.cpp, function setupCC(), line 214)

SCF: Error NotSupported: (propagating from src/api/Session.cpp, function capture(), line 815)

(Argus) Error NotSupported: (propagating from src/api/ScfCaptureThread.cpp, function run(), line 109)

Failed to get IFrame interface #0

Argus Version: 0.97.3 (single-process)

Camera count: 3

(Argus) Objects still active during exit: [CameraProvider (0x558a399f00): refs: 1, cref: 0]

Looks like we will be able to get hw sync working soon with LI’s help, but we still need to find a way to handle auto-correction. The same auto-correction values (AWB, AGC, etc.) need to be applied to all 4 cameras simultaneously, otherwise the images will look different, which is a big problem for stitching or stereo vision.

According to the syncSensor code, if there are multiple cameras in the same capture session, the same AC is applied to all of them calculated from the first device. This would be the easiest solution, but as discussed above, we can only add 3 devices now, adding the 4th doesn’t work. How do we solve this?

  1. What is necessary to support 4 cameras per capture session? Can we fix this on our side? If not, when can NVIDIA provide a fix?
  2. If there is some hardware limitation that makes it impossible to have 4 cameras per session, how do we make sure auto-correction parameters are always in sync (for every frame!) between multiple sessions?
  1. Can’t fix by any partner or user. Need nvidia help to fixed it.
  2. Please make sure both of 4 camera can capture the same result individually with the same setting(Gain/Expoure/FrameRate)
  1. Please provide a timeframe for this

  2. They are exactly the same hardware, so they can product the same result, it’s the matter of the ISP controlling them at exactly the same time.

Do you mean those 4 cameras get the same result by worked individually but get different result by single session per multiple source?

If we manage the cameras in different sessions, the AC algorithms might adjust them differently which is a problem, because when you put two images together, the color on them will not match exactly. See the attached example (taken from this article)

The only way to avoid this error is to make sure the AC algorithm sets the same parameters for all of the cameras and at the same time.
As I said earlier, this is automatically handled when they are all in a the same session, but if that is not possible, we need another way to handle “syncing” AC parameters between capture sessions.

@ShaneCCC , @JerryChang any update here? We need a resolution to this issue.

To reiterate:

  1. Need to be able to put 4 cameras in a session; OR
  2. Need a way to perfectly sync auto-correction between multiple sessions

Replace the attached lib at /usr/lib/aarch64-xxxx/tegra/ for the 4 cameras in one session.
libnvscf_32.4.so.txt (8.2 MB) libnvargus_32.4.so.txt (951.7 KB)

Replaced libnvscf.so and libnvargus.so in /usr/lib/aarch64-linux-gnu/tegra, rebooted the OS, then tried to run my code, but four cameras still don’t work… The issue is the same, all four captures return the same image.

There is a lot of output when the code runs:

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2

OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

NvPclHwGetModuleList: WARNING: Could not map module to ISP config string

NvPclHwGetModuleList: No module data found

OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree

---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----

GDC: setparams: m_power 1.000000

GDC: setparams: m_power 1.000000

MidtoneMeter Points[0] {0.000000, 1.000000}

MidtoneMeter Points[1] {0.500000, 1.000000}

MidtoneMeter Points[2] {0.800000, 0.000000}

Modified BiasTable: lux 100.000000 MidtoneDefault 0.000000 and 0.700000, 1.200000

Modified BiasTable: lux 600.000000 MidtoneDefault 0.300000 and 0.700000, 1.500000

ClipHelper

allowIspClipping: true

maxIspDownscale: 4.0:1 4096

maxIspOutWidth: 6144,4096

ispIn: (4056 x 3040)

PRU enabled: false, interleaved input: (0 x 0)

postProcessingSize: (4056 x 3040)

postIspClip: (0.00,0.00, 1.00,1.00)

ispOut[0]: (4056 x 3040)

ispClip[0]: (0.00,0.00, 1.00,1.00)

ispOut[1]: (0 x 0)

ispClip[1]: (0.00,0.00, 1.00,1.00)

out[0] 4056x3040 req (0.00,0.00, 1.00,1.00) final (0.00,0.00, 1.00,1.00) isp from isp[0]

StageGroup 0x7f58000d00 parent=(nil) 4056x3040 (1 exposure) obufMask=f finalMask=0

stages[0] = 35 SensorCaptureStage(in = 12, outA= 6, outB = 12, outThumb = 12, outMeta = 7, outStats = 12) routed

StageGroup 0x7f580014b0 parent=0x7f58000d00 4056x3040 (1 exposure) obufMask=f finalMask=f

stages[0] = 27 MemoryToISPCaptureStage(in = 6, outA= 0, outB = 12, outThumb = 4, outMeta = 12, outStats = 5) routed

m_bufStates[0] = 0 attached output done readOrder=0 writeOrder=2 group=0x7f580014b0 fbs=isp0

4056x3040 BL U8_V8_ER 420SP

m_bufStates[1] = 1 attached output done readOrder=0 writeOrder=2 group=0x7f58000d00 fbs=none

4056x3040 BL U8_V8_ER 420SP

m_bufStates[2] = 2 attached output done readOrder=0 writeOrder=2 group=0x7f58000d00 fbs=none

4056x3040 BL U8_V8_ER 420SP

m_bufStates[3] = 3 attached output done readOrder=0 writeOrder=2 group=0x7f58000d00 fbs=none

4056x3040 BL U8_V8_ER 420SP

m_bufStates[4] = 4 attached readOrder=0 writeOrder=2 group=0x7f580014b0 AF fbs=none

640x480 Pitch U8_V8_ER 420SP

m_bufStates[5] = 5 readOrder=0 writeOrder=2 group=0x7f580014b0 fbs=none

524288x1 Pitch NonColor8

m_bufStates[6] = 6 readOrder=1 writeOrder=1 group=0x7f58000d00 fbs=none

4056x3040 Pitch BayerS16RGGB

m_bufStates[7] = 7 readOrder=0 writeOrder=1 group=0x7f58000d00 fbs=none

4056x1 Pitch NonColor8

GraphHelper blit pixel count=73981440 != ClipHelper blit pixel count=0

GetRunningInfo ++ ET Range {0.00002, 0.16577} dre {1.000000, 1.000000} GainRane {1.00000, 354.00000}, CurrentFrameRate 0.000, FrameRateRange{1.46253, 30.00000}, ReadOutTime 0.000 PclDynData Aperture step 0

LimitCalc: higlight limits low 0.000003 high 2608.114502 and input 0.005787

runBreakDownAlgorithm output —: convergenceStatus: 1, framerate 30.00000, requiredCain 0.00579 totalAchievedCGains 0.00579 with AA 0.17361 {step 0, f-Number 2.40000}, ET 0.00200, AG 16.66667, ISPDG 1.00000

WB: Gains: WBGains 1.000000, 1.000000, 1.000000 1.000000

lux: cutoff {0.250000, 0.750000}

isAHdrEnable 0 , DRE 1.000000, currentSceneLux 79.558952, linearMetric 0.012497, cameraEfficiency 0.150800 cg highlight 0.005787

(Argus) Objects still active during exit: [CameraProvider (0x55c2397f00): refs: 1, cref: 0]

Argus Version: 0.97.3 (single-process)

Camera count: 4

DONE!

Could you try smaller resolution? I have verify with internal tools with 2592x1944

Tried 4056x3040, 3840x2160, 1920x1080, 2028x1520 they all have the same issue.

Can you please ask your devs to look at my code (attached to the first post) to verify that we’re doing the capture correctly?