Recording with multiple camera sensors using driveworks recorder tool

alon.eshel · July 14, 2024, 4:54pm

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

Hi,
In our setup we use 6 Entron F008A120RM0AV2 cameras (connected to MATE-EX connectors A,C).
I’m trying to make a recording using your recorder tool (DriveWorks SDK Reference: Basic Recording Tool) and the attached rig file.
I do manage to stream the 6 camera sensors at 30fps using the following command:
sudo /opt/nvidia/drive-linux/samples/nvmedia/nvsipl/test/camera/nvsipl_camera --platform-config 'F008A120RM0AV2_CPHY_x4' --link-enable-masks '0x1111 0x0000 0x0011 0x0000' --showfps --disableISP1Output --disableISP2Output

I get this output:

Pipeline: 0 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 1 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 2 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 3 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 8 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
Pipeline: 9 ISP Output: 0 is using YUV 420 SEMI-PLANAR UINT8 BL REC_709ER
MAX96712: Revision 5 detected
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX96712 Link 1: PHY optimization was enabled
MAX96712 Link 0: PHY optimization was enabled
MAX96712 Link 1: PHY optimization was enabled
MAX96712 Link 2: PHY optimization was enabled
MAX96712 Link 3: PHY optimization was enabled
MAX9295: Revision 8 detected!
MAX9295: Revision 8 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
MAX9295: Revision 8 detected!
MAX9295: Revision 8 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
MAX9295: Revision 8 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
MAX9295: Revision 8 detected!
Sensor AR0820 GRBG Rev 2.1 detected!
nvsipl_camera: Opened NITO file for module "F008A120RM0AV2", file name: "/usr/share/camera/F008A120RM0AV2.nito" 
nvsipl_camera: Opened NITO file for module "F008A120RM0AV2", file name: "/usr/share/camera/F008A120RM0AV2.nito" 
nvsipl_camera: Opened NITO file for module "F008A120RM0AV2", file name: "/usr/share/camera/F008A120RM0AV2.nito" 
nvsipl_camera: Opened NITO file for module "F008A120RM0AV2", file name: "/usr/share/camera/F008A120RM0AV2.nito" 
nvsipl_camera: Opened NITO file for module "F008A120RM0AV2", file name: "/usr/share/camera/F008A120RM0AV2.nito" 
nvsipl_camera: Opened NITO file for module "F008A120RM0AV2", file name: "/usr/share/camera/F008A120RM0AV2.nito" 
Enter 'les' followed by sensor ID to enable LED
Enter 'lds' followed by sensor ID to disable LED
Enter 'dl' followed by sensor ID to disable the link
Enter 'el' followed by sensor ID to enable the link without module reset
Enter 'elr' followed by sensor ID to enable the link with module reset
Enter 'cm' to check the module availability
Enter 'q' to quit the application
-
Output
Sensor0_Out1    Frame rate (fps):               28.9989
Sensor1_Out1    Frame rate (fps):               28.9989
Sensor2_Out1    Frame rate (fps):               28.9989
Sensor3_Out1    Frame rate (fps):               28.9989
Sensor8_Out1    Frame rate (fps):               28.9989
Sensor9_Out1    Frame rate (fps):               28.9989

Output
Sensor0_Out1    Frame rate (fps):               29.9989
Sensor1_Out1    Frame rate (fps):               29.9989
Sensor2_Out1    Frame rate (fps):               29.9989
Sensor3_Out1    Frame rate (fps):               29.9989
Sensor8_Out1    Frame rate (fps):               29.9989
Sensor9_Out1    Frame rate (fps):               29.9989

Output
Sensor0_Out1    Frame rate (fps):               29.9989
Sensor1_Out1    Frame rate (fps):               29.9989
Sensor2_Out1    Frame rate (fps):               29.9989
Sensor3_Out1    Frame rate (fps):               29.9989
Sensor8_Out1    Frame rate (fps):               29.9989
Sensor9_Out1    Frame rate (fps):               29.9989

Output
Sensor0_Out1    Frame rate (fps):               29.9976
Sensor1_Out1    Frame rate (fps):               29.9976
Sensor2_Out1    Frame rate (fps):               29.9976
Sensor3_Out1    Frame rate (fps):               29.9976
Sensor8_Out1    Frame rate (fps):               29.9976
Sensor9_Out1    Frame rate (fps):               29.9976

Sensor0         Frame captured: 238             Frame drops: 0          Frame discontinuities: 0
Sensor1         Frame captured: 238             Frame drops: 0          Frame discontinuities: 0
Sensor2         Frame captured: 238             Frame drops: 0          Frame discontinuities: 0
Sensor3         Frame captured: 238             Frame drops: 0          Frame discontinuities: 0
Sensor8         Frame captured: 238             Frame drops: 0          Frame discontinuities: 0
Sensor9         Frame captured: 238             Frame drops: 0          Frame discontinuities: 0
SUCCESS

However, when I run your recorder tool (DriveWorks SDK Reference: Basic Recording Tool) I notice several issues with the recordings files:

the time difference between consecutive timestamps is not representive of 30fps.
the amount of frames recorded is not equal between all camera sensors.
each camera sensor timestamps file begins at a different timestamp, regardless if “async-record”=0 or 1. for example:

camera_sample0.bin.timestamps: 1720717998043563 (Thursday, 11 July 2024 17:13:18.043)
camera_sample1.bin.timestamps: 1720718000310207 (Thursday, 11 July 2024 17:13:20.310)
camera_sample2.bin.timestamps: 1720718002876873 (Thursday, 11 July 2024 17:13:22.876)
camera_sample3.bin.timestamps: 1720718005076880 (Thursday, 11 July 2024 17:13:25.076)
camera_sample4.bin.timestamps: 1720718007510131 (Thursday, 11 July 2024 17:13:27.510) 	
camera_sample5.bin.timestamps: 1720718012210146 (Thursday, 11 July 2024 17:13:32.210)

I also tried settings in the rig camera parameters a higher file-buffer-size and setting skip-eeprom=1 but it didn’t make a difference.
I also tried adding a time sensor like you suggested in this topic How to use drivework to synchronize time with multiple cameras? - #3 by SivaRamaKrishnaNV but it didn’t make a difference either.

what I would like to achieve:

record at 30fps (also, would like to understand if it’s possible to configure to other fps rates)
all 6 timestamps files will consist of the same timestamps (cameras synced)
all 6 timestamps files will consist of the same amount of frames timestamps.

How can i achieve that? What modification are required in the rig file? This is essential for me to create datasets for simulations and also to verify the rig file is configured properly.
nvidia_recorder_tool_output.txt (94.6 KB)
6_cameras_rig.txt (7.2 KB)

Many thanks,
Alon

SivaRamaKrishnaNV · July 14, 2024, 6:05pm

Dear @alon.eshel,
Could you please check the write speed to storage using dd command? I would expect framedrop to happen if the write speed lower than expected BW needed to store.
You can use frame-mod=2 in rig file camera parameters to select alternate frame(15FPS) to store.

alon.eshel · July 14, 2024, 8:05pm

Thanks for the quick reply.

I ran this command:
sudo dd if=/dev/zero of=/home/nvidia/testing_nvsipl/6_cameras_testing/recordings/testfile bs=1G count=1 oflag=dsync

got this output:

1+0 records in
1+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.90136 s, 370 MB/s

Ran it several times to validate. please correct me if should run it in a different way.

Seems too low compared to the required BW (6 x 30 x 8MB).
So what can we do?
Does it mean that streaming 6 cameras at 30FPS is not possible?

Thanks,
Alon

SivaRamaKrishnaNV · July 15, 2024, 1:19am

Dear @alon.eshel ,
May I know your needed recorder encoding format like raw/h264/h265/mp4? You can select the required encoding format in rig file using format=.... output-format=..... parameters. For more details, check DriveWorks SDK Reference: Configuration Reference
Can you confirm if you press s to start recording and q to quit recording when using recorder tool?

Keep increasing cameras one by one and run the recorder tool about 10 secs( time between s and q) and see the number of timestamp entries in timestamp file.

Please share the observed results and logs in above test.

alon.eshel · July 15, 2024, 2:56pm

Dear @alon.eshel ,
May I know your needed recorder encoding format like raw/h264/h265/mp4? You can select the required encoding format in rig file using format=.... output-format=..... parameters. For more details, check DriveWorks SDK Reference: Configuration

I set format=h264 for simulation purposes in driveworks and output-format=processed since I currently don’t require the raw.

Can you confirm if you press s to start recording and q to quit recording when using recorder tool?

Yes I’m starting the recorder as the manual says, s to start, s /dev/null to stop and q to quit.

Keep increasing cameras one by one and run the recorder tool about 10 secs( time between s and q) and see the number of timestamp entries in timestamp file.

Please share the observed results and logs in above test.

After some trial and error with less cameras I came to the conclusion the issue is due to the limitation of the writing BW - when using fewer camera sensors the timestamps are as expected.

I set the following params: frame-mod=2,isp0-res=1924X1084 to 5 camera sensors and frame-mod=2 to the 6th. This way I manage to record 6 camera sensors at 15FPS - 1 at 8MP and 5 at 2MP resolution. This is the best setup that works with the limitation of the writing BW.

If you have a better solution please share.

SivaRamaKrishnaNV · July 15, 2024, 3:15pm

Dear @alon.eshel,
The write speed to the storage is the limitation. Please see if you have any SSD with good write speed that can be connected via USB port to meet your need.

system · August 8, 2024, 4:45pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.