Distributed recording - (ssh-copi-id failed

Autonomous Vehicles DRIVE AGX Orin DRIVE AGX Orin General
1

Please provide the following info (tick the boxes after creating this topic):
Software Version
DRIVE OS 6.0.8.1
DRIVE OS 6.0.6
DRIVE OS 6.0.5
DRIVE OS 6.0.4 (rev. 1)
DRIVE OS 6.0.4 SDK
other

Target Operating System
Linux
QNX
other

Hardware Platform
DRIVE AGX Orin Developer Kit (940-63710-0010-300)
DRIVE AGX Orin Developer Kit (940-63710-0010-200)
DRIVE AGX Orin Developer Kit (940-63710-0010-100)
DRIVE AGX Orin Developer Kit (940-63710-0010-D00)
DRIVE AGX Orin Developer Kit (940-63710-0010-C00)
DRIVE AGX Orin Developer Kit (not sure its number)
other

SDK Manager Version
1.9.3.10904
other

Host Machine Version
native Ubuntu Linux 20.04 Host installed with SDK Manager
native Ubuntu Linux 20.04 Host installed with DRIVE OS Docker Containers
native Ubuntu Linux 18.04 Host installed with DRIVE OS Docker Containers
other

I am trying to run recorder-tui with two agx orin and multiple cameras. I have made a rig directory with rig file names as ip address. when I try to run I get a warning ssh-copy-id failed and the application terminates

3

Could you provide more details about the “ssh-copy-id” issue you’re encountering? It seems to be related to setting up passwordless authentication between systems, but it’s unclear how it’s connected to the recorder-tui tool. Could you elaborate on the specific error message or issue you’re facing with “ssh-copy-id”?

4

I have setup two orins and also configured SSH keys. Furthermore, I have also modified SSH config and when I use terminal from one orin to ssh into another I can without any password requirements. Both orins are on same network. However when I run recorder-tui with a rig_dir and 2 json config file each for the two orin from the main orin, I am getting the ssh-copy-id error and the app is terminating. The rig files in rig directory are 192.168.1.1.json and 192.168.1.4.json .1 and.4 being the ip address of the two orins

5

nvidia@tegra-ubuntu:/usr/local/driveworks/tools/capture$ sudo ./recorder-tui /home/nvidia/go/192.168.1.30.json
./recorder-tui:1296: SyntaxWarning: “is” with a literal. Did you mean “==”?
if hostname is “local”:
./recorder-tui:1310: SyntaxWarning: “is not” with a literal. Did you mean “!=”?
if hostname is not “local”:
running recorder-tui with arguments: Namespace(aes_key=None, aes_key_encrypted=None, backend=False, bbr=False, description=‘’, disable_encryption=True, enable_encryption=False,
lane_id=None, log_dir_path=‘/tmp’, login_id=None, no_mount_check=False, no_sudo=False, non_interaction=False, remote_dw_path=‘’, rig=‘/home/nvidia/go/192.168.1.30.json’, route=N
one, rsa_key=‘/home/nvidia/.ssh/recorder-aiinfra.pem’, rsa_key_md5=None, skip_init_input=False, tag=‘NONE’)
./recorder-tui:250: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
if ip_addr.replace(“.”, “”, 3).isnumeric() and (not utils.checkLocalIP(ip_addr)):
[15-03-2024 13:05:03] TimeSource Nvpss : PTP ioctl returned error. Synchronized time will not be available from this timesource.
[15-03-2024 13:05:03] SDK: No resources(.pak) mounted from ‘/usr/local/driveworks-5.10/data’. Please adjust path or some modules won’t function properly.
[15-03-2024 13:05:03] TimeSource Nvpss : PTP ioctl returned error. Synchronized time will not be available from this timesource.
Copying rigs and setup code for local…
Platform setup for local…
./recorder-tui:250: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
if ip_addr.replace(“.”, “”, 3).isnumeric() and (not utils.checkLocalIP(ip_addr)):
./recorder-tui:250: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
if ip_addr.replace(“.”, “”, 3).isnumeric() and (not utils.checkLocalIP(ip_addr)):
[ 5500.893288] nvpps nvpps0: unable to switch mode. Only timer mode is supported
192.168.1.30.json: Backend abruptly exited in waitForRecorderToStart
Cleanup started …
Waiting (timeout=30s) for child process 1/1 to exit cleanly…
Cleaning up sensor initializers …
Performing platform recorder cleanup, if found …
Errors detected during recording
First 100 lines of std error log:
/usr/local/driveworks-5.10/tools/capture/recorder-tui:1296: SyntaxWarning: “is” with a literal. Did you mean “==”?
if hostname is “local”:
/usr/local/driveworks-5.10/tools/capture/recorder-tui:1310: SyntaxWarning: “is not” with a literal. Did you mean “!=”?
if hostname is not “local”:
/usr/local/driveworks-5.10/tools/capture/recorder-tui:250: DeprecationWarning: tostring() is deprecated. Use tobytes() instead.
if ip_addr.replace(“.”, “”, 3).isnumeric() and (not utils.checkLocalIP(ip_addr)):
[15-03-2024 13:05:03] Platform: Detected Drive Orin P3710
[15-03-2024 13:05:03] TimeSource: monotonic epoch time offset is 1710516803147972
[15-03-2024 13:05:03] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[15-03-2024 13:05:03] TimeSource Nvpss : PTP ioctl returned error. Synchronized time will not be available from this timesource.
[15-03-2024 13:05:03] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[15-03-2024 13:05:03] PTP Time is available from Eth Driver
[15-03-2024 13:05:04] Platform: currently selected GPU device integrated ID 0
[15-03-2024 13:05:04] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘./resources’: VirtualFileSystem: Failed to mount ‘./resources/resources.pak’
[15-03-2024 13:05:04] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/tmp/recorder-4607/data’: VirtualFileSystem: Failed to mount ‘/tmp/recorder-4607/dat
a/resources.pak’
[15-03-2024 13:05:04] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.10/tools/capture/…/…/data’: VirtualFileSystem: Failed to m
ount ‘/usr/local/driveworks-5.10/tools/capture/…/…/data/resources.pak’
[15-03-2024 13:05:04] Context::mountResourceCandidateDataPath resource FAILED to mount from ‘/usr/local/driveworks-5.10/data’: VirtualFileSystem: Failed to mount ‘/usr/local/dri
veworks-5.10/data/resources.pak’
[15-03-2024 13:05:04] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/tools/captur
e/…/…/targets/aarch64-Linux/lib/libdw_base.so.5.10
[15-03-2024 13:05:04] Context::findResourcesPackageInPathWalk: Could not find ./resources/resources.pak in upto 7 parent directories from /usr/local/driveworks-5.10/targets/aarc
h64-Linux/lib/libdw_base.so.5.10
[15-03-2024 13:05:04] SDK: No resources(.pak) mounted, some modules will not function properly
[15-03-2024 13:05:04] egl::Display: found 1 EGL devices
[15-03-2024 13:05:04] egl::Display: use drm device: drm-nvdc
[15-03-2024 13:05:04] TimeSource: monotonic epoch time offset is 1710516803147973
[15-03-2024 13:05:04] TimeSourceVibranteLinux: detect valid PTP interface mgbe2_0
[15-03-2024 13:05:04] TimeSource Nvpss : PTP ioctl returned error. Synchronized time will not be available from this timesource.
[15-03-2024 13:05:04] TimeSource: Could not detect valid PTP time source at nvpps. Fallback to mgbe2_0
[15-03-2024 13:05:04] PTP Time is available from Eth Driver
[15-03-2024 13:05:04] Initialize DriveWorks SDK v5.10.87
[15-03-2024 13:05:04] Release build with GNU 9.3.0 from buildbrain-branch-0-g9a5b4670e12 against Drive PDK v6.0.6.0
[15-03-2024 13:05:04] Rig overlay directory does not exist: /storage/driveworks/data/vehicle_cfg
[15-03-2024 13:05:04] Platform: currently selected GPU device integrated ID 0
[15-03-2024 13:05:04] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
[15-03-2024 13:05:04] No valid data file found for RSWL_IMX490 in parameter string: camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,frame-mod
=3,format=h265,isp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito (using configuration folder /tmp/recorder-4607/)
[15-03-2024 13:05:04] No valid data file found for FSWL_IMX490 in parameter string: camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=1,output-format=processed,frame-mod
=3,format=h265,isp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito (using configuration folder /tmp/recorder-4607/)
[15-03-2024 13:05:04] No valid data file found for FSWR_IMX490 in parameter string: camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=2,output-format=processed,frame-mod
=3,format=h265,isp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito (using configuration folder /tmp/recorder-4607/)
[15-03-2024 13:05:04] No valid data file found for RSWR_IMX490 in parameter string: camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=3,output-format=processed,frame-mod
=3,format=h265,isp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito (using configuration folder /tmp/recorder-4607/)
[15-03-2024 13:05:04] No valid data file found for AVM_Left_IMX623 in parameter string: camera-name=IMX623_RGGB,interface=csi-ef,CPHY-mode=1,link=0,output-format=processed,frame
-mod=3,format=h265,isp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX623_RGGB.nito (using configuration folder /tmp/recorder-4607/)
[15-03-2024 13:05:04] No valid data file found for AVM_Right_IMX623 in parameter string: camera-name=IMX623_RGGB,interface=csi-ef,CPHY-mode=1,link=1,output-format=processed,fram
e-mod=3,format=h265,isp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX623_RGGB.nito (using configuration folder /tmp/recorder-4607/)
[15-03-2024 13:05:04] rig::DatabaseOverlayer: Failed to open a file ‘/tmp/car_vin’ - file likely does not exist - skipping overlay
[15-03-2024 13:05:04] SensorFactory::createSensor() → time.nvpps, nvpps-device=/dev/nvpps0
[15-03-2024 13:05:04] TimeSensor: failed to set NVPPS parameters. Error: Invalid argument
[15-03-2024 13:05:04] This platform does not support NVPPS GPIO mode. Fallback to timer mode
[15-03-2024 13:05:04] TimeSensor: initialized with no UTC time reference.
[15-03-2024 13:05:04] EndpointNVPPS: started on /dev/nvpps0
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl, camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=0,output-format=processed,frame-mod=3,format=h265,i
sp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl.master,
[15-03-2024 13:05:04] CameraMaster::parseDevBlock Getting device info list.
[15-03-2024 13:05:04] CameraClient, setting isp-mode to yuv420-uint8
[15-03-2024 13:05:04] CameraClient, isp-mode set to yuv420-uint8
[15-03-2024 13:05:04] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = IMX490_RGGB Link = 0 GroupInit = 0 RecCfg = 1
[15-03-2024 13:05:04] Camera Match Name: IMX490_RGGB Description: Sony IMX490 RGGB module - 120-deg FOV, MIPI-IMX490, MAX9295 linkIndex: 4294967295 serInfo.Name: MAX9295
[15-03-2024 13:05:04] Client, Setting up information for camera ID 0
[15-03-2024 13:05:04] Client, successfully found info for camera ID 0 bound to id 0
[15-03-2024 13:05:04] CameraClient: using NITO found at /usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl, camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=1,output-format=processed,frame-mod=3,format=h265,i
sp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] CameraClient, setting isp-mode to yuv420-uint8
[15-03-2024 13:05:04] CameraClient, isp-mode set to yuv420-uint8
[15-03-2024 13:05:04] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = IMX490_RGGB Link = 1 GroupInit = 0 RecCfg = 1
[15-03-2024 13:05:04] Camera Match Name: IMX490_RGGB Description: Sony IMX490 RGGB module - 120-deg FOV, MIPI-IMX490, MAX9295 linkIndex: 4294967295 serInfo.Name: MAX9295
[15-03-2024 13:05:04] Client, Setting up information for camera ID 1
[15-03-2024 13:05:04] Client, successfully found info for camera ID 1 bound to id 1
[15-03-2024 13:05:04] CameraClient: using NITO found at /usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl, camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=2,output-format=processed,frame-mod=3,format=h265,i
sp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] CameraClient, setting isp-mode to yuv420-uint8
[15-03-2024 13:05:04] CameraClient, isp-mode set to yuv420-uint8
[15-03-2024 13:05:04] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = IMX490_RGGB Link = 2 GroupInit = 0 RecCfg = 1
[15-03-2024 13:05:04] Camera Match Name: IMX490_RGGB Description: Sony IMX490 RGGB module - 120-deg FOV, MIPI-IMX490, MAX9295 linkIndex: 4294967295 serInfo.Name: MAX9295
[15-03-2024 13:05:04] Client, Setting up information for camera ID 2
[15-03-2024 13:05:04] Client, successfully found info for camera ID 2 bound to id 2
[15-03-2024 13:05:04] CameraClient: using NITO found at /usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl, camera-name=IMX490_RGGB,interface=csi-ab,CPHY-mode=1,link=3,output-format=processed,frame-mod=3,format=h265,i
sp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] CameraClient, setting isp-mode to yuv420-uint8
[15-03-2024 13:05:04] CameraClient, isp-mode set to yuv420-uint8
[15-03-2024 13:05:04] devBlock: 0 Slave = 0 Interface = csi-ab Camera_name = IMX490_RGGB Link = 3 GroupInit = 0 RecCfg = 1
[15-03-2024 13:05:04] Camera Match Name: IMX490_RGGB Description: Sony IMX490 RGGB module - 120-deg FOV, MIPI-IMX490, MAX9295 linkIndex: 4294967295 serInfo.Name: MAX9295
[15-03-2024 13:05:04] Client, Setting up information for camera ID 3
[15-03-2024 13:05:04] Client, successfully found info for camera ID 3 bound to id 3
[15-03-2024 13:05:04] CameraClient: using NITO found at /usr/share/camera/IMX490_RGGB_FoV120.nito
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl, camera-name=IMX623_RGGB,interface=csi-ef,CPHY-mode=1,link=0,output-format=processed,frame-mod=3,format=h265,i
sp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX623_RGGB.nito
[15-03-2024 13:05:04] CameraClient, setting isp-mode to yuv420-uint8
[15-03-2024 13:05:04] CameraClient, isp-mode set to yuv420-uint8
[15-03-2024 13:05:04] devBlock: 1 Slave = 0 Interface = csi-ef Camera_name = IMX623_RGGB Link = 0 GroupInit = 0 RecCfg = 1
[15-03-2024 13:05:04] Camera Match Name: IMX623_RGGB Description: Sony IMX623 RGGB module - 120-deg FOV, MIPI-IMX623, MAX96717F linkIndex: 4294967295 serInfo.Name: MAX96717F
[15-03-2024 13:05:04] Client, Setting up information for camera ID 4
[15-03-2024 13:05:04] Client, successfully found info for camera ID 4 bound to id 4
[15-03-2024 13:05:04] CameraClient: using NITO found at /usr/share/camera/IMX623_RGGB.nito
[15-03-2024 13:05:04] SensorFactory::createSensor() → camera.gmsl, camera-name=IMX623_RGGB,interface=csi-ef,CPHY-mode=1,link=1,output-format=processed,frame-mod=3,format=h265,i
sp-mode=yuv420-uint8,nito-file=/usr/share/camera/IMX623_RGGB.nito
[15-03-2024 13:05:04] CameraClient, setting isp-mode to yuv420-uint8
[15-03-2024 13:05:04] CameraClient, isp-mode set to yuv420-uint8
[15-03-2024 13:05:04] devBlock: 1 Slave = 0 Interface = csi-ef Camera_name = IMX623_RGGB Link = 1 GroupInit = 0 RecCfg = 1
[15-03-2024 13:05:04] Camera Match Name: IMX623_RGGB Description: Sony IMX623 RGGB module - 120-deg FOV, MIPI-IMX623, MAX96717F linkIndex: 4294967295 serInfo.Name: MAX96717F
[15-03-2024 13:05:04] Client, Setting up information for camera ID 5
[15-03-2024 13:05:04] Client, successfully found info for camera ID 5 bound to id 5
[15-03-2024 13:05:04] CameraClient: using NITO found at /usr/share/camera/IMX623_RGGB.nito
[15-03-2024 13:05:04] SIPLMaster::SIPLMaster: Setting up master camera
[15-03-2024 13:05:04] CameraMaster available outputs: 1
[15-03-2024 13:05:04] CameraMaster available outputs: 1
[15-03-2024 13:05:04] CameraMaster available outputs: 1
[15-03-2024 13:05:04] CameraMaster available outputs: 1
[15-03-2024 13:05:04] CameraMaster available outputs: 1
First 100 lines of std out log:
running recorder-tui with arguments: Namespace(aes_key=None, aes_key_encrypted=None, backend=True, bbr=False, description=‘’, disable_encryption=True, enable_encryption=False, l
ane_id=None, log_dir_path=‘/tmp’, login_id=None, no_mount_check=False, no_sudo=False, non_interaction=False, remote_dw_path=‘’, rig=‘/tmp/recorder-4607/192.168.1.30.json’, route
=None, rsa_key=‘/home/nvidia/.ssh/recorder-aiinfra.pem’, rsa_key_md5=None, skip_init_input=False, tag=‘NONE’)
Rig: 192.168.1.30.json recorder-tui: started
MAX96712: Revision 5 detected
MAX96712: Revision 5 detected
MAX96712 Link 0: PHY optimization was enabled
MAX96712 Link 1: PHY optimization was enabled
MAX96712 Link 0: PHY optimization was enabled
MAX96712 Link 1: PHY optimization was enabled
MAX96712 Link 2: PHY optimization was enabled
MAX96712 Link 3: PHY optimization was enabled
Rig: 192.168.1.30.json CurrentMount: /media/sda1_ssd
Rig: 192.168.1.30.json Rollover: 1
Rig: 192.168.1.30.json EndOfMountsInfo
Rig: 192.168.1.30.json Error: recorder exited abnormally: cannot send cmd
When reporting bugs, please attach full log and files /tmp/recorder-tui-4607/.log and /tmp/recorder-4607/.log
WARNING: Forcibly exited. Some data might have been lost!
WARNING: Use q to exit gracefully, next time.
here is the log

6

Thank you for sharing the detailed log. To better assist you, could you please point out the specific snippet where the SSH copy-id error occurs?

Please also provide the log files mentioned.

7

for distributed recording which Ethernet port to use between two orin 1g or 10g?

8

The log files are empty and still it just warn about ssh-copy-id. I checked all the setting related to ssh and they are working fine when i use terminal to ssh into second orin from first. the application does not start with distributed recording. It is urgent please help

9

Are you referring to the ‘Distributed Recording’ document to attempt this setup on two Orin devkits? It’s not available in the DriveWorks documentation for Orin.

10

Yes I am. Could we have a quick call on this issue. Do i need to use Aurix IP address on the Nvidia Orins or the IP address I setup for eqos_0 socket?

11

The document regarding Distributed Recording has indeed been removed from Orin, and as a result, it hasn’t undergone testing by our QA team. Therefore, it’s expected that it may not function as intended.

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.