Wifi-usb ssh connection drops when connecting more then 3 Azure Kinekts, multiple Azures connection to Jetson

Hello! I have a Jetson nano dev board with nvidia-jetpack 4.5.1-b17 and Ubuntu 18.04.5 LTS and 5V 4A Power Supply Charger Adapter. My goal is to connect 4 Azure Kinekts and do a frame capture - it does not need to be a real-time, so I have created a code that turns on and off each of the Kinekts one by one and captures a frame.

I have set up a headless ssh connection and patched with this updates my TP-Link AC600 usb-wifi stich and in addition runned sudo iw dev wlan0 set power_save off GitHub - cilynx/rtl88x2BU_WiFi_linux_v5.3.1_27678.20180430_COEX20180427-5959: rtl88x2bu driver updated for current kernels.

After connecting to Jetson via ssh i set export DISPLAY=:0

Now, I can capture frames remotely via ssh when 3 azures are plugged in +usbwifi stick. When I connect 4th azure, the ssh connection drops/freezes.

I have tried beforehand connecting all 4 azures and ethernet cable and everything worked just fine. Is this a specific problem related to the weird power consumptions of USB-wifi? Any help would be appreciated!

Hi,
The performance of USB dongle is possibly dominated by CPU capability. May be similar to this topic:
The speed of 5G dongle - #27 by DaneLLL

The same USB ethernet dongle provides difference performance result on Xavier and Jetson Nano. Please execute sudo tegrastats to check if you see high CPU loading with the USB-wifi device.

Hi,
Here is output of sudo tegrastats:
RAM 943/3964MB (lfb 569x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [1%@1224,0%@1224,0%@1224,0%@1224] EMC_FREQ 0%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@27C CPU@32.5C PMIC@100C GPU@29.5C AO@34.5C thermal@31.25C POM_5V_IN 1559/1559 POM_5V_GPU 0/0 POM_5V_CPU 155/155
RAM 944/3964MB (lfb 569x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [2%@1224,0%@1224,3%@1224,0%@1224] EMC_FREQ 0%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@27.5C CPU@32.5C PMIC@100C GPU@29.5C AO@34.5C thermal@31.25C POM_5V_IN 1559/1559 POM_5V_GPU 0/0 POM_5V_CPU 155/155
RAM 944/3964MB (lfb 569x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [1%@1224,0%@1224,0%@1224,0%@1224] EMC_FREQ 0%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@27.5C CPU@32.5C PMIC@100C GPU@29.5C AO@34.5C thermal@31C POM_5V_IN 1598/1572 POM_5V_GPU 0/0 POM_5V_CPU 155/155
RAM 944/3964MB (lfb 569x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [1%@1224,0%@1224,0%@1224,0%@1224] EMC_FREQ 0%@1600 GR3D_FREQ 8%@76 VIC_FREQ 0%@192 APE 25 PLL@27.5C CPU@33C PMIC@100C GPU@29.5C AO@34.5C thermal@31C POM_5V_IN 1789/1626 POM_5V_GPU 77/19 POM_5V_CPU 233/174
RAM 944/3964MB (lfb 569x4MB) SWAP 0/1982MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [1%@1224,0%@1224,0%@1224,0%@1224] EMC_FREQ 0%@1600 GR3D_FREQ 0%@76 VIC_FREQ 0%@192 APE 25 PLL@27C CPU@32.5C PMIC@100C GPU@29.5C AO@34.5C thermal@31C POM_5V_IN 1559/1612 POM_5V_GPU 0/15 POM_5V_CPU 155/170

Updates:
Now i tried 3 Kinekts+ Ethernet cable from the Wifi repeater and everything is ok, but when I connect 4th camera i get following error:

[LOG]: Number of connected devices  4
[2021-06-15 17:25:52.457] [error] [t=10119] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/color/uvc_camerareader.cpp (165): Start(). Failed to start streaming: Unknown error
[2021-06-15 17:25:52.457] [error] [t=10119] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/color/color.cpp (194): color->m_spCameraReader->Start(width, height, fps, config->color_format, &color_capture_available, color) returned failure in color_start()
[2021-06-15 17:25:52.457] [error] [t=10119] /__w/1/s/extern/Azure-Kinect-Sensor-SDK/src/sdk/k4a.c (909): color_start(device->color, config) returned failure in k4a_device_start_cameras()
Start K4A cameras failed!
  File "capture_mult_streams.py", line 40, in <module>
    pyK4A.device_start_cameras(device_config)
  File "../pyKinectAzure/pyKinectAzure.py", line 151, in device_start_cameras
    _k4a.VERIFY(self.k4a.k4a_device_start_cameras(self.device_handle,self.config.current_config),"Start K4A cameras failed!")
  File "../pyKinectAzure/_k4a.py", line 570, in VERIFY
    traceback.print_stack()

The solution from here libusb and libuvc errors when using multiple devices on Linux · Issue #485 · microsoft/Azure-Kinect-Sensor-SDK · GitHub did not help.

Again, I am using a one-by-one camera recording, not a simultaneous one, here is the script:

import sys
sys.path.insert(1, '../pyKinectAzure/')
import numpy as np
from pyKinectAzure import pyKinectAzure,_k4a, postProcessing
import cv2
import time
import os

paths = []
sleep_seconds = 5

# Path to the module
modulePath = r'/usr/lib/aarch64-linux-gnu/libk4a.so.1.4'

if __name__ == "__main__":
	# Initialize the library with the path containing the module
	pyK4A = pyKinectAzure(modulePath)

	# Open device
	num_connected_devices = pyK4A.device_get_installed_count()
	print("[LOG]: Number of connected devices ",num_connected_devices)
	#pyK4A.device_stop_cameras()
	#pyK4A.device_stop_imu()
	for dev_id in range(num_connected_devices):
		#whileTrue:
		paths.append("_"+str(dev_id))
		#if dev_id==0:
		#	continue
		if not os.path.exists(paths[dev_id]):
			os.mkdir(paths[dev_id])

		pyK4A.device_open(index=dev_id)
		# Modify camera configuration
		device_config = pyK4A.config
		device_config.camera_fps = _k4a.K4A_FRAMES_PER_SECOND_5
		device_config.color_resolution = _k4a.K4A_COLOR_RESOLUTION_720P
		device_config.depth_mode = _k4a.K4A_DEPTH_MODE_NFOV_UNBINNED

		# Start cameras using modified configuration
		pyK4A.device_start_cameras(device_config)
		depthSensorCalibration = _k4a.k4a_calibration_t()
		while True:
			# Get capture
			pyK4A.device_get_capture()
			#time.sleep(sleep_seconds)
			# Get the depth image from the capture
			depth_image_handle = pyK4A.capture_get_depth_image()
			color_image_handle = pyK4A.capture_get_color_image()

			# Check the image has been read correctly
			if depth_image_handle and color_image_handle:
				# Read and convert the image data to numpy array:
				depth_image = pyK4A.image_convert_to_numpy(depth_image_handle)
				color_image = pyK4A.image_convert_to_numpy(color_image_handle)
				transformed_depth_image = pyK4A.transform_depth_to_color(depth_image_handle, color_image_handle)	

				cv2.imwrite(os.path.join(paths[dev_id],"depth.png"), transformed_depth_image.astype(np.uint16))
				cv2.imwrite(os.path.join(paths[dev_id],"color.png"), color_image)

				# Release the image
				pyK4A.image_release(depth_image_handle)
				pyK4A.image_release(color_image_handle)
				pyK4A.capture_release()
				pyK4A.device_stop_cameras()
				pyK4A.device_close()
				print("[LOG]: Capture saved from dev_id",dev_id)
				time.sleep(3)
				break

Hi,
You probably hit constraint of USB bandwidth:
connected more than two usb cameras problem on deepstream-app (Jetson Nano Dev Kit) - #12 by DaneLLL

The bandwidth may be insufficient for 4 cameras. Or power supply is insufficient and needs external power supply.

Hi @DaneLLL ,
I am not using cameras at the same time - I turn them on and off one by one and I checked the USB bandwidth consumption to verify that only one USB is used.

Secondly, I am using an external power 5V supply.