Hey,
I’m experiencing many different issues with my multicamera setup using nvarguscamerasrc.
First of all my setup:
- Jetson Orin 8GB with OE4T/meta-tegra scarthgap image (Jetson Linux 36.4.4)
- jetson clocks are boosted
- up to 4x AR0544 imager
- each imager is triggered by a hardware trigger for synchronization reasons
- each imager gets its own gstreamer pipeline. Each pipeline looks like this:
My usecases:
Stream images between 5 Hz and 20 Hz, which are hardware triggered. Pause the stream within 4 seconds and start the stream at any later point.
1. Starting multiple streams at the same time crashes gstreamer (solved by myself)
Simply execute this command below, will result into crashes:
gst-launch-1.0 \
nvarguscamerasrc sensor-id=0 sensor-mode=0 ! fakesink \
nvarguscamerasrc sensor-id=1 sensor-mode=0 ! fakesink \
nvarguscamerasrc sensor-id=2 sensor-mode=0 ! fakesink \
nvarguscamerasrc sensor-id=3 sensor-mode=0 ! fakesink
Logs
[...]
Jan 01 00:20:33 ovp93x eugust[1139]: nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
Jan 01 00:20:33 ovp93x eugust[1139]: End-Of-Stream reached.
nvargus-daemon does not produce any logs, even if you activate verbose output
→ Solved this issue by adding a inter process mutex across each ArgusCamera::execute
// Use file lock for inter-process synchronization to prevent race conditions
// in Argus daemon when multiple nvarguscamerasrc instances start simultaneously
// across different processes (e.g., multiple gst-launch-1.0 invocations)
{
ScopedFileLock initLock(ARGUS_INIT_LOCK_FILE);
if (!initLock.isValid()) {
GST_WARNING("Could not create lock file %s (errno=%d), proceeding without lock\n", ARGUS_INIT_LOCK_FILE.c_str(), errno);
} else if (!initLock.isLocked()) {
GST_WARNING("Could not acquire lock (errno=%d), proceeding without lock\n", errno);
} else {
GST_INFO("Camera %d acquired initialization lock\n", cameraIndex);
}
GST_ARGUS_PRINT("Starting repeat capture requests.\n");
Request* captureRequest = src->request.get();
src->request_ptr = captureRequest;
iCaptureSession->capture(captureRequest);
if (iCaptureSession->capture(captureRequest) == 0)
ORIGINATE_ERROR("Failed to start capture request");
// Add a small delay to let the Argus daemon fully complete this camera's setup
// before allowing the next camera to initialize
if (initLock.isLocked()) {
std::this_thread::sleep_for(ARGUS_INIT_SETTLE_TIME);
GST_INFO("Camera %d releasing initialization lock\n", cameraIndex);
}
}
2. PAUSE the stream
A classic gstreamer state change from GST_STATE_PLAYING to GST_STATE_PAUSED does not work. In that case, nvargus-daemon run into a timeout. I already know that this is a common issue, so instead of GST_STATE_PAUSED i set it to GST_STATE_READY. As I understood, this will stop the whole argus capture session, which does not lead to a nvargus-daemon timeout. (I did try enableCamInfiniteTimeout=1 but i only received 1 Frame and then nvargus-daemon crashed)
Anyway, this takes approximately 5 seconds to set all 4 streams to pause, which is far too long for our use case! Switching only one stream to READY takes ~200ms. Is there somehow a way to speed up setting the stream to PAUSE / READY?
I already noticed that the line UniqueObj<Frame> frame(iFrameConsumer->acquireFrame(src->acquire_timeout, &frame_status)); (within StreamConsumer::threadExecute) can block the teardown up to src->acquire_timeout seconds (default 5s). Reducing the timeout number would lead to false positive “eof” interpretations, which shutdowns pipeline. Also tried to call the acquireFrame in much smaller slices of 100ms each slice up to the acquire_timeout. This increased the teardown speed, but leads to other crashes in the libargus after 100 to 500 state changes. Did not further investigate the internal Argus state machine behavior.
I also tested the state switch with only one imager connected and noticed some other issues after some time:
nvargus-daemon logs
Jan 01 02:57:03 ovp93x nvargus-daemon[885]: === NVIDIA Libargus Camera Service (0.99.33)=== Listening for connections...=== eugust[1136]: Connection established (FFFF819DB8C0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Jan 01 02:57:03 ovp93x nvargus-daemon[885]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 01 02:57:03 ovp93x nvargus-daemon[885]: NvPclHwGetModuleList: No module data found
Jan 01 02:57:03 ovp93x nvargus-daemon[885]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Jan 01 02:57:03 ovp93x nvargus-daemon[885]: ---- imager: Found override file [/var/nvidia/nvcam/settings/ar0544_centerleft_ifmar0544.isp]. ----
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: CAM: serial no file already exists, skips storing again=== eugust[1136]: CameraProvider initialized (0xffff7cb19c20)CAM: serial no file already exists, skips storing againCAM: serial no file already exists, skips storing again(NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 277)
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: PowerServiceCore:handleRequests: timePassed = 1603
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 2064)
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: updateOutputSettings: Set Control failed. Use cached values
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 277)
Jan 01 02:59:29 ovp93x nvargus-daemon[885]: PowerServiceCore:handleRequests: timePassed = 754
Jan 01 02:59:32 ovp93x nvargus-daemon[885]: CAM: serial no file already exists, skips storing againPowerServiceCore:handleRequests: timePassed = 1683
Jan 01 02:59:35 ovp93x nvargus-daemon[885]: CAM: serial no file already exists, skips storing again(NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
Jan 01 02:59:35 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 277)
Jan 01 02:59:35 ovp93x nvargus-daemon[885]: PowerServiceCore:handleRequests: timePassed = 1650
Jan 01 02:59:35 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
Jan 01 02:59:35 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 2064)
Jan 01 02:59:35 ovp93x nvargus-daemon[885]: updateOutputSettings: Set Control failed. Use cached values
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 277)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 277)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: PowerServiceCore:handleRequests: timePassed = 484
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function setControlValMultiple(), line 792)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function setDeviceControls(), line 2064)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: updateOutputSettings: Set Control failed. Use cached values
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvCamV4l2) Error IoctlFailed: (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamv4l2/v4l2_device.cpp, function streamControl(), line 1661)
Jan 01 02:59:36 ovp93x nvargus-daemon[885]: (NvOdmDevice) Error IoctlFailed: (propagating from dvs/git/dirty/git-master_linux/camera-partner/imager/src/devices/V4L2SensorViCsi.cpp, function apply(), line 277)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 281470681743360 draining session frameStart events 3
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 529)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Sensor GUID 1 is in error state. Skipping requests, capture sequence ID = 281470681743361 continue draining session frameStart events 2
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Sensor GUID 1 is in error state. Skipping requests, capture sequence ID = 281470681743362 continue draining session frameStart events 1
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 543)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Sensor 1 already in same state
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 0 draining session frameEnd events 3
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Sensor 1 already in same state
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 1 draining session frameEnd events 2
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Sensor 1 already in same state
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/CaptureServiceDeviceSensor.cpp, function setErrorState(), line 100)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Timeout!! Skipping requests on sensor GUID 1, capture sequence ID = 2 draining session frameEnd events 1
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/services/capture/FusaCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 646)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error Timeout: Sending critical error event for Session 1
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (in src/api/Session.cpp, function sendErrorEvent(), line 1039)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error BadParameter: CC has already been disposed (in src/components/CaptureContainerManager.cpp, function dispose(), line 161)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: SCF: Error InvalidState: Session has suffered a critical failure (in src/api/Session.cpp, function capture(), line 734)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: (Argus) Error InvalidState: (propagating from src/api/ScfCaptureThread.cpp, function run(), line 110)
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: === eugust[1136]: Connection closed (FFFF819DB8C0)=== eugust[1136]: WARNING: CameraProvider was not destroyed before client connection terminated.=== eugust[1136]: The client may have abnormally terminated. Destroying CameraProvider...=== eugust[1136]: CameraProvider destroyed (0xffff7cb19c20)=== eugust[1136]: WARNING: Cleaning up 1 outstanding requests...=== eugust[1136]: WARNING: Cleaning up 1 outstanding stream settings...=== eugust[1136]: WARNING: Cleaning up 1 outstanding queues...=== eugust[1136]: WARNING: Cleaning up 1 outstanding sessions...PowerServiceCore:handleRequests: timePassed = 4027
Jan 01 02:59:40 ovp93x nvargus-daemon[885]: === eugust[1136]: WARNING: Cleaning up 1 outstanding streams...(Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 1106)
Jan 01 02:59:46 ovp93x nvargus-daemon[885]: === eugust[1136]: NOTE: Destroy all libargus objects before destroying the CameraProvider to avoid these warnings.=== eugust[1136]: Connection cleaned up (FFFF819DB8C0)=== eugust[1643]: Connection established (FFFF819DB8C0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
Jan 01 02:59:46 ovp93x nvargus-daemon[885]: NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
Jan 01 02:59:46 ovp93x nvargus-daemon[885]: NvPclHwGetModuleList: No module data found
Jan 01 02:59:46 ovp93x nvargus-daemon[885]: OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
Jan 01 02:59:46 ovp93x nvargus-daemon[885]: ---- imager: Found override file [/var/nvidia/nvcam/settings/ar0544_centerleft_ifmar0544.isp]. ----
Throwing away the Images to simulate a pause is not a option. This would a) still add system load to the device and b) could cause some issues with our hardware trigger, because this needs to be stopped sometimes to reconfigure the whole synchronization stuff.
Based on these errors, it seems that PAUSE a stream may not be supported in nvarguscamerasrc at all. Is this interpretation correct?
