Problem with IMX477 on 32.7.5 in slave mode. (bug in nvarguscamerasrc?)

Guess this one is more for RidgeRun or Leopard Imaging.
Brief:
Platform: Jetson Nano, Ubuntu 18.04
Application with IMX477 works with L4T 32.4.4 [JP 4.4.1], but crash on L4T 32.7.5. [JP 4.6.5].

We developed an application that uses IMX477 in slave mode. We did the necessary modification on the kernel driver imx477, and everything worked fine.
Lately, we have decided to upgrade L4T to 32.7.5. We applied the same modifications on the driver as in 32.4.4. When we tried a simple pipeline the stream crashed.

Pipeline printed the following errors:

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting..

(for more see: L4T 32.7.5 without modified nvarguscamerasrc.so, IMX in slave mode)

A similar issue was described in Some basic problems with nvarguscamerasrc plugin for gstreamer - #12 by david.fernandez
So we tried to increase 10x TIMEOUTS. We downloaded corresponding gst-nvarguscamera sources for 32.7.5, changed the TIMEOUTS, and compiled it on external jetson nano (L4T 32.4.4). Then we transferred built libgstnvarguscamerasrc.so from external jetson nano. To our targeted device with the camera into the /usr/lib/aarch64-linux-gnu/gstreamer-1.0. We tried to run the stream, and the pipeline failed because of Caught SIGSEGV (for more see: Output of pipeline: L4T 32.7.5 with modified nvarguscamerasrc.so, IMX in master mode.)

I’m not sure if I did something wrong but the new compiled library ( 287 592 B) had a very different size from the original one ( 68 392 B).

We tried to check if it’s working in master mode. With the original nvarguscamerasrc.so and in master mode stream was working without problem (but frames were not in sync with the external device), but after changing nvarguscamerasrc.so we got the same the SIGSEGV.

Was the way I implemented new nvarguscamerasrc.so correct? What is the reason for nvbuf_utils: Can not get HW buffer from FD... Exiting..?

################################## EXPERIMENTS RESULTS ##################################
Tested pipeline:

gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 \
        ! "video/x-raw(memory:NVMM),width=4056,height=3040,framerate=30/1" ! nvvidconv ! pngenc \
        ! multifilesink location=/media/sd_card/camera_frames/slave_%03d.png

==========================================================================================
Pipeline is working with : L4T 32.7.5 without modification on nvarguscamerasrc.so, IMX in master mode.
In the output of pipeline I don’t see any issues, also in dmesg everything look fine.


Output of pipeline: L4T 32.7.5 without modified nvarguscamerasrc.so, IMX in slave mode.

GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
EOS on shutdown enabled -- waiting for EOS after Error
Waiting for EOS...

Output of dmesg

...
[  382.486848] imx477 30-0010: IMX477:: start streaming 1
[  384.058433] fence timeout on [ffffffc0cce5fe40] after 1500ms
[  384.058496] fence timeout on [ffffffc0cce5fcc0] after 1500ms
[  384.058520] name=[nvhost_sync:3], current value=0 waiting value=1
[  384.058559] ---- mlocks ----
[  384.058581] name=[nvhost_sync:4], current value=0 waiting value=1
[  384.058614] ---- mlocks ----

[  384.058660] ---- syncpts ----

[  384.058704] ---- syncpts ----
[  384.058738] id 3 (54600000.isp_0) min 0 max 3 refs 4 (previous client : )
[  384.058784] id 4 (54600000.isp_1) min 0 max 3 refs 4 (previous client : )
...

Look like nvargus is crashing.


Output of pipeline: L4T 32.7.5 with modified nvarguscamerasrc.so, IMX in master mode.

GST_ARGUS: Available Sensor modes :
Caught SIGSEGV
#0  0x0000007f80c4aef8 in __GI___poll (fds=0x55943a95e0, nfds=547622323080, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000007f80d57f58 in  () at /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#2  0x0000005594391320 in  ()
Spinning.  Please run 'gdb gst-launch-1.0 8234' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

In dmesg I see only this:

...
Casual camera setting 
...
[ 4259.378287] imx477 30-0010: IMX477:: start streaming 0
[ 4259.406675] imx477 30-0010: IMX477::stop streaming
[ 4259.407854] imx477 30-0010: imx477_power_off:

Stream starts and after less than 30 ms it’s powered off.


Output of pipeline: L4T 32.7.5 with modified nvarguscamerasrc.so, IMX in slave mode.

GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
Caught SIGSEGV
#0  0x0000007f88c9cef8 in __GI___poll (fds=0x558e021c30, nfds=547756876680, timeout=<optimized out>) at ../sysdeps/unix/sysv/linux/poll.c:41
#1  0x0000007f88da9f58 in  () at /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0
#2  0x000000558dedb250 in  ()
Spinning.  Please run 'gdb gst-launch-1.0 5900' to continue debugging, Ctrl-C to quit, or Ctrl-\ to dump core.

In dmesg I see only this:

...
[ 1255.940558] imx477 30-0010: IMX477:: start streaming 1
[ 1255.986135] imx477 30-0010: IMX477::stop streaming
[ 1255.987230] imx477 30-0010: imx477_power_off:

################################## END OF EXPERIMENTS RESULTS ##################################

Ok we fixed the SIGSEGV, it was caused because libgstnvarguscamerasrc.so was built on L4T 32.4.4. Build on 32.7.5 fix this issue.

I successfully applied new lib with longer timeouts:

static const uint64_t WAIT_FOR_EVENT_TIMEOUT  = 30000000000;
static const uint64_t ACQUIRE_FRAME_TIMEOUT   = 50000000000;

But after incising 10x I still have following problem when running pipeline:

nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...

Any Idea how to solve this, or how to approach solving it?

hello NucleoIris,

such failure… nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
this is usually due to capture buffer allocation faliure.

let’s narrow down the issue,
could you please check Applications Using V4L2 IOCTL Directly to examine basic functionality with V4L2 IOCTL.

besides, please also check camera sensor streams frames continuously and finite timeouts.
you may also enable Infinite Timeout Support for confirmation.

Thanks JerryChang for the reply.
I found out (with dbg prints) that gstnvarguscamerasrc.cpp is failing on this condition:

if (iEvent->getEventType() == EVENT_TYPE_ERROR) inside the while loop of bool StreamConsumer::threadExecute(GstNvArgusCameraSrc *src)
I’m not sure what to do with that, this check wasn’t presented in previous version of nvarguscamerasrc.

=========================================================================================
I used the following command to check Applications Using V4L2 IOCTL Directly. It created just an empty file and it was stuck.
v4l2-ctl --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video2 --stream-to=imx477.raw

=========================================================================================
This is the output when I start the pipeline with enableCamInfiniteTimeout=1 nvargus-daemon.

=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[6938]: Connection established (7F7F4961D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[6938]: CameraProvider initialized (0x7f78928ff0)SCF: Error BadParameter:  (propagating from src/services/capture/NvViCsiHw.cpp, function openViCsi(), line 127)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function open(), line 309)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function openSource(), line 355)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function openSource(), line 485)
SCF: Error BadParameter:  (propagating from src/api/Session.cpp, function initialize(), line 268)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function createSession(), line 585)
(Argus) Error BadParameter:  (propagating from src/api/CaptureSessionImpl.cpp, function initialize(), line 123)
(Argus) Error BadParameter:  (propagating from src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 292)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/argus/src/api/CaptureSessionImpl.cpp:200) (in Mutex.cpp, function lock(), line 79)
(Argus) Error InvalidState: Element not found (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamerautils/inc/Vector.h, function remove(), line 172)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
=== gst-launch-1.0[6938]: CameraProvider destroyed (0x7f78928ff0)=== gst-launch-1.0[6938]: Connection closed (7F7F4961D0)=== gst-launch-1.0[6938]: Connection cleaned up (7F7F4961D0)

Meanwhile output of the pipeline is following:

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Error generated. gstnvarguscamerasrc.cpp, execute:774 Failed to create CaptureSession
Got EOS from element "pipeline0".
Execution ended after 0:00:00.448594375
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

=========================================================================
This is the output when I start the pipeline only with nvargus-daemon.

=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[6697]: Connection established (7F75A1A1D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[6697]: CameraProvider initialized (0x7f70994d10)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing againSCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 608)
Error: waitIspStatsFinished Something went wrong with waiting on stats
Error: waitIspStatsFinished Something went wrong with waiting on stats
SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 492)
SCF: Error InvalidState: Something went wrong with waiting on Isp frame end (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 550)
SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error InvalidState: Worker thread IspHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Error: waitCsiFrameStart timeout guid 2
************VI/CSI Debug Registers**********
VI_CFG_INTERRUPT_MASK_0 = 0x00000000
VI_CFG_INTERRUPT_STATUS_0 = 0x00000000
VI_CSI_0_ERROR_STATUS_0 = 0x00000000
VI_CSI_0_ERROR_INT_MASK_0 = 0x00000000
VI_CSI_1_ERROR_STATUS_0 = 0x00000000
VI_CSI_1_ERROR_INT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_A_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_A_STATUS_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_B_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_B_STATUS_0 = 0x00000000
CSI_CSI_CIL_A_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_CIL_A_STATUS_0 = 0x00000000
CSI_CSI_CILA_STATUS_0 = 0x00000000
CSI_CSI_CIL_B_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_CIL_B_STATUS_0 = 0x00000000
CSI_CSI_CILB_STATUS_0 = 0x00000000
CSI_CSI_READONLY_STATUS_0 = 0x00000000
CSI_DEBUG_CONTROL_0 = 0x7e271000
CSI_DEBUG_COUNTER_0_0 = 0x00000000
CSI_DEBUG_COUNTER_1_0 = 0x00000000
CSI_DEBUG_COUNTER_2_0 = 0x00000000
*****************************************
SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error Timeout: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceIsp.cpp, function waitCompletion(), line 423)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 951)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1034)
PowerServiceCore:handleRequests: timePassed = 3024
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc
=== gst-launch-1.0[6697]: Connection closed (7F75A1A1D0)=== gst-launch-1.0[6697]: There were pending client requests in the server at the time of CameraProvider destruction, and they failed to complete within a reasonable time (1500 ms). This may be due to a hardware or libargus error. Forced destruction will now proceed, which may leave the libargus server in a bad state.=== gst-launch-1.0[6697]: WARNING: CameraProvider was not destroyed before client connection terminated.=== gst-launch-1.0[6697]:          The client may have abnormally terminated. Destroying CameraProvider...=== gst-launch-1.0[6697]: CameraProvider destroyed (0x7f70994d10)=== gst-launch-1.0[6697]: WARNING: Cleaning up 1 outstanding requests...=== gst-launch-1.0[6697]: WARNING: Cleaning up 1 outstanding stream settings...=== gst-launch-1.0[6697]: WARNING: Cleaning up 1 outstanding queues...=== gst-launch-1.0[6697]: WARNING: Cleaning up 1 outstanding sessions...waitForIdleLocked remaining request 104
waitForIdleLocked remaining request 103
SCF: Error Timeout: waitForIdle() timed out (in src/api/Session.cpp, function waitForIdleLocked(), line 928)
(Argus) Error Timeout:  (propagating from src/api/CaptureSessionImpl.cpp, function destroy(), line 169)

hello NucleoIris,

let’s keep checking basic functionality with v4l2 IOCTL,
please setup terminal for running $ dmesg --follow to gather complete kernel logs.

BTW,
it looks you’ve verify the sensor stream with master mode on r32.7.5, right?
please also refer to Verifying the V4L2 Sensor Driver to examine your sensor driver.

Hello JerryChang,

Yes, I verified the functionality of the driver and the stream in master mode, everything is working in the master mode.

I rebooted the jetson and when I run again v4l2-ctl --set-fmt-video=width=4056,height=3040,pixelformat=RG10 --stream-mmap --stream-count=1 -d /dev/video2 --stream-to=imx477.raw
It created a file imx477.raw with size 24’709’120 B. I’m not sure why it wasn’t working previously. In the previous comment, I first started and stopped nvargus, maybe that’s why it wasn’t working on the first time.
Now I tried

  1. v4l2-ctl - working, creates a file with 24MB
  2. v4l2-ctl - working, creates a file with 24MB
  3. start nvargus
  4. start pipeline (fails)
  5. stop nvargus
  6. start `v4l2-ctl’ - not working, stuck and file has 0B

This is the compliance result for the camera in int slave mode. (It faild on fail: v4l2-test-formats.cpp(1184): ret && node->has_frmintervals test VIDIOC_G/S_PARM: FAIL).

v4l2-compliance SHA   : not available

Driver Info:
        Driver name   : tegra-video
        Card type     : vi-output, imx477 30-0010
        Bus info      : platform:54080000.vi:2
        Driver version: 4.9.255
        Capabilities  : 0x84200001
                Video Capture
                Streaming
                Extended Pix Format
                Device Capabilities
        Device Caps   : 0x04200001
                Video Capture
                Streaming
                Extended Pix Format

Compliance test for device /dev/video2 (not using libv4l2):

Required ioctls:
        test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
        test second video open: OK
        test VIDIOC_QUERYCAP: OK
        test VIDIOC_G/S_PRIORITY: OK
        test for unlimited opens: OK

Debug ioctls:
        test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
        test VIDIOC_LOG_STATUS: OK

Input ioctls:
        test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
        test VIDIOC_ENUMAUDIO: OK (Not Supported)
        test VIDIOC_G/S/ENUMINPUT: OK
        test VIDIOC_G/S_AUDIO: OK (Not Supported)
        Inputs: 1 Audio Inputs: 0 Tuners: 0

Output ioctls:
        test VIDIOC_G/S_MODULATOR: OK (Not Supported)
        test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
        test VIDIOC_ENUMAUDOUT: OK (Not Supported)
        test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
        test VIDIOC_G/S_AUDOUT: OK (Not Supported)
        Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
        test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
        test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
        test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
        test VIDIOC_G/S_EDID: OK (Not Supported)

Test input 0:

        Control ioctls:
                test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
                test VIDIOC_QUERYCTRL: OK
                test VIDIOC_G/S_CTRL: OK
                test VIDIOC_G/S/TRY_EXT_CTRLS: OK
                test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
                test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
                Standard Controls: 1 Private Controls: 18

        Format ioctls:
                test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
                fail: v4l2-test-formats.cpp(1184): ret && node->has_frmintervals
                test VIDIOC_G/S_PARM: FAIL
                test VIDIOC_G_FBUF: OK (Not Supported)
                test VIDIOC_G_FMT: OK
                test VIDIOC_TRY_FMT: OK
                test VIDIOC_S_FMT: OK
                test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
                test Cropping: OK (Not Supported)
                test Composing: OK (Not Supported)
                test Scaling: OK (Not Supported)

        Codec ioctls:
                test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
                test VIDIOC_G_ENC_INDEX: OK (Not Supported)
                test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

        Buffer ioctls:
                test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
                test VIDIOC_EXPBUF: OK

Test input 0:


Total: 43, Succeeded: 42, Failed: 1, Warnings: 0

Driver in master mode (which is working) fails also on the same test.


This of the output of the nvargus-daemon, with camera in master mode :

=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[5493]: Connection established (7F9A0801D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[5493]: CameraProvider initialized (0x7f94ae8f40)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing again

This of the output of the nvargus-daemon, with camera in slave mode :

=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[4953]: Connection established (7FA7DE31D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[4953]: CameraProvider initialized (0x7fa0ae9370)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing againSCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 608)
Error: waitIspStatsFinished Something went wrong with waiting on stats
SCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 492)
SCF: Error InvalidState: Something went wrong with waiting on Isp frame end (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 550)
SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error InvalidState: Worker thread IspHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
Error: waitIspStatsFinished Something went wrong with waiting on stats
Error: waitCsiFrameStart timeout guid 2
************VI/CSI Debug Registers**********
VI_CFG_INTERRUPT_MASK_0 = 0x00000000
VI_CFG_INTERRUPT_STATUS_0 = 0x00000000
VI_CSI_0_ERROR_STATUS_0 = 0x00000000
VI_CSI_0_ERROR_INT_MASK_0 = 0x00000000
VI_CSI_1_ERROR_STATUS_0 = 0x00000000
VI_CSI_1_ERROR_INT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_A_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_A_STATUS_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_B_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_B_STATUS_0 = 0x00000000
CSI_CSI_CIL_A_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_CIL_A_STATUS_0 = 0x00000000
CSI_CSI_CILA_STATUS_0 = 0x00000000
CSI_CSI_CIL_B_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_CIL_B_STATUS_0 = 0x00000000
CSI_CSI_CILB_STATUS_0 = 0x00000000
CSI_CSI_READONLY_STATUS_0 = 0x00000000
CSI_DEBUG_CONTROL_0 = 0x6e271000
CSI_DEBUG_COUNTER_0_0 = 0x00000000
CSI_DEBUG_COUNTER_1_0 = 0x00000000
CSI_DEBUG_COUNTER_2_0 = 0x00000000
*****************************************
SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error Timeout: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceIsp.cpp, function waitCompletion(), line 423)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 951)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1034)
PowerServiceCore:handleRequests: timePassed = 3029
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc

hello NucleoIris,

is it possible to evaluate how long it took for camera output frames to CSI channel in slave mode?

Thanks for answer JerryChang.

Not sure how to measure it, if you can provide me some guidance i can do it. (Accessing HW layer of MIPI would be very difficult).

I measured strobe signal from the IMX, in the slave mode:
When I used nvargus, the period between first two pulses is 582 ms (requested frame rate is 30Hz), and there are only two pulses on during running pipeline.

When I used v4l2-ctl, the period between first two pulses was 283 ms, there were again only two pulses and nothing else.Even if I set --stream-count=5 there are still just two pulses, but file is 5x larger.

Normally there should be one pulse for every provided frame. At the init there might be some extra pulses.

hello NucleoIris,

I assume it’s timeout due to 1st frame coming to Argus layer beyond the timeout limit (2500ms by default)
It’s doubted you did not enable infinite timeout correctly since the time out error still reported.

anyways, could you please also test with gst pipeline with nvarguscamerasrc plugin.
there’re added elements, event-wait and acquire-wait properties to make the frame output timeout configurable from the plugin.
for instance,
$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 event-wait=6000000000 acquire-wait=6000000000 ! "video/x-raw(memory:NVMM),framerate=22/1" ! nvvidconv ! xvimagesink

Hello JerryChang,

As I was writing in my first comment I tried that (based on another forum issue), but in v32.7.5 those parameters are not available from comandline.

pipeline:

# gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 event-wait=6000000000 acquire-wait=6000000000 ! "video/x-raw,width=4056,height=3040,framerate=30/1" ! multifilesink location=/media/sd_card/frames_test/slaveX_%03d.png

WARNING: erroneous pipeline: no property "event-wait" in element "nvarguscamerasrc0"

enableCamInfiniteTimeout=1 nvargus-daemon response:

root@localhost:/home/nano# enableCamInfiniteTimeout=1 nvargus-daemon
=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[6757]: Connection established (7FAD41C1D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[6757]: CameraProvider initialized (0x7fa8992210)=== gst-launch-1.0[6757]: CameraProvider destroyed (0x7fa8992210)=== gst-launch-1.0[6757]: Connection closed (7FAD41C1D0)=== gst-launch-1.0[6757]: Connection cleaned up (7FAD41C1D0)

Thats why i originaly tried to modifie nvarguscamerasrc.cpp to increase those timeouts there. It didn’t helped.


I run again enableCamInfiniteTimeout=1 nvargus-daemon and the pipeline.
Output of the pipeline :

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Error generated. gstnvarguscamerasrc.cpp, execute:774 Failed to create CaptureSession
Got EOS from element "pipeline0".
Execution ended after 0:00:00.421548698
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

output of the nvargus-daemon:

root@localhost:/home/nano# enableCamInfiniteTimeout=1 nvargus-daemon
=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[5255]: Connection established (7FB29EC1D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[5255]: CameraProvider initialized (0x7fac991790)SCF: Error BadParameter:  (propagating from src/services/capture/NvViCsiHw.cpp, function openViCsi(), line 127)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function open(), line 309)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function openSource(), line 355)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function openSource(), line 485)
SCF: Error BadParameter:  (propagating from src/api/Session.cpp, function initialize(), line 268)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function createSession(), line 585)
(Argus) Error BadParameter:  (propagating from src/api/CaptureSessionImpl.cpp, function initialize(), line 123)
(Argus) Error BadParameter:  (propagating from src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 292)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/argus/src/api/CaptureSessionImpl.cpp:200) (in Mutex.cpp, function lock(), line 79)
(Argus) Error InvalidState: Element not found (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamerautils/inc/Vector.h, function remove(), line 172)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
=== gst-launch-1.0[5255]: CameraProvider destroyed (0x7fac991790)=== gst-launch-1.0[5255]: Connection closed (7FB29EC1D0)=== gst-launch-1.0[5255]: Connection cleaned up (7FB29EC1D0)

could you please also share $ gst-inspect-1.0 nvarguscamerasrc for cross-check.

Sure, here it is:

Factory Details:
  Rank                     primary (256)
  Long-name                NvArgusCameraSrc
  Klass                    Video/Capture
  Description              nVidia ARGUS Camera Source
  Author                   Viranjan Pagar <vpagar@nvidia.com>, Amit Pandya <apandya@nvidia.com>

Plugin Details:
  Name                     nvarguscamerasrc
  Description              nVidia ARGUS Source Component
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
  Version                  1.0.0
  License                  Proprietary
  Source module            nvarguscamerasrc
  Binary package           NvARGUSCameraSrc
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstNvArgusCameraSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { (string)NV12, (string)P010_10LE }
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvarguscamerasrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1
  typefind            : Run typefind before negotiating (deprecated, non-functional)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: true
  silent              : Produce verbose output ?
                        flags: readable, writable
                        Boolean. Default: true
  timeout             : timeout to capture in seconds (Either specify timeout or num-buffers, not both)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  wbmode              : White balance affects the color temperature of the photo
                        flags: readable, writable
                        Enum "GstNvArgusCamWBMode" Default: 1, "auto"
                           (0): off              - GST_NVCAM_WB_MODE_OFF
                           (1): auto             - GST_NVCAM_WB_MODE_AUTO
                           (2): incandescent     - GST_NVCAM_WB_MODE_INCANDESCENT
                           (3): fluorescent      - GST_NVCAM_WB_MODE_FLUORESCENT
                           (4): warm-fluorescent - GST_NVCAM_WB_MODE_WARM_FLUORESCENT
                           (5): daylight         - GST_NVCAM_WB_MODE_DAYLIGHT
                           (6): cloudy-daylight  - GST_NVCAM_WB_MODE_CLOUDY_DAYLIGHT
                           (7): twilight         - GST_NVCAM_WB_MODE_TWILIGHT
                           (8): shade            - GST_NVCAM_WB_MODE_SHADE
                           (9): manual           - GST_NVCAM_WB_MODE_MANUAL
  saturation          : Property to adjust saturation value
                        flags: readable, writable
                        Float. Range:               0 -               2 Default:               1
  sensor-id           : Set the id of camera sensor to use. Default 0.
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 0
  sensor-mode         : Set the camera sensor mode to use. Default -1 (Select the best match)
                        flags: readable, writable
                        Integer. Range: -1 - 255 Default: -1
  total-sensor-modes  : Query the number of sensor modes available. Default 0
                        flags: readable
                        Integer. Range: 0 - 255 Default: 0
  exposuretimerange   : Property to adjust exposure time range in nanoseconds
                        Use string with values of Exposure Time Range (low, high)
                        in that order, to set the property.
                        eg: exposuretimerange="34000 358733000"
                        flags: readable, writable
                        String. Default: null
  gainrange           : Property to adjust gain range
                        Use string with values of Gain Time Range (low, high)
                        in that order, to set the property.
                        eg: gainrange="1 16"
                        flags: readable, writable
                        String. Default: null
  ispdigitalgainrange : Property to adjust digital gain range
                        Use string with values of ISP Digital Gain Range (low, high)
                        in that order, to set the property.
                        eg: ispdigitalgainrange="1 8"
                        flags: readable, writable
                        String. Default: null
  tnr-strength        : property to adjust temporal noise reduction strength
                        flags: readable, writable
                        Float. Range:              -1 -               1 Default:              -1
  tnr-mode            : property to select temporal noise reduction mode
                        flags: readable, writable
                        Enum "GstNvArgusCamTNRMode" Default: 1, "NoiseReduction_Fast"
                           (0): NoiseReduction_Off - GST_NVCAM_NR_OFF
                           (1): NoiseReduction_Fast - GST_NVCAM_NR_FAST
                           (2): NoiseReduction_HighQuality - GST_NVCAM_NR_HIGHQUALITY
  ee-mode             : property to select edge enhnacement mode
                        flags: readable, writable
                        Enum "GstNvArgusCamEEMode" Default: 1, "EdgeEnhancement_Fast"
                           (0): EdgeEnhancement_Off - GST_NVCAM_EE_OFF
                           (1): EdgeEnhancement_Fast - GST_NVCAM_EE_FAST
                           (2): EdgeEnhancement_HighQuality - GST_NVCAM_EE_HIGHQUALITY
  ee-strength         : property to adjust edge enhancement strength
                        flags: readable, writable
                        Float. Range:              -1 -               1 Default:              -1
  aeantibanding       : property to set the auto exposure antibanding mode
                        flags: readable, writable
                        Enum "GstNvArgusCamAeAntiBandingMode" Default: 1, "AeAntibandingMode_Auto"
                           (0): AeAntibandingMode_Off - GST_NVCAM_AEANTIBANDING_OFF
                           (1): AeAntibandingMode_Auto - GST_NVCAM_AEANTIBANDING_AUTO
                           (2): AeAntibandingMode_50HZ - GST_NVCAM_AEANTIBANDING_50HZ
                           (3): AeAntibandingMode_60HZ - GST_NVCAM_AEANTIBANDING_60HZ
  exposurecompensation: property to adjust exposure compensation
                        flags: readable, writable
                        Float. Range:              -2 -               2 Default:               0
  aelock              : set or unset the auto exposure lock
                        flags: readable, writable
                        Boolean. Default: false
  aeregion            : Property to set region of interest for auto exposure
                        with values of ROI coordinates (left, top, right, bottom)
                        and weight (float number) in that order, to set the property
                        use for example: aeregion="0 0 256 256 1"
                        flags: readable, writable
                        String. Default: null
  awblock             : set or unset the auto white balance lock
                        flags: readable, writable
                        Boolean. Default: false
  bufapi-version      : set to use new Buffer API
                        flags: readable, writable
                        Boolean. Default: false

If I don’t start nvargus-daemon there is also this error at the beginning of the print.(Which make sense)

root@localhost:/home/nano# gst-inspect-1.0 nvarguscamerasrc
(Argus) Error FileOperationFailed: Connecting to nvargus-daemon failed: No such file or directory (in src/rpc/socket/client/SocketClientDispatch.cpp, function openSocketConnection(), line 205)
(Argus) Error FileOperationFailed: Cannot create camera provider (in src/rpc/socket/client/SocketClientDispatch.cpp, function createCameraProvider(), line 106)

hello NucleoIris,

please give it a try to update libgstnvarguscamerasrc.so and libnvscf.so with attachment based-on r32.7.5 release version.
for instance, Topic301820_Aug06.7z (1.9 MB)

Hello JerryChang,

I updated those two files with the one you provided.

EDIT:
I replace following files:
/usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
/usr/lib/aarch64-linux-gnu/tegra/libnvscf.so
END OF EDIT

In the print of gst-inspect-1.0 nvarguscamerasrc I can now see parameters event-waitand acquire-wait.

Factory Details:
  Rank                     primary (256)
  Long-name                NvArgusCameraSrc
  Klass                    Video/Capture
  Description              nVidia ARGUS Camera Source
  Author                   Viranjan Pagar <vpagar@nvidia.com>, Amit Pandya <apandya@nvidia.com>

Plugin Details:
  Name                     nvarguscamerasrc
  Description              nVidia ARGUS Source Component
  Filename                 /usr/lib/aarch64-linux-gnu/gstreamer-1.0/libgstnvarguscamerasrc.so
  Version                  1.0.0
  License                  Proprietary
  Source module            nvarguscamerasrc
  Binary package           NvARGUSCameraSrc
  Origin URL               http://nvidia.com/

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseSrc
                         +----GstNvArgusCameraSrc

Pad Templates:
  SRC template: 'src'
    Availability: Always
    Capabilities:
      video/x-raw(memory:NVMM)
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
                 format: { (string)NV12, (string)P010_10LE }
              framerate: [ 0/1, 2147483647/1 ]

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "nvarguscamerasrc0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  blocksize           : Size in bytes to read per buffer (-1 = default)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 4294967295 Default: 4096
  num-buffers         : Number of buffers to output before sending EOS (-1 = unlimited)
                        flags: readable, writable
                        Integer. Range: -1 - 2147483647 Default: -1
  typefind            : Run typefind before negotiating (deprecated, non-functional)
                        flags: readable, writable, deprecated
                        Boolean. Default: false
  do-timestamp        : Apply current stream time to buffers
                        flags: readable, writable
                        Boolean. Default: true
  silent              : Produce verbose output ?
                        flags: readable, writable
                        Boolean. Default: true
  timeout             : timeout to capture in seconds (Either specify timeout or num-buffers, not both)
                        flags: readable, writable
                        Unsigned Integer. Range: 0 - 2147483647 Default: 0
  wbmode              : White balance affects the color temperature of the photo
                        flags: readable, writable
                        Enum "GstNvArgusCamWBMode" Default: 1, "auto"
                           (0): off              - GST_NVCAM_WB_MODE_OFF
                           (1): auto             - GST_NVCAM_WB_MODE_AUTO
                           (2): incandescent     - GST_NVCAM_WB_MODE_INCANDESCENT
                           (3): fluorescent      - GST_NVCAM_WB_MODE_FLUORESCENT
                           (4): warm-fluorescent - GST_NVCAM_WB_MODE_WARM_FLUORESCENT
                           (5): daylight         - GST_NVCAM_WB_MODE_DAYLIGHT
                           (6): cloudy-daylight  - GST_NVCAM_WB_MODE_CLOUDY_DAYLIGHT
                           (7): twilight         - GST_NVCAM_WB_MODE_TWILIGHT
                           (8): shade            - GST_NVCAM_WB_MODE_SHADE
                           (9): manual           - GST_NVCAM_WB_MODE_MANUAL
  saturation          : Property to adjust saturation value
                        flags: readable, writable
                        Float. Range:               0 -               2 Default:               1
  sensor-id           : Set the id of camera sensor to use. Default 0.
                        flags: readable, writable
                        Integer. Range: 0 - 255 Default: 0
  sensor-mode         : Set the camera sensor mode to use. Default -1 (Select the best match)
                        flags: readable, writable
                        Integer. Range: -1 - 255 Default: -1
  total-sensor-modes  : Query the number of sensor modes available. Default 0
                        flags: readable
                        Integer. Range: 0 - 255 Default: 0
  exposuretimerange   : Property to adjust exposure time range in nanoseconds
                        Use string with values of Exposure Time Range (low, high)
                        in that order, to set the property.
                        eg: exposuretimerange="34000 358733000"
                        flags: readable, writable
                        String. Default: null
  gainrange           : Property to adjust gain range
                        Use string with values of Gain Time Range (low, high)
                        in that order, to set the property.
                        eg: gainrange="1 16"
                        flags: readable, writable
                        String. Default: null
  ispdigitalgainrange : Property to adjust digital gain range
                        Use string with values of ISP Digital Gain Range (low, high)
                        in that order, to set the property.
                        eg: ispdigitalgainrange="1 8"
                        flags: readable, writable
                        String. Default: null
  tnr-strength        : property to adjust temporal noise reduction strength
                        flags: readable, writable
                        Float. Range:              -1 -               1 Default:              -1
  tnr-mode            : property to select temporal noise reduction mode
                        flags: readable, writable
                        Enum "GstNvArgusCamTNRMode" Default: 1, "NoiseReduction_Fast"
                           (0): NoiseReduction_Off - GST_NVCAM_NR_OFF
                           (1): NoiseReduction_Fast - GST_NVCAM_NR_FAST
                           (2): NoiseReduction_HighQuality - GST_NVCAM_NR_HIGHQUALITY
  ee-mode             : property to select edge enhnacement mode
                        flags: readable, writable
                        Enum "GstNvArgusCamEEMode" Default: 1, "EdgeEnhancement_Fast"
                           (0): EdgeEnhancement_Off - GST_NVCAM_EE_OFF
                           (1): EdgeEnhancement_Fast - GST_NVCAM_EE_FAST
                           (2): EdgeEnhancement_HighQuality - GST_NVCAM_EE_HIGHQUALITY
  ee-strength         : property to adjust edge enhancement strength
                        flags: readable, writable
                        Float. Range:              -1 -               1 Default:              -1
  aeantibanding       : property to set the auto exposure antibanding mode
                        flags: readable, writable
                        Enum "GstNvArgusCamAeAntiBandingMode" Default: 1, "AeAntibandingMode_Auto"
                           (0): AeAntibandingMode_Off - GST_NVCAM_AEANTIBANDING_OFF
                           (1): AeAntibandingMode_Auto - GST_NVCAM_AEANTIBANDING_AUTO
                           (2): AeAntibandingMode_50HZ - GST_NVCAM_AEANTIBANDING_50HZ
                           (3): AeAntibandingMode_60HZ - GST_NVCAM_AEANTIBANDING_60HZ
  exposurecompensation: property to adjust exposure compensation
                        flags: readable, writable
                        Float. Range:              -2 -               2 Default:               0
  aelock              : set or unset the auto exposure lock
                        flags: readable, writable
                        Boolean. Default: false
  aeregion            : Property to set region of interest for auto exposure
                        with values of ROI coordinates (left, top, right, bottom)
                        and weight (float number) in that order, to set the property
                        use for example: aeregion="0 0 256 256 1"
                        flags: readable, writable
                        String. Default: null
  awblock             : set or unset the auto white balance lock
                        flags: readable, writable
                        Boolean. Default: false
  bufapi-version      : set to use new Buffer API
                        flags: readable, writable
                        Boolean. Default: false
  event-wait          : Set the waiting time for the event before timeout. Default 3000000000 (in nanosec)
                        flags: readable, writable
                        Unsigned Integer64. Range: 3000000000 - 18446744073709551615 Default: 3000000000
  acquire-wait        : Set the waiting time for the acquireFrame before timeout. Default 5000000000 (in nanosec)
                        flags: readable, writable
                        Unsigned Integer64. Range: 5000000000 - 18446744073709551615 Default: 5000000000

I started enableCamInfiniteTimeout=1 nvargus-daemon and run a pipepline with those parameters:

pipeline:

gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 event-wait=6000000000 acquire-wait=6000000000 ! "video/x-raw(memory:NVMM),width=4056,height=3040,framerate=30/1" ! nvvidconv ! pngenc ! multifilesink location=/media/sd_card/frames_test/slaveX_%03d.png

It had crashed and this was the output of the pipeline

Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Error generated. /home/jerry/L4T/t186/r32.7.x/multimedia/nvgstreamer/gst-nvarguscamera/gstnvarguscamerasrc.cpp, execute:751 Failed to create CaptureSession
Got EOS from element "pipeline0".
Execution ended after 0:00:00.472993646
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

In the meantime this was in log of nvargus:

root@localhost:/home/nano# enableCamInfiniteTimeout=1 nvargus-daemon
=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[6031]: Connection established (7F840041D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[6031]: CameraProvider initialized (0x7f7cae99b0)SCF: Error BadParameter:  (propagating from src/services/capture/NvViCsiHw.cpp, function openViCsi(), line 127)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function open(), line 309)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function openSource(), line 355)
SCF: Error BadParameter:  (propagating from src/services/capture/CaptureService.cpp, function openSource(), line 485)
SCF: Error BadParameter:  (propagating from src/api/Session.cpp, function initialize(), line 268)
SCF: Error BadParameter:  (propagating from src/api/CameraDriver.cpp, function createSession(), line 585)
(Argus) Error BadParameter:  (propagating from src/api/CaptureSessionImpl.cpp, function initialize(), line 123)
(Argus) Error BadParameter:  (propagating from src/api/CameraProviderImpl.cpp, function createCaptureSessionInternal(), line 292)
(NvCameraUtils) Error InvalidState: Mutex not initialized (/dvs/git/dirty/git-master_linux/camera/argus/src/api/CaptureSessionImpl.cpp:200) (in Mutex.cpp, function lock(), line 79)
(Argus) Error InvalidState: Element not found (in /dvs/git/dirty/git-master_linux/camera/utils/nvcamerautils/inc/Vector.h, function remove(), line 172)
(NvCameraUtils) Error InvalidState: Mutex has not been initialized (in Mutex.cpp, function unlock(), line 88)
=== gst-launch-1.0[6031]: CameraProvider destroyed (0x7f7cae99b0)=== gst-launch-1.0[6031]: Connection closed (7F840041D0)=== gst-launch-1.0[6031]: Connection cleaned up (7F840041D0)

In dmesg there is typical setup of the camera and stop in 46 ms

... setup of IMX ...
[ 1158.705475] imx477 30-0010: IMX477:: start streaming 1
[ 1158.751031] imx477 30-0010: IMX477::stop streaming
[ 1158.751679] imx477 30-0010: imx477_power_off:

My previous comment was using enableCamInfiniteTimeout=1 nvargus-daemon.
If I do same experiment but run just nvargus-daemon, pipeline still fails but it gets a bit further.

Pipeline:

root@localhost:/home/nano# gst-launch-1.0 -e nvarguscamerasrc sensor-id=2 event-wait=6000000000 acquire-wait=6000000000 ! "video/x-raw(memory:NVMM),width=4056,height=3040,framerate=30/1" ! nvvidconv ! pngenc ! multifilesink location=/media/sd_card/frames_test/slaveX_%03d.png
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 4056 x 3040 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 22000, max 333333000;

GST_ARGUS: 3840 x 2160 FR = 59.999999 fps Duration = 16666667 ; Analog Gain range min 1.000000, max 16.000000; Exposure Range min 55000, max 333333000;

GST_ARGUS: Running with following settings:
   Camera index = 2
   Camera mode  = 0
   Output Stream W = 4056 H = 3040
   seconds to Run    = 0
   Frame Rate = 29.999999
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
nvbuf_utils: dmabuf_fd -1 mapped entry NOT found
nvbuf_utils: Can not get HW buffer from FD... Exiting...
CONSUMER: ERROR OCCURRED
ERROR: from element /GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0: CANCELLED
Additional debug info:
Argus Error Status
EOS on shutdown enabled -- waiting for EOS after Error
Waiting for EOS...

nvargus-daemon output:

root@localhost:/home/nano# nvargus-daemon
=== NVIDIA Libargus Camera Service (0.98.3)=== Listening for connections...=== gst-launch-1.0[4926]: Connection established (7F944101D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: NVIDIA Camera virtual enumerator not found in proc device-tree
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing againLSC: LSC surface is not based on full res!
---- imager: Found override file [/var/nvidia/nvcam/settings/camera_overrides.isp]. ----
CAM: serial no file already exists, skips storing again=== gst-launch-1.0[4926]: CameraProvider initialized (0x7f8c991920)SCF: Error BadValue: NvPHSSendThroughputHints (in src/common/CameraPowerHint.cpp, function sendCameraPowerHint(), line 56)
CAM: serial no file already exists, skips storing againSCF: Error Timeout: ISP port 0 timed out! (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 492)
SCF: Error InvalidState: Something went wrong with waiting on Isp frame end (in src/services/capture/NvIspHw.cpp, function waitIspFrameEnd(), line 550)
SCF: Error InvalidState:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error InvalidState: Worker thread IspHw frameComplete failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout: ISP Stats timed out! (in src/services/capture/NvIspHw.cpp, function waitIspStatsFinished(), line 608)
Error: waitIspStatsFinished Something went wrong with waiting on stats
Error: waitIspStatsFinished Something went wrong with waiting on stats
Error: waitCsiFrameStart timeout guid 2
************VI/CSI Debug Registers**********
VI_CFG_INTERRUPT_MASK_0 = 0x00000000
VI_CFG_INTERRUPT_STATUS_0 = 0x00000000
VI_CSI_0_ERROR_STATUS_0 = 0x00000000
VI_CSI_0_ERROR_INT_MASK_0 = 0x00000000
VI_CSI_1_ERROR_STATUS_0 = 0x00000000
VI_CSI_1_ERROR_INT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_A_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_A_STATUS_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_B_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_PIXEL_PARSER_B_STATUS_0 = 0x00000000
CSI_CSI_CIL_A_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_CIL_A_STATUS_0 = 0x00000000
CSI_CSI_CILA_STATUS_0 = 0x00000000
CSI_CSI_CIL_B_INTERRUPT_MASK_0 = 0x00000000
CSI_CSI_CIL_B_STATUS_0 = 0x00000000
CSI_CSI_CILB_STATUS_0 = 0x00000000
CSI_CSI_READONLY_STATUS_0 = 0x00000000
CSI_DEBUG_CONTROL_0 = 0x6a271000
CSI_DEBUG_COUNTER_0_0 = 0x00000000
CSI_DEBUG_COUNTER_1_0 = 0x00000000
CSI_DEBUG_COUNTER_2_0 = 0x00000000
*****************************************
SCF: Error Timeout:  (propagating from src/common/Utils.cpp, function workerThread(), line 116)
SCF: Error Timeout: Worker thread ViCsiHw frameStart failed (in src/common/Utils.cpp, function workerThread(), line 133)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceIsp.cpp, function waitCompletion(), line 423)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 951)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1034)
PowerServiceCore:handleRequests: timePassed = 3033
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceEvent.cpp, function wait(), line 59)
Error: Camera HwEvents wait, this may indicate a hardware timeout occured,abort current/incoming cc

dmesg:

... setup of IMX ...
[  103.955388] imx477 30-0010: imx477_set_frame_rate: val: 30000000
[  103.956333] imx477 30-0010: IMX477:: start streaming 1
[  104.006563] imx477 30-0010: imx477_set_gain :val = 1000099, gain = 1
[  104.041987] imx477 30-0010: imx477_set_gain :val = 1000000, gain = 0
[  104.044229] imx477 30-0010: imx477_set_gain :val = 1000099, gain = 1
[  105.531742] fence timeout on [ffffffc0f6ec0f00] after 1500ms
[  105.531783] fence timeout on [ffffffc0d93930c0] after 1500ms
[  105.531832] name=[nvhost_sync:3], current value=0 waiting value=1
[  105.531854] name=[nvhost_sync:4], current value=0 waiting value=1
[  105.531893] ---- mlocks ----
[  105.531904] ---- mlocks ----


[  105.531973] ---- syncpts ----
[  105.531982] ---- syncpts ----
[  105.532044] id 3 (54600000.isp_0) min 0 max 3 refs 4 (previous client : )
... rest of nvargus crash ...

hello NucleoIris,

this is due to VI/CSI already in error state, it’s not even wait for sync point to fetch frames.

Hello JerryChang,
I started nvargus and pipeline from clear boot. There wasn’t any application running, do you think the problem is in probing of the sensor? Because probing was successful.

Our use case uses camera in slave mode which is triggered by synchronization device at 30 Hz. I would guess that problem may be in nvargus where it’s waiting for the first frame. Based on our use-case this delay can be up to 66 ms (2/FPS). Is this delay acceptable for nvargus?

Is src/services/capture/NvIspHw.cppavailable? So I could understand bit more the errror?

Do you know what should be next step in troubleshooting this problem?

hello NucleoIris,

no, it’s expecting camera frames arrived when camera app executed.
I saw some camera use-case with external trigger, however, they’re tested on rel-35 or later.

sorry, we don’t public release the camera sources.

Thanks for answer JerryChang,

We are using Jetson Nano so we can’t go on L4T v35.
As I said the beginning it was working with L4T 32.4.4, but not on L4T 32.7.5. So maybe some timeout check was added. If so would it be possible to provide us nvargus with extended timeouts?