Camera capture works on v4l2-ctl but hangs with nvarguscamerasrc

fabian.solano · May 5, 2021, 2:51pm

Hello,

Currently I am working in a camera project with the following setup:

Jetson AGX Xavier Developer Kit
GMSL2 (Serializer: MAX9295, Deserializer: MAX96712)
Camera AR0234

I have 3 cameras connected to a single deserializer. The drivers are working for ser/des and for the camera.

However I am having the following behavior:

1- Load the driver, capture with v4l2-ctl from 3 cameras simultaneously works correctly, stop capture, restart capture works correctly.

2- Load the driver, capture with v4l2-ctl from 3 cameras simultaneously works correctly, stop capture, try to capture from GStreamer using nvarguscamerasrc, does not work. The pipeline just hangs and no frames are received.

3- Load the driver, capture with GStreamer using nvarguscamerasrc from 3 cameras simultaneously works correctly, stop capture, try to capture again with GStreamer using nvarguscamerasrc, does not work. The pipeline just hangs and no frames are received.

Conclusion: After the first capture, no matter the method, capture again from nvarguscamerasrc is failing. If the first capture is with v4l2-ctl I can stop and restart the capture without problems.

---- Logs -----

Log from nvargusdaemon (correct first capture with GStreamer):

root@localhost:/home/nvidia# /usr/sbin/nvargus-daemon
=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== gst-launch-1.0[8659]: Connection established (7FABF8E1D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: virtual device driver node not found in proc device-tree: /proc/device-tree/tegra-virtual-camera-platform/modules/module0/drivernode1
OFParserGetVirtualDevice: virtual device driver node not found in proc device-tree: /proc/device-tree/tegra-virtual-camera-platform/modules/module1/drivernode0
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
=== gst-launch-1.0[8659]: CameraProvider initialized (0x7fa4964a20)LSC: LSC surface is not based on full res!
=== gst-launch-1.0[8659]: CameraProvider destroyed (0x7fa4964a20)=== gst-launch-1.0[8659]: Connection closed (7FABF8E1D0)=== gst-launch-1.0[8659]: Connection cleaned up (7FABF8E1D0)

Log from nvargusdaemon (failed second capture with GStreamer):

root@localhost:/home/nvidia# /usr/sbin/nvargus-daemon
=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== gst-launch-1.0[8759]: Connection established (7F8A6191D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: virtual device driver node not found in proc device-tree: /proc/device-tree/tegra-virtual-camera-platform/modules/module0/drivernode1
OFParserGetVirtualDevice: virtual device driver node not found in proc device-tree: /proc/device-tree/tegra-virtual-camera-platform/modules/module1/drivernode0
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
=== gst-launch-1.0[8759]: CameraProvider initialized (0x7f84b85ed0)LSC: LSC surface is not based on full res!
(NvCapture) Error Timeout: capture waiting for status timeout (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture_hw_linux.c, function NvCaptureLinuxGetCaptureStatus(), line 362)
(NvCapture) Error Timeout:  (propagating from /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureDequeue(), line 1055)
SCF: Error Timeout:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 851)
SCF: Error InvalidState: Timeout waiting on frame end sensor guid 0, capture sequence ID = 1, channel = 1/1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 858)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 966)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 335)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 938)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1032)
PowerServiceCore:handleRequests: timePassed = 3030
(NvCapture) Error Timeout: capture waiting for status timeout (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture_hw_linux.c, function NvCaptureLinuxGetCaptureStatus(), line 362)
(NvCapture) Error Timeout:  (propagating from /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureDequeue(), line 1055)
SCF: Error Timeout:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 851)
SCF: Error InvalidState: Timeout waiting on frame end sensor guid 0, capture sequence ID = 4, channel = 1/1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 858)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 4 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 966)
PowerServiceCore:handleRequests: timePassed = 1506

If any other log information is needed, please let me know.

Thanks.

Regards,
Fabian
www.ridgerun.com

ShaneCCC · May 6, 2021, 3:27am

Suppose it could be the MAX9xxx chip not reset correctly or something wrong cause it.

ShaneCCC · May 6, 2021, 3:35am

How about the case only one sensor start/stop capture by argus?

fabian.solano · May 6, 2021, 5:20pm

Hi @ShaneCCC ,

With one sensor the result is the same.

1- Load the driver, capture with v4l2-ctl from 1 camera works correctly, stop capture, restart capture works correctly.

2- Load the driver, capture with v4l2-ctl from 1 camera works correctly, stop capture, try to capture from GStreamer using nvarguscamerasrc, does not work . The pipeline just hangs and no frames are received.

3- Load the driver, capture with GStreamer using nvarguscamerasrc from 1 camera works correctly, stop capture, try to capture again with GStreamer using nvarguscamerasrc, does not work . The pipeline just hangs and no frames are received.

Regards,

ShaneCCC · May 7, 2021, 4:18am

Could you check if restart nvargus-daemon before restart the gstreamer pipeline.

fabian.solano · May 10, 2021, 4:11pm

Hi @ShaneCCC ,

If I restart nvargus-daemon it does not make any difference. I tried with:

sudo service nvargus-daemon stop
sudo service nvargus-daemon start

Also to enable the log, as before, I used:

sudo su
service nvargus-daemon stop
export enableCamPclLogs=1
export enableCamScfLogs=1
/usr/sbin/nvargus-daemon

Try to capture again with a bare minimum pipeline:

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! nvvidconv ! fakesink silent=false -v

And it gives the following result:

=== NVIDIA Libargus Camera Service (0.97.3)=== Listening for connections...=== gst-launch-1.0[10008]: Connection established (7F8EF061D0)OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module0
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module1
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module2
OFParserListModules: module list: /proc/device-tree/tegra-camera-platform/modules/module3
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
NvPclHwGetModuleList: WARNING: Could not map module to ISP config string
NvPclHwGetModuleList: No module data found
OFParserGetVirtualDevice: virtual device driver node not found in proc device-tree: /proc/device-tree/tegra-virtual-camera-platform/modules/module0/drivernode1
OFParserGetVirtualDevice: virtual device driver node not found in proc device-tree: /proc/device-tree/tegra-virtual-camera-platform/modules/module1/drivernode0
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
=== gst-launch-1.0[10008]: CameraProvider initialized (0x7f88964b30)LSC: LSC surface is not based on full res!
(NvCapture) Error Timeout: capture waiting for status timeout (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture_hw_linux.c, function NvCaptureLinuxGetCaptureStatus(), line 362)
(NvCapture) Error Timeout:  (propagating from /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureDequeue(), line 1055)
SCF: Error Timeout:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 851)
SCF: Error InvalidState: Timeout waiting on frame end sensor guid 0, capture sequence ID = 1, channel = 1/1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 858)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 966)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 335)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 938)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1032)
PowerServiceCore:handleRequests: timePassed = 3019
(NvCapture) Error Timeout: capture waiting for status timeout (in /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture_hw_linux.c, function NvCaptureLinuxGetCaptureStatus(), line 362)
(NvCapture) Error Timeout:  (propagating from /dvs/git/dirty/git-master_linux/camera/capture/nvcapture/capture.c, function NvCaptureDequeue(), line 1055)
SCF: Error Timeout:  (propagating from src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 851)
SCF: Error InvalidState: Timeout waiting on frame end sensor guid 0, capture sequence ID = 4, channel = 1/1 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 858)
SCF: Error InvalidState: Something went wrong with waiting on frame end (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameEnd(), line 939)
SCF: Error InvalidState: Timeout waiting on frame start sensor guid 0, capture sequence ID = 4 (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 966)
SCF: Error InvalidState: Something went wrong with waiting on frame start (in src/services/capture/NvCaptureViCsiHw.cpp, function waitCsiFrameStart(), line 1021)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDeviceViCsi.cpp, function waitCompletion(), line 335)
SCF: Error Timeout:  (propagating from src/services/capture/CaptureServiceDevice.cpp, function pause(), line 938)
SCF: Error Timeout: During capture abort, syncpoint wait timeout waiting for current frame to finish (in src/services/capture/CaptureServiceDevice.cpp, function handleCancelSourceRequests(), line 1032)
PowerServiceCore:handleRequests: timePassed = 3007

What I really dont understand is why with v4l2-ctl I cant stop/start pipe without any problems. As it is the same driver and it should it same start/stop sequence from enabling camera capture.

Regards,
Fabian
www.ridgerun.com

ShaneCCC · May 11, 2021, 3:48am

Looks like the MAX929x doesn’t reset well cause it. Is it possible bypass MAX929x to check?

fabian.solano · May 13, 2021, 7:48pm

Hi @ShaneCCC ,

What do you mean with bypass MAX929x?

Regards,

fabian.solano · May 13, 2021, 8:38pm

@ShaneCCC

I did a test killing nvargus-daemon and trying to capture again from v4l2-ctl and it works. Seems the problem is related to GStreamer and nvarguscamerasrc.

I am on Jetpack 4.4.

Also to provide more information this issue began to happen when enabling the support on ser/des to use Virtual Channel and Metadata.

Regards,

ShaneCCC · May 14, 2021, 2:49am

Bypass MAX929x mean connect without ser/des

fabian.solano · May 14, 2021, 2:29pm

@ShaneCCC

Our hardware only supports ser/des interface. Is there any other test we can do?

Regards,

ShaneCCC · May 17, 2021, 6:51am

Can try if ser/des output test pattern to check?

kayccc · May 26, 2021, 5:37am

Hi fabian.solano,

Any test result can be shared?

fabian.solano · May 26, 2021, 9:12pm

Hello @ShaneCCC and @kayccc ,

Thanks for the follow up on this topic. I have been doing a lot of tests.

1- The test pattern does not change the behavior.

2- I started checking the “reset” idea that you mentioned before.

It seems it has something to do with this. I added a one-shot reset config by writing to the deserializer on the camera driver after the start_stream/stop_stream functions are called.

Also, as seen on the following video, I have to do a one-shot reset to the serializer also for it to work. The behavior is the same, the first capture with GStreamer after the driver load works perfectly. But after the first time I have to do the one-reset thing.

The problem is that this approach does not work once I add the same I2C write on the driver.

Do you think I has something to do with the timing of when this I2C write is called?
Again the problem is related to GStreamer with nvarguscamerasrc. It never happens when using v4l-ctl, which makes me think I has something to do to the way argus reset/start/stop the stream.

Also the approach with the one-shot reset through I2C write on user space does not always work, after a pipeline has failed and I had to restart nvargus-daemon. Only after a board reboot it works again. If wait too much before writing the I2C one-shot, it will not work, and then I would have to restart the daemon.

Any other guess would be very valuable to us, as this is currently blocking our project.

Thanks.

ShaneCCC · May 27, 2021, 4:58am

Have you try infinite timeout?

sudo service nvargus-daemon stop
sudo enableCamInfiniteTimeout=1 nvargus-daemon

fabian.solano · June 3, 2021, 8:19pm

Hi @ShaneCCC ,

Using enableCamInfiniteTimeout allows me to use the I2C one-shot reset from userspace any time. So that definitely worked for testing. But I am still not able to add that to the driver.

I have noticed that it might be something related to timing when GStreamer nvargus is initializing if the driver does the one-shot reset it does not work.

Where can I add in the capture subsystem a call to a custom driver function to call after nvargus has ended with its initialization process?

Regards,
Fabian

ShaneCCC · June 4, 2021, 2:56am

It’s could be the sensor set mode consume much time cause it. Have a try add set_mode_delay_ms in the device tree to try.

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/camera_sensor_prog.html#wwpID0E0Q50HA

fabian.solano · June 11, 2021, 3:13pm

Hi @ShaneCCC ,

I managed to finally add the one-shot reset into the driver. I had to create a I2C sub device to access the serializer and correctly do the one-shot reset before start_stream.

Thanks for your help.