Camera Provider Unavailable

If I have a program using Argus that crashes for some reason that doesn’t allow the destructor on the “CameraProvider” to be run, all future calls to “Argus::CameraProvider::create” fail with status code 3 (unavailable). So far my only way to fix the problem has been rebooting the TX2, which makes me think that there are some zombie processes or a driver that need to be explicitly released before being able to create a new “CameraProvider”.

Does anyone know of any way to get Argus::CameraProvider::create to succeed again after a hard crash? Preferably from C++ so that I don’t need to create some kind of wrapping script around this process.

So I isolated the specific problem I was having that was causing this issue. I believe that there is a bug in the Argus daemon that causes it to block while destroying sessions in specific cases. The daemon being blocked was causing a timeout while attempting to create the “CameraProvider”

Below are the logs I got from the Argus daemon during testing with annotations

nvidia@dnncam-0:~$ sudo argus_daemon
=== Listening for connections... ===
=== Connection 7FA00451E0 established === <- starting our program normally
PCLHW_DTParser
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16LoadOverridesFile: looking for override file [/data/nvcam/settings/imx274_bottom_A6V26.isp] 6/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx274_bottom_A6V26.isp] 7/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx274_bottom_A6V26.isp] 8/16---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
CameraProvider result: provider=0x7f98bc6000, shim=0x7f98bbaa40, status=0, rpc status=1, size=9
LSC: LSC surface is not based on full res!
PowerServiceCore:handleRequests: timePassed = 704
SCF: Error InvalidState: NonFatal ISO BW requested not set. Requested = 4989600 Set = 4687500 (in src/services/power/PowerServiceCore.cpp, function setCameraBw(), line 474)
/== CLEANUP 0x7f98bbaa40 ==\ <- ctrl-c'ing the node
Destroying real provider 0x7f98bc6000
\== CLEANUP DONE ==/
(Argus) Error EndOfFile: (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function recvThreadCore(), line 138)
(Argus) Error EndOfFile: (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function run(), line 56)
=== Connection 7FA00451E0 closed ===
Cleaning up 0 requests...
Cleaning up 0 queues...
Cleaning up 0 streams...
Cleaning up 0 stream settings...
=== Connection 7FA00451E0 cleaned up ===
=== Connection 7FA00451E0 established === <- starting our program that aborts quickly after starting a repeating capture
PCLHW_DTParser
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16LoadOverridesFile: looking for override file [/data/nvcam/settings/imx274_bottom_A6V26.isp] 6/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx274_bottom_A6V26.isp] 7/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx274_bottom_A6V26.isp] 8/16---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
CameraProvider result: provider=0x7f993729d0, shim=0x7f99372ac0, status=0, rpc status=1, size=9
LSC: LSC surface is not based on full res!
SCF: Error InvalidState: Unable to connect as stream producer (propagating from src/services/gl/EGLStreamProducer.cpp, function connect(), line 140)
SCF: Error InvalidState: (propagating from src/services/gl/GLService.cpp, function createEGLStreamProducer(), line 565)
SCF: Error InvalidState: (propagating from src/api/Session.cpp, function createEGLStreamProducer(), line 1370)
(Argus) Error InvalidState: (propagating from src/api/OutputStreamImpl.cpp, function connectProducer(), line 141)
(Argus) Error InvalidState: (propagating from src/api/CaptureSessionImpl.cpp, function submitCaptureRequests(), line 283)
(Argus) Error InvalidState: (propagating from src/api/CaptureSessionImpl.cpp, function threadFunction(), line 777)
(Argus) Error EndOfFile: (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function recvThreadCore(), line 138)
(Argus) Error EndOfFile: (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function run(), line 56)
=== Connection 7FA00451E0 closed ===
/== CLEANUP 0x7f99372ac0 ==\
Destroying session 0x7f99373f20
<--- all future attempts at starting our program times out while calling CameraProvider::create
<-- restart daemon
=== Connection 7F9E2C91E0 established === <-- starting our program normally
PCLHW_DTParser
LoadOverridesFile: looking for override file [/Calib/camera_override.isp] 1/16LoadOverridesFile: looking for override file [/data/nvcam/settings/camera_overrides.isp] 2/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/camera_overrides.isp] 3/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/camera_overrides.isp] 4/16LoadOverridesFile: looking for override file [/data/nvcam/camera_overrides.isp] 5/16LoadOverridesFile: looking for override file [/data/nvcam/settings/imx274_bottom_A6V26.isp] 6/16LoadOverridesFile: looking for override file [/opt/nvidia/nvcam/settings/imx274_bottom_A6V26.isp] 7/16LoadOverridesFile: looking for override file [/var/nvidia/nvcam/settings/imx274_bottom_A6V26.isp] 8/16---- imager: No override file found. ----
LSC: LSC surface is not based on full res!
CameraProvider result: provider=0x7f99372950, shim=0x7f99372a00, status=0, rpc status=1, size=9
LSC: LSC surface is not based on full res!
/== CLEANUP 0x7f99372a00 ==\ <-- killing our program with SIGABRT after a few seconds

  Destroying real provider 0x7f99372950
\== CLEANUP DONE ==/
(Argus) Error EndOfFile:  (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function recvThreadCore(), line 138)
(Argus) Error EndOfFile:  (propagating from libs/rpc_socket_server/ServerSocketManager.cpp, function run(), line 56)
=== Connection 7F9E2C91E0 closed ===
Cleaning up 0 requests...
Cleaning up 0 queues...
Cleaning up 0 streams...
Cleaning up 0 stream settings...
=== Connection 7F9E2C91E0 cleaned up ===

I’m going to do a little more testing to ensure avoiding an abort/crash until I have at least one frame will solve this problem

@cooperm111o
Did you break argus_daemon during the argus_camera be starting? I am curious why are you doing this for.

== CLEANUP 0x7f98bbaa40 ==\ <- ctrl-c’ing the node

Sorry if I was unclear. What I was doing there was cleanly shutting down my client application.

This issue appears if our client application crashes while the EGL stream is being initialized. I was able to mitigate the problem by adding a call to “IStream::waitUntilConnected()” immediately after submitting the repeating capture request. However I’m still worried that if something unexpected happens, this will cause the freeze I had been triggering in the “argus_daemon”

Can you provide detail information step by step to repo it.
The client should be able to launch if argus_daemon restarted.

Hi cooperm111o,

Any update? Or issue has been clarified and fixed?

Thanks

Sorry, I’ve been intending to write up a basic executable that better demonstrates the problem. I’ll try to have that posted sometime in the next few days