Restart Argus Session

Robotics & Edge Computing Jetson & Embedded Systems Jetson TX2
1

We have a Jetson TX2 system running 28.2.1 that occasionally sees a camera hardware error. That error propagates up into my code, but I’m unable to resurrect my Argus pipeline until I restart my program. I’ve tried releasing all the Argus objects and recreating them, but I get an error when I’m trying to create a CaptureSession:

(Argus) Error InvalidState: CaptureProvider destroyed (in src/api/CameraProviderImpl.cpp, function createCaptureSession(), line 189)

I can’t find any mention of a CaptureProvider in the multimedia API. Is there are way to get back to a clean Argus state?

2

Hi r.jameson,
for the hardware error you mentioned, is it sync point timeout? Does it happen with default camera ov5693?

3

Hi DaneLLL, it is a sync point timeout. We’re using a custom carrier with 6 Sony sensors from Leopard, and I can only reliably reproduce the problem when I start all six cameras and have not run the jetson_clocks script. Nevertheless, it seems like it might have happened on some of our hardware in the field, so I’d like to be able to recover from it, even if the likelihood of it recurring is low.

4

hello r.jameson,

could you please have below experiments to narrow down the issues.

  1. could you please try to access camera sensors via v4l2-ctl to verify the sensor driver status
v4l2-ctl -d /dev/videoX --set-fmt-video=width=2592,height=1944,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
  1. could you please try to access each camera sensors with gst-launch to verify cameras individually.
gst-launch-1.0 nvarguscamerasrc sensor-id=X ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvoverlaysink -ev
5

Hi JerryChang, thanks for your help. I’ve run the tests you asked, and the results are at at the bottom, however, I’m not primarily concerned with the hardware errors. I’m more interested in why I can’t release the cameras and restart Argus. I can release the objects (Argus::UniqueObject.reset()), but when I try to recreate the capture session, I get:

(Argus) Error InvalidState: CaptureProvider destroyed (in src/api/CameraProviderImpl.cpp, function createCaptureSession(), line 189)

While I’m releasing the objects, I sometimes get:

(Argus) Error InvalidState: Unknown stream deleted. (in src/api/CaptureSessionImpl.cpp, function outputStreamDeleted(), line 650)

I don’t know if that’s related. It’s entirely possible that the stream hadn’t yet connected to the producer.

Any thoughts on why I can’t restart Argus? Is the Argus source available somewhere?

Here are the results of the tests you mentioned. To me, it looks OK, except that I would expect the frame rate to be 60, which is the frame rate of the sensor at 1080p. (Also, there doesn’t appear to be a sensor-id parameter to nvarguscamerasrc).

$ v4l2-ctl -d /dev/video0 --set-fmt-video=width=1920,height=1080,pixelformat=RG10 --set-ctrl bypass_mode=0 --stream-mmap --stream-count=100
<<<<<<<<<<<<<<<<<<<<<<< 21.78 fps
<<<<<<<<<<<<<<<<<<<<<< 21.78 fps
<<<<<<<<<<<<<<<<<<<<<< 21.71 fps
<<<<<<<<<<<<<<<<<<<<< 21.69 fps
<<<<<<<<<<<<

$ gst-launch-1.0 nvarguscamerasrc ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! filesink location=test.raw
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
GST_ARGUS: Running with following settings:
   Camera index = 0 
   Camera mode  = 0 
   Output Stream W = 1920 H = 1080 
   seconds to Run    = 0 
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected...
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
^Chandling interrupt.
Interrupt: Stopping pipeline ...
Execution ended after 0:00:04.575952116
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL ...
Freeing pipeline ...
WARNING Argus: 10 client objects still exist during shutdown:
	548224145024 (0x7f78000c48)
	548224145184 (0x7f78015658)
	548224145344 (0x7f780156e8)
	548224145504 (0x7f780157b8)
	548224147520 (0x7f78000cd8)
	548224199296 (0x7f78000da8)
	548224201888 (0x7f78000e88)
	548224390160 (0x7f78000f38)
	548224421648 (0x7f78001088)
	548224422544 (0x7f78000b00)
6

hello r.jameson,

  1. are you trying to stream all 6-cameras with 1920x1080@60-fps ?
  2. the reporting frame rate from v4l2 standard control did not reach 60-fps, please review your sensor drivers since there might be issues with your low-level drivers.
  3. we don’t have Argus sources available, could you please refer to Topic 1037809 to add error handling mechanism.
7

@r.jameson
What’s the sensor you get from Leopard. Have you consult with them for the multiple camera issue instead of find a way to reset argus?

8

Hi JerryChang and ShaneCCC. Thanks for your help! I applied the patches you mentioned. When I have the clocks turned up (jetson_clocks.sh), the failures are rare, like once in 8 hours. Without the clocks turned up, failures are more frequent. I’m not sure I’ll ever be confident that all failures are eliminated, so I’d really like to be able to restart Argus which it detects a problem. Is that possible? If not, I have to abort my program and let systemd restart it, which loses a bunch of state that I’d like to keep around.

9

hello r.jameson,

since the reporting fps from v4l2-ctls did not reach your sensor capability.
suggest you to consult with Leopard team for multiple camera solutions.
thanks