Repeated frames using gstreamer with multiple cameras in OpenCV

Autonomous Machines Jetson & Embedded Systems Jetson Nano
,
1

I’m using OpenCV 4.4.1 and a Jetson Nano (JP 4.4) with 3 raspberry pi cameras.

My goal is to process the latest frame grabbed from the ‘next’ camera source in a round robin fashion every 3 seconds. For now, ‘processing’ just means saving the frame to disk. My program continuously grabs frames from all 3 cameras in separate threads to ensure the latest frame is retrieved and the buffer doesn’t grow large while the main thread handles processing.

The problem I’m seeing is that typically at least 1/3 cameras will be frozen on a frame while the other cameras remain real time. So the frame read from the VideoCapture will be an old frame. This doesn’t happen immediately after starting the program. Sometimes this will happen 10 min later, other times a few hours later. I have seen all 3 cameras “frozen” on a past frame as well.

My gstreamer fstring:

nvarguscamerasrc sensor_id={id} !
      video/x-raw(memory:NVMM), width={self.frame_width}, height={self.frame_height},
      format=(string)NV12, framerate=30/1 ! nvvidconv flip-method={flip_method} !
      video/x-raw, width={self.frame_width}, height={self.frame_height},
      format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR
      ! appsink

I have a class CameraStream which manages a single camera, so 3 CameraStream’s are initialized with the gstreamer string above. To ensure that the cameras aren’t being throttled, I call cap.read in a thread for each camera.

  # CameraStream methods..
  def start(self):
    t = Thread(target=self.get, args=())
    t.setDaemon(True)
    t.start()
    
  def get(self):
    while not self.stopped:
      (self.grabbed, self.frame) = self.cap.read()
      cv2.waitKey(1)

Then in the main thread I process a frame on an interval:

while True:
  image = self.cameras[frame_idx].frame
  now_s = time.time()
  if (now_s - start_s) < self.args['interval'] or image is None:
    continue
  ''' Code to process frame here - for now, we save to disk .. '''
  start_s = time.time()
  frame_idx = (frame_idx + 1) % len(self.cameras)

Edit - For my sanity I checked that saving to disk wasn’t the issue by checking the latest frame against a previously saved frame in the main thread when processing:

      # Temp check to see if frame is repeating.
      if self.old_frames[frame_idx] is not None:
        if np.array_equal(self.old_frames[frame_idx], image):
          print(f"Same frame idx {frame_idx}")

Edit2 - I’ve done some more testing. I believe cap.read is hanging or blocking indefinitely and not retrieving anything now, because I cannot get the thread to stop by changing the stopped variable above after a repeated frame is detected. This would explain the ‘repeated’ frames because the thread would never update the frame.

3

Hi,
Please apply the steps and check if it helps:

  1. Run sudo nvpmodel -m 0 and sudo jetson_clocks
  2. Set sync=0 to appsink

Since there i sadditional memory copy when running OpenCV on Jetson platforms( please check this post ), the performance is dominated by CPU capability. For multi-camera use-case, we would suggest run gstreamer or jetson_multimedia_api.

1 Like
5

I applied the steps and also turned the interval I’m saving to disk down to 3 seconds instead of 2. To further reduce cpu usage I also changed my code so that the loop in the main thread will call time.sleep for 3 seconds instead of checking for elapsed time (if anyone also has a similar issue, this is big. time.sleep is much more efficient than polling for time).

The device has been running for 24 hr now without issue. I’m going to test more at 2 seconds interval, but I believe at this interval it was also having issues.

So I think this may confirm that it is a CPU issue.

Thank you!