Deepstream corrupted frames

I am using deepstream codded with python running on multiple rtps streams , in the process once and a while i am taking the received image from the buffer using.

n_frame = pyds.get_nvds_buf_surface(hash(gst_buffer), frame_meta.batch_id)
frame_copy = np.array(n_frame, copy=True, order='C')
frame_copy = cv2.cvtColor(frame_copy, cv2.COLOR_RGBA2BGRA)
image_bytes = cv2.imencode('.bmp', frame_copy)[1].tobytes()
Afterward saved to file (in this thread)

99% of the time everything works like a charm, but sometimes I receive black images or images with an RGB issue or rotated.

The RTSP stream is valid and I am using the code based on the python examples
Is anybody familiar with that case?