ATOMP_FRAME_TRUNCATED error when encoding and doing CUDA processing with three cameras

Autonomous Machines Jetson & Embedded Systems Jetson TX2
, , ,
1

Continuing topic 145687 as I still have the issue (was busy with other things and got told to make a new topic if this still was an issue by DaneLLL).

I’m now running the same setup but with a higher input resolution (3840x2160 instead of 2880x2160), and the crashes seem to be more frequent.

The issue is still the same (ATOMP_FRAME_TRUNCATED error when encoding and doing CUDA processing), but after investigating some more it does seem like it doesn’t happen when only using 2 cameras, and seems somewhat related to cpu load?

First of all, the issue only seems to occur when encoding. I’ve tried running with output to nvoverlaysink instead of encoding with omxh264enc or nvv4l2h264enc. When outputting to a screen the CPU usage as stated by htop is at around 85%, spread fairly evenly across cores (no core goes above 50%). This seems to be stable (ran over several days without the issue happening).
When outputting to /dev/null through a nvv4l2h264enc, the total cpu usage for gstreamer processes is at around 93%, still spread fairly evenly across cores (no core going above 50%). However, after some amount of time (ranging from a few seconds to a tens of minutes) the pipeline crashes with ERROR: from element /GstPipeline:pipeline0/GstV4l2Src:v4l2src2: Could not read from resource. (have had it happen with all three of the sensors).

I have ran the pipeline encoding and outputting to /dev/null with input from two sensors (but otherwise exactly the same) and left it running overnight, and after 18 hours it was still running, so it seems like this might be related to the amount of data coming in from the three sensors?

Could there still be some DVFS happening causing pauses and making the ATOMP buffer overflow?

3

Hi,
DVFS is enabled for hardware encoder. Please set the property:

  maxperf-enable      : Enable or Disable Max Performance mode
                        flags: readable, writable, changeable only in NULL or READY state
                        Boolean. Default: false

to keep hardware encoder at max freq for a try. You can run sudo tegrastats to confirm hardware encoder(NVENC) is at max freq.

4

Hi DaneLLL!

Thank you for the suggestion!
I have tried setting this but sadly this still doesn’t work. My pipeline is still crashing with the same error, and the kernel debug logs still show the ATOMP_FRAME_TRUNCATED error when things crash. tegrastats did confirm NVENC was locked at maximum frequency.

5

hello kamnxt,

is this related to multiple camera use-case, could you still see the same failure with single camera?

6

Hi JerryChang!
It does seem to be related to the multiple camera use-case. I’ve actually had it run fine with two cameras for a few days without crashing, while it crashes after just minutes with three cameras.

7

hello kamnxt,

please refer to software features, you should check CSI and USB Camera Features.
it’s only claim multiple camera with resolution at 1920×1440@30-fps, the resolution and fps matters.
since your use-case already beyond the software supports, suggest you should have dual camera solution or reduce the resolution to have verificaition.
thanks

8

Ah, I see. Interestingly enough, running the same three cameras at the same resolution at once and outputting to an nvoverlaysink works perfectly fine (only thing changed is “omxh264enc ! matroskamux” to “nvoverlaysink”) - with three cameras running at 3840x2160. Runs perfectly fine for days without crashing.