Decoding stops after running "00_video_decode_test" for a long time

UNA_H · August 25, 2023, 9:58am

Hi,

Our customer is facing decoding stoppage after running “00_video_decode_test” for a long time.
This problem is reproduced with the “XavierNX evaluation board” + “TX2NX SOM” + “00_video_decode_test” sample software.

Software Environment:
L4T Driver Package (BSP)
https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/t186/jetson_linux_r32.6.1_aarch64.tbz2

Sample Root Filesystem
https://developer.nvidia.com/embedded/l4t/r32_release_v6.1/t186/tegra_linux_sample-root-filesystem_r32.6.1_aarch64.tbz2

The reproduction procedure is as follows：

Copy and install the deb files to jetson devkit
Get the dev package by the following method
$sudo apt install -d libv4l-dev
Download from t186 and common of Jetpack 4.6 at Index
See the compressed file (TX2NX_decode_stop_problem.zip) for the deb package and a sample application modified to reproduce this issue.

TX2NX_decode_stop_problem.zip (73.5 MB)

Build the modified sample code
$cd ~/jetson_multimedia_api/samples/00_video_decode_test/
$make CUDA_PATH=/usr/local/cuda-10.2
Running the Sample Application
$./video_decode H264 --max-perf --disable-rendering -loop 0 --blocking-mode 0 -o dummy /tmp/xxxx.h264
“xxx.h264” is any H.264 stream file"
Waiting for problem to occur.
Frame count stopped after displaying 88888800 approximately 5.5 hours after runnnig.

What is causing this problem? Please let me know how to solve this problem.

Best Regards,
UNA

DaneLLL · August 28, 2023, 1:31am

Hi,
The latest release for TX2 is Jetpack 4.6.4(r32.7.4). We would suggest upgrade to latest release and try again.

UNA_H · August 28, 2023, 2:08am

Hi DaneLLL,

Thank you for your reply.

We have confirmed that this issue is still occurring in Jetpack 4.6.4 (r32.7.4). The reproduced environment in R32.7.4 is shared in the following compressed file.

JetPack_4.6.4_test_env.zip (531.2 KB)

Note that this problem occurs even with the release default sample software after more than 35 hours of continuous execution.

Best Regards,
UNA

UNA_H · August 28, 2023, 2:49am

Note that this problem occurs even with the release default sample software after more than 35 hours of continuous execution.

Correction regarding the above. The ploblem occurs in about 35 days, not 35 hours. This seems to occur when the number of frames increases.

Best Regards,
UNA

DaneLLL · August 28, 2023, 9:13am

Hi,
As a quick solution, is it possible to terminate the process and re-create it before hitting the issue? It may not be easy to check further if it occurs in about 35 days. We would suggest apply a quick solution for the use-case.

UNA_H · August 28, 2023, 9:37am

Hi,

It is not possible to propose a solution by re-creating the process. We need to know the root cause of this problem.
This problem appears to occur when the frame count reaches around 88888800, is this a limitation due to Jetson’s H/W(especially the internal decoder)?

Best Regards,
UNA

DaneLLL · August 29, 2023, 2:55am

Hi,
We will check this further and update. Please realize it may take some time.

And it would be great if you can share a method that can show the issue in a shorter time.

mjnhb20011 · September 3, 2023, 11:42pm

Hi, I reproduced it within 3 days in our program.

I received only I frame every second and queued 5 times to decoder.

last log in buffer count was around 88890000.

mjnhb20011 · September 4, 2023, 11:39pm

video_decode_main.cpp (78.7 KB)
video_decode.h (3.7 KB)

run command:
./video_decode MPEG4 --disable-rendering --max-perf --stats -v4l2-memory-out-plane 1 -v4l2-memory-cap-plane 1 ./decoder_test.mp4

Here is how I reproduce this issue. I tested it with 2,552 minutes of mp4 video only incoded with I frame.

it takes almost 9~10 hours to reproduce issue.

it stops working around capture_plane.getTotalDequeuedBuffers() value is 88888000~88889000

this is the end of the output of program

>>>>> number of buffers allocated/requested on the capture plane: 7
>>>>> number of planes buffers on capture plane for the currently set format: 2
>>>>> number of buffers currently queued on the capture plane: 7
>>>>> number of total number of buffers dequeued from the capture plane: 88884001
>>>>> sizeof v4l2_buf : 88, bytesused: 123, length: 2
stream read: 4000000, chunk_count: 622054
stream read: 4000000, chunk_count: 622055
stream read: 4000000, chunk_count: 622056
stream read: 4000000, chunk_count: 622057
stream read: 4000000, chunk_count: 622058
stream read: 4000000, chunk_count: 622059
stream read: 4000000, chunk_count: 622060
>>>>> number of buffers allocated/requested on the capture plane: 7
>>>>> number of planes buffers on capture plane for the currently set format: 2
>>>>> number of buffers currently queued on the capture plane: 7
>>>>> number of total number of buffers dequeued from the capture plane: 88885001
>>>>> sizeof v4l2_buf : 88, bytesused: 123, length: 2
stream read: 4000000, chunk_count: 622061
stream read: 4000000, chunk_count: 622062
stream read: 4000000, chunk_count: 622063
stream read: 4000000, chunk_count: 622064
stream read: 4000000, chunk_count: 622065
stream read: 4000000, chunk_count: 622066
stream read: 4000000, chunk_count: 622067
>>>>> number of buffers allocated/requested on the capture plane: 7
>>>>> number of planes buffers on capture plane for the currently set format: 2
>>>>> number of buffers currently queued on the capture plane: 7
>>>>> number of total number of buffers dequeued from the capture plane: 88886001
>>>>> sizeof v4l2_buf : 88, bytesused: 123, length: 2
stream read: 4000000, chunk_count: 622068
stream read: 4000000, chunk_count: 622069
stream read: 4000000, chunk_count: 622070
stream read: 4000000, chunk_count: 622071
stream read: 4000000, chunk_count: 622072
stream read: 4000000, chunk_count: 622073
stream read: 4000000, chunk_count: 622074
>>>>> number of buffers allocated/requested on the capture plane: 7
>>>>> number of planes buffers on capture plane for the currently set format: 2
>>>>> number of buffers currently queued on the capture plane: 7
>>>>> number of total number of buffers dequeued from the capture plane: 88887001
>>>>> sizeof v4l2_buf : 88, bytesused: 123, length: 2
stream read: 4000000, chunk_count: 622075
stream read: 4000000, chunk_count: 622076
stream read: 4000000, chunk_count: 622077
stream read: 4000000, chunk_count: 622078
stream read: 4000000, chunk_count: 622079
stream read: 4000000, chunk_count: 622080
stream read: 4000000, chunk_count: 622081
>>>>> number of buffers allocated/requested on the capture plane: 7
>>>>> number of planes buffers on capture plane for the currently set format: 2
>>>>> number of buffers currently queued on the capture plane: 7
>>>>> number of total number of buffers dequeued from the capture plane: 88888001
>>>>> sizeof v4l2_buf : 88, bytesused: 123, length: 2
stream read: 4000000, chunk_count: 622082
stream read: 4000000, chunk_count: 622083
stream read: 4000000, chunk_count: 622084
stream read: 4000000, chunk_count: 622085
stream read: 4000000, chunk_count: 622086
stream read: 4000000, chunk_count: 622087

DaneLLL · September 5, 2023, 6:16am

Hi,
Thanks for the information. We are checking it. Will update when there is new status.

mjnhb20011 · September 12, 2023, 11:51pm

Hi, any updates on this issue?

when can you share the patch?

when processing buffer count is 88888888, the problem occured when proccesing capture_plane.dqbuffer

DaneLLL · September 13, 2023, 3:29am

Hi,
We are checking the issue. Will update when there is further finding.

DaneLLL · September 27, 2023, 1:27am

Hi,
Please note we shall have this fixed in next Jetpack 4.6.5, 5.1.3 and 6.0.

mjnhb20011 · September 27, 2023, 1:31am

Hi, when will be new version release?

Can I fix this issue without upgrade jetpack? (like copy and paste new version of libv4l2~.so)

DaneLLL · September 27, 2023, 2:32am

Hi,
Please refer to the announcement:
JetPack 4 Reaches End of Life

For a quick fix, please contact vendor of the custom board for help. Thanks.