Jetson nano nvv4l2h264enc stop working sometimes

Autonomous Machines Jetson & Embedded Systems Jetson Nano
1

Hi
I use nvv4l3h264enc to stream the video from camera to PC. I have noticed that the video on PC frozen sometimes. I run tegrastats on jetson nano to monitor the Jetson nano resources use state and found. When the video on PC frozen, the NVENC disappeared from the tegrastats terminal. Can I say in this case, the nvv4l2h264enc stopped?
Please refer to the tegrastats output data from video stream to stopped below for your info

RAM 2298/3956MB (lfb 2x4MB) SWAP 11/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [74%@1479,60%@1479,55%@1479,59%@1479] EMC_FREQ 15%@1600 GR3D_FREQ 30%@153 NVENC 716 APE 25 PLL@51.5C CPU@51C iwlwifi@60C PMIC@50C GPU@50C AO@59.5C thermal@51C POM_5V_IN 5248/4449 POM_5V_GPU 196/78 POM_5V_CPU 1645/1215
RAM 2301/3956MB (lfb 2x4MB) SWAP 11/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [71%@1479,55%@1479,58%@1479,52%@1479] EMC_FREQ 17%@1600 GR3D_FREQ 35%@153 NVENC 716 APE 25 PLL@51.5C CPU@52C iwlwifi@62C PMIC@50C GPU@50C AO@59.5C thermal@51.25C POM_5V_IN 5699/4455 POM_5V_GPU 195/79 POM_5V_CPU 1952/1218
RAM 2303/3956MB (lfb 2x4MB) SWAP 11/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [57%@1224,51%@1224,48%@1224,52%@1224] EMC_FREQ 17%@1600 GR3D_FREQ 0%@153 NVENC 716 APE 25 PLL@50.5C CPU@50.5C iwlwifi@62C PMIC@50C GPU@49.5C AO@59.5C thermal@51C POM_5V_IN 3763/4452 POM_5V_GPU 118/79 POM_5V_CPU 713/1216
RAM 2303/3956MB (lfb 2x4MB) SWAP 11/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@1224,0%@1224,0%@1224,2%@1224] EMC_FREQ 13%@1600 GR3D_FREQ 0%@76 APE 25 PLL@50C CPU@49.5C iwlwifi@63C PMIC@50C GPU@51.5C AO@58.5C thermal@50C POM_5V_IN 2480/4443 POM_5V_GPU 0/79 POM_5V_CPU 319/1212
RAM 2303/3956MB (lfb 2x4MB) SWAP 11/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [23%@1224,18%@1224,8%@1224,10%@1224] EMC_FREQ 10%@1600 GR3D_FREQ 21%@76 APE 25 PLL@50C CPU@50C iwlwifi@63C PMIC@50C GPU@49.5C AO@58.5C thermal@50.5C POM_5V_IN 2910/4437 POM_5V_GPU 119/79 POM_5V_CPU 598/1209

Would you please let me know what’s wrong with it and how can I fix it?
Thanks

CX

2

Further info on this issue for your reference
below is the part of message from gst_v4l2_enc_latency_18436.log
When the video on laptop frozen ( the video streaming stopped), I opened the gst_v4l2_enc_latency_18436.log file and found it stopped at frame 6160. Waiting for a while, the video streaming started again, the log file updated. I can see it has a big jump on pts number at frame 6163.

  1. What is the meaning of pts number?
  2. why it has a big jump after video streaming stopped and started by itself?
KPI: v4l2: frameNumber= 6158 encoder= 11 ms pts= 312779327332

KPI: v4l2: frameNumber= 6159 encoder= 11 ms pts= 312820980018

KPI: v4l2: frameNumber= 6160 encoder= 16 ms pts= 312862647331

KPI: v4l2: frameNumber= 6161 encoder= 15 ms pts= 312904322227

KPI: v4l2: frameNumber= 6162 encoder= 12 ms pts= 312945986435

KPI: v4l2: frameNumber= 6163 encoder= 11 ms pts= 410829067052

KPI: v4l2: frameNumber= 6164 encoder= 11 ms pts= 410847611420

KPI: v4l2: frameNumber= 6165 encoder= 11 ms pts= 410866870234

KPI: v4l2: frameNumber= 6166 encoder= 10 ms pts= 410903694486

gst_v4l2_enc_latency_18436.log (1.2 MB)

Please refer to attached file for details
Thanks

CX

4

Hi,
From the tegrastats it looks the system is overheating and goes to throttling. Do you see it on developer kit or your custom board?

5

It is on developer kit. Customer board is not available at this moment.

6

Hi
all the temperature data are below 60C except iwlwifi, is it overheating?
Thanks

CX

7

Any help on this issue please.
Thanks

CX

8

Hi,
At idle state tegrastats is like:

RAM 1699/3956MB (lfb 91x4MB) SWAP 32/1978MB (cached 4MB) IRAM 0/252kB(lfb 252kB)
 CPU [11%@204,2%@204,1%@204,3%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@
21C CPU@23.5C PMIC@50C GPU@23C AO@29C thermal@23.25C POM_5V_IN 1940/1940 POM_5V_
GPU 0/0 POM_5V_CPU 126/126
RAM 1699/3956MB (lfb 91x4MB) SWAP 32/1978MB (cached 4MB) IRAM 0/252kB(lfb 252kB)
 CPU [9%@102,1%@102,2%@102,2%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@2
0.5C CPU@23.5C PMIC@50C GPU@22.5C AO@28.5C thermal@23.25C POM_5V_IN 1940/1940 PO
M_5V_GPU 0/0 POM_5V_CPU 126/126
RAM 1699/3956MB (lfb 91x4MB) SWAP 32/1978MB (cached 4MB) IRAM 0/252kB(lfb 252kB)
 CPU [9%@102,0%@102,2%@102,2%@102] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@2
1C CPU@23.5C PMIC@50C GPU@23C AO@29C thermal@23.25C POM_5V_IN 1940/1940 POM_5V_G
PU 0/0 POM_5V_CPU 126/126
RAM 1699/3956MB (lfb 91x4MB) SWAP 32/1978MB (cached 4MB) IRAM 0/252kB(lfb 252kB)
 CPU [10%@102,2%@102,0%@204,3%@204] EMC_FREQ 1%@1600 GR3D_FREQ 0%@76 APE 25 PLL@
21C CPU@23.5C PMIC@50C GPU@23C AO@29C thermal@23.25C POM_5V_IN 1940/1940 POM_5V_
GPU 0/0 POM_5V_CPU 126/126

The temperature looks high in your tegrastats. Please check if you add a fan towards the developer kit. See if temperature can be lower and have improvement in performance.

9

Hi
Below is the tegrastats output of the Jetson nano developer kit just after boots up without running any user application program. The temperature on device is similar to yours. Video streaming at after boots up (lower temperature) works fine. I did not notice that from which temperature, the nvv4l2h264enc stopped or Jetson nano system goes to throttling. If it is overheating, can I say “Jetson nano may enter throttling if the temperature is above 50C”? I may ignore the throttling on developer kit. Would you please let me know what is the Jetson nano product module working temperature without throttling?
Thanks

CX

RAM 1361/3956MB (lfb 401x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [11%@102,2%@102,3%@204,4%@204] EMC_FREQ 3%@1600 GR3D_FREQ 0%@76 APE 25 PLL@19.5C CPU@20C iwlwifi@30C PMIC@50C GPU@19.5C AO@27C thermal@19.75C POM_5V_IN 2022/2020 POM_5V_GPU 0/34 POM_5V_CPU 80/80
RAM 1361/3956MB (lfb 401x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [13%@204,6%@204,5%@204,4%@204] EMC_FREQ 3%@1600 GR3D_FREQ 25%@76 APE 25 PLL@19.5C CPU@20.5C iwlwifi@30C PMIC@50C GPU@20C AO@27C thermal@19.75C POM_5V_IN 2140/2035 POM_5V_GPU 40/35 POM_5V_CPU 161/90
RAM 1361/3956MB (lfb 401x4MB) SWAP 0/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [22%@307,19%@307,16%@307,14%@307] EMC_FREQ 4%@1600 GR3D_FREQ 30%@76 APE 25 PLL@20C CPU@20.5C iwlwifi@30C PMIC@50C GPU@19.5C AO@27C thermal@20C POM_5V_IN 2338/2069 POM_5V_GPU 80/40 POM_5V_CPU 282/111
10

Hi
I did an experiment. Run my application program (image processing using OpenCV, saving the video as file to MicroSD card and streaming the video to PC via WIFI) video after boots up. Below is the tegrastats output at first point of encoder stopped work. It looks like an overheating resulted throttling. The temperature is just above 60C. Is it the highest temperature Jetson nano can work normally (without throttling)?

RAM 2446/3956MB (lfb 1x4MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [82%@1479,100%@1479,94%@1479,81%@1479] EMC_FREQ 24%@1600 GR3D_FREQ 10%@307 NVENC 716 APE 25 PLL@63C CPU@64.5C iwlwifi@66C PMIC@50C GPU@62C AO@72C thermal@63.25C POM_5V_IN 6367/5565 POM_5V_GPU 347/184 POM_5V_CPU 2696/2154
RAM 2446/3956MB (lfb 1x4MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [85%@1479,90%@1479,84%@1479,98%@1479] EMC_FREQ 24%@1600 GR3D_FREQ 50%@307 NVENC 716 APE 25 PLL@63C CPU@64.5C iwlwifi@67C PMIC@50C GPU@61.5C AO@72C thermal@63C POM_5V_IN 6203/5565 POM_5V_GPU 308/184 POM_5V_CPU 2619/2154
RAM 2448/3956MB (lfb 1x4MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [85%@1479,90%@1479,83%@1479,100%@1479] EMC_FREQ 24%@1600 GR3D_FREQ 60%@307 NVENC 716 APE 25 PLL@63C CPU@64C iwlwifi@68C PMIC@50C GPU@61.5C AO@72C thermal@63.25C POM_5V_IN 6461/5565 POM_5V_GPU 346/184 POM_5V_CPU 2692/2154
RAM 2452/3956MB (lfb 4x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [58%@1479,100%@1479,41%@1479,98%@1479] EMC_FREQ 21%@1600 GR3D_FREQ 0%@307 APE 25 PLL@62C CPU@64C iwlwifi@68C PMIC@50C GPU@61C AO@71.5C thermal@63.25C POM_5V_IN 4237/5565 POM_5V_GPU 78/184 POM_5V_CPU 1801/2154
RAM 2452/3956MB (lfb 4x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [13%@1479,100%@1479,3%@1479,100%@1479] EMC_FREQ 14%@1600 GR3D_FREQ 0%@76 APE 25 PLL@62C CPU@63.5C iwlwifi@68C PMIC@50C GPU@63.5C AO@71C thermal@62.5C POM_5V_IN 3514/5564 POM_5V_GPU 0/184 POM_5V_CPU 1774/2154
RAM 2453/3956MB (lfb 4x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [16%@1479,100%@1479,10%@1479,100%@1479] EMC_FREQ 9%@1600 GR3D_FREQ 0%@76 APE 25 PLL@61.5C CPU@63.5C iwlwifi@68C PMIC@50C GPU@63.5C AO@71C thermal@62C POM_5V_IN 3508/5564 POM_5V_GPU 0/184 POM_5V_CPU 1774/2154
11

Another case captured (see below), It looks the encoder stopped work is NOT the overheating as the temperature is below 50C.

RAM 2521/3956MB (lfb 2x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [84%@1479,61%@1479,64%@1479,63%@1479] EMC_FREQ 28%@1600 GR3D_FREQ 24%@384 NVENC 716 APE 25 PLL@35.5C CPU@36.5C iwlwifi@48C PMIC@50C GPU@34.5C AO@42.5C thermal@35.5C POM_5V_IN 5513/3510 POM_5V_GPU 156/105 POM_5V_CPU 1912/1038
RAM 2522/3956MB (lfb 2x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [81%@1479,62%@1479,58%@1479,60%@1479] EMC_FREQ 28%@1600 GR3D_FREQ 21%@384 NVENC 716 APE 25 PLL@35C CPU@36C iwlwifi@49C PMIC@50C GPU@34.5C AO@42.5C thermal@35.5C POM_5V_IN 5474/3510 POM_5V_GPU 195/105 POM_5V_CPU 1834/1038
RAM 2522/3956MB (lfb 2x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [34%@1224,14%@1224,14%@1224,14%@1224] EMC_FREQ 22%@1600 GR3D_FREQ 0%@76 APE 25 PLL@34C CPU@35C iwlwifi@49C PMIC@50C GPU@36C AO@41.5C thermal@34.75C POM_5V_IN 2323/3510 POM_5V_GPU 0/105 POM_5V_CPU 280/1038
RAM 2522/3956MB (lfb 2x2MB) SWAP 38/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [18%@1224,6%@1224,10%@1224,6%@1224] EMC_FREQ 16%@1600 GR3D_FREQ 0%@76 APE 25 PLL@33.5C CPU@34.5C iwlwifi@49C PMIC@50C GPU@35.5C AO@42C thermal@34.5C POM_5V_IN 2363/3510 POM_5V_GPU 0/105 POM_5V_CPU 320/1038
12

Hi,
Do you see the issue on specific Jetson Nano? Or happens to all developer kits? Also would like to know rate of the issue. How many times does it happen in 10-minute run?

13

It occurred on the developer kit I am working on it about one year now. The issue occurred randomly. When the encoder stopped, I keeps the video streaming program running. It takes number of seconds or number minutes sometimes, the encoder start working again. The encoder stopping / starting status repeats randomly as well. Streaming the video more than one hour may have no any issue, but sometimes has the issue in one minute. The issue occurred more frequently recently.
I did not test it on other Jetson nano board yet.

14

Hi
Would you please let me know:

  1. If it is overheating resulted the throttling, is there any way to disable the throttling temporary or change the throttling threshold value?
  2. Is there any way to get the debug data for you to investigate why the encoder does not working reliably?
    Thanks

CX

15

Look at the tegrastats output below, there is a short time (3 lines recording data) the encoder stopped working, and then it started again.

RAM 2731/3956MB (lfb 4x512kB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [87%@1479,63%@1479,70%@1479,67%@1479] EMC_FREQ 32%@1600 GR3D_FREQ 32%@460 NVENC 716 APE 25 PLL@60.5C CPU@62C iwlwifi@67C PMIC@50C GPU@59.5C AO@69C thermal@61C POM_5V_IN 6246/6219 POM_5V_GPU 546/305 POM_5V_CPU 2221/2189
RAM 2750/3956MB (lfb 4x512kB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [67%@1479,64%@1479,42%@1479,45%@1479] EMC_FREQ 29%@1600 GR3D_FREQ 0%@460 NVENC 716 APE 25 PLL@60C CPU@62.5C iwlwifi@68C PMIC@50C GPU@59C AO@68.5C thermal@60.75C POM_5V_IN 3775/6214 POM_5V_GPU 119/305 POM_5V_CPU 1190/2187
RAM 2753/3956MB (lfb 4x512kB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@1224,77%@1224,0%@1224,0%@1224] EMC_FREQ 20%@1600 GR3D_FREQ 0%@76 APE 25 PLL@58.5C CPU@59.5C iwlwifi@68C PMIC@50C GPU@60.5C AO@68C thermal@60.5C POM_5V_IN 2170/6206 POM_5V_GPU 0/304 POM_5V_CPU 361/2183
RAM 2753/3956MB (lfb 4x512kB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [13%@1224,0%@1224,3%@1224,1%@1224] EMC_FREQ 14%@1600 GR3D_FREQ 0%@76 APE 25 PLL@58.5C CPU@59C iwlwifi@68C PMIC@50C GPU@60.5C AO@67.5C thermal@59.25C POM_5V_IN 2130/6197 POM_5V_GPU 0/304 POM_5V_CPU 321/2180
RAM 2753/3956MB (lfb 4x512kB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [14%@1224,0%@1224,3%@1224,0%@1224] EMC_FREQ 9%@1600 GR3D_FREQ 0%@76 APE 25 PLL@58.5C CPU@59.5C iwlwifi@68C PMIC@50C GPU@60C AO@67.5C thermal@59C POM_5V_IN 2130/6189 POM_5V_GPU 0/303 POM_5V_CPU 321/2176
RAM 2751/3956MB (lfb 4x512kB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [32%@1479,12%@1479,12%@1479,29%@1479] EMC_FREQ 9%@1600 GR3D_FREQ 98%@230 NVENC 716 APE 25 PLL@59.5C CPU@61.5C iwlwifi@67C PMIC@50C GPU@58.5C AO@68C thermal@59.75C POM_5V_IN 5344/6187 POM_5V_GPU 551/304 POM_5V_CPU 1729/2175
RAM 2716/3956MB (lfb 2x2MB) SWAP 52/1978MB (cached 0MB) IRAM 0/252kB(lfb 252kB) CPU [60%@1479,41%@1479,42%@1479,97%@1479] EMC_FREQ 10%@1600 GR3D_FREQ 61%@230 NVENC 716 APE 25 PLL@61C CPU@63C iwlwifi@67C PMIC@50C GPU@59C AO@69C thermal@58.75C POM_5V_IN 6547/6188 POM_5V_GPU 505/304 POM_5V_CPU 2840/2176
16

Hi.
We have not seen NVENC disappear in middle of encoding. Probably we don’t verify the use-case. Please share your gstreamer command so that we can try. Probably your Jetson Nano is borken and easily gets overheated. If it is still under warranty you can consider do RMA.

Jetson FAQ | NVIDIA Developer

17

I tried on other Jetson nano developer kit with carry board B. It has the same issue.
The attached file is my test program
main.cpp (1.5 KB)

18

Hi,
Please also share command for building the sample.

We don’t have USB camera generating 2592x1944p24. Will try standard resolution 1920x1080p30 on Jetpack 4.6.

19

I am using Qt Creator to build the file.
This is Qt Creator Pro file:
test_camera.pro (877 Bytes)

This is MakeFile generated by Qt creator:
Makefile (20.5 KB)

20

Hi,
Is it possible to run the sample without qt? Looks like attached main.cpp should be runnable on default Jetpack 4.6.

21

Yes, you may run the test program without Qt. I just use Qt creator to develop the code, build and debug the code.