GPU: Not enough priv cmd buffer space

surajtn30 · July 13, 2016, 2:22am

We have been using the jetson TX1 to record videos from multiple cameras and also run some other computer vision application. The load on the jetson TX1 is stays in about 1.2-2.0, after running these applications for a few days the load spikes upto a very high number(28-33) and keeps increasing and locks us out of the ssh. The only way resolve this right now is a power cycle. I was able get the syslog, at that particular point.

Jul 12 01:45:00 jetson kernel: [100123.293156] 510-gpu.0, pid 889, refs: 2:
Jul 12 01:45:00 jetson kernel: [100123.293160]  in use idle not busy
Jul 12 01:45:00 jetson kernel: [100123.293168] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Jul 12 01:45:00 jetson kernel: [100123.293168] HEADER: 60400000 COUNT: 00000000
Jul 12 01:45:00 jetson kernel: [100123.293168] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
Jul 12 01:45:00 jetson kernel: [100123.293175]
Jul 12 01:45:00 jetson kernel: [100123.293181] 511-gpu.0, pid 889, refs: 2:
Jul 12 01:45:00 jetson kernel: [100123.293185]  in use idle not busy
Jul 12 01:45:00 jetson kernel: [100123.293194] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Jul 12 01:45:00 jetson kernel: [100123.293194] HEADER: 60400000 COUNT: 00000000
Jul 12 01:45:00 jetson kernel: [100123.293194] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000

Jul 12 01:46:22 jetson kernel: [100204.892885] gk20a gpu.0: __gk20a_channel_syncpt_incr: not enough priv cmd buffer space
Jul 12 01:46:22 jetson kernel: [100204.900913] gk20a gpu.0: gk20a_submit_channel_gpfifo: fail
Jul 12 01:46:22 jetson kernel: [100205.177060] gk20a gpu.0: __gk20a_channel_syncpt_incr: not enough priv cmd buffer space
Jul 12 01:46:22 jetson kernel: [100205.185058] gk20a gpu.0: gk20a_submit_channel_gpfifo: fail
Jul 12 01:46:23 jetson kernel: [100205.797440] gk20a gpu.0: __gk20a_channel_syncpt_incr: not enough priv cmd buffer space
Jul 12 01:46:23 jetson kernel: [100205.805436] gk20a gpu.0: gk20a_submit_channel_gpfifo: fail
Jul 12 01:46:23 jetson kernel: [100205.824281] gk20a gpu.0: __gk20a_channel_syncpt_incr: not enough priv cmd buffer space
Jul 12 01:46:23 jetson kernel: [100205.832301] gk20a gpu.0: gk20a_submit_channel_gpfifo: fail

We use gstreamer version 1.6.0 and the following gstreamer pipeline is run for multiple cameras:

gst-launch-1.0 -e rtspsrc location='rtsp://10.154.13.219:554/mediainput/h264/stream_1' short-header=true ! rtph264depay ! decodebin ! videorate ! "video/x-raw,framerate=30/1"     ! omxvp8enc ! splitmuxsink  location=test%d.webm max-size-time=100000000000 muxer=qtmux

Any help would be much appreciated.

jachen · July 14, 2016, 7:11am

Hello, surajtn30:
Would you please provide more details about the issue you met?

SDK version used? (R23 or R24, …)
64bit or 32bit?
’ after running these applications for a few days’
‘a few days’ means 2, 3, or more?
is that possible for us to reproduce the issue in your side? What kind of ‘computer vision application’ you are running? is that app based on VX, openCV, CUDA, or whatever?
Can you observe the resources (memory, file handler, etc) usage in system during the application running and check whether there’s any memory leakage issue?

br
ChenJian

surajtn30 · July 14, 2016, 9:18pm

ChenJian,

The SDK version is R23
32 bit
We saw this happen twice, few days meaning 2 days.
I do not have a way to reproduce the issue yet, the computer vision application is an openCV application.
I do see a gradual increase in memory usage in all the gstreamer processes. It increases anout 0.2% every day.I was running gstreamer 1.6.0, and then tested with 1.7.91 and in both of them I see the same issue.

surajtn30 · July 15, 2016, 6:26pm

ChenJian,
I was able to get more logs when the above happened again. From looks of it looks there is enough memory left. I have included tegrastats and perfkit results. The syslog shows the same results as last time. Let me know if I need to grab more logs to help us debug this.

Tegrastats

RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,20%,100%,19%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,11%,100%,12%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,12%,100%,17%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,24%,100%,22%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,15%,100%,13%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,20%,100%,13%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,28%,100%,23%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2630/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,26%,94%,18%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,100%,18%,17%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,100%,14%,18%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,18%,18%,96%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,19%,12%,100%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,18%,15%,100%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734
RAM 2631/3854MB (lfb 1x1MB) SWAP 0/0MB (cached 0MB) cpu [100%,18%,16%,100%]@1734 EMC 0%@1600 AVP 0%@115 VDE 0 GR3D 0%@76 EDP limit 1734

Perfkit results:

gpu_idle,1,gpu,ratio,MAXWELL_GPU_SYS0,BOTH,GPU,Activity,UINT64,"Cycles the graphics engine and compute engine is idle."
l2_slice0_read_sectors_fb0,2,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector reads from the L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice0_read_sectors_atomic_fb0,3,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector reads for ATOM/RED to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice0_read_sectors_tex_fb0,4,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector reads from TEX to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice0_write_sectors_fb0,5,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector writes to the L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice0_write_sectors_atomic_fb0,6,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector writes for ATOM/RED to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice0_write_sectors_tex_fb0,7,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector writes from TEX to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice1_read_sectors_fb0,8,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector reads from the L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice1_read_sectors_atomic_fb0,9,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector reads for ATOM/RED to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice1_read_sectors_tex_fb0,10,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector reads from TEX to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice1_write_sectors_fb0,11,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector writes to the L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice1_write_sectors_atomic_fb0,12,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector writes for ATOM/RED to L2 cache in the given slice and FB partition. A sector is 32 bytes."
l2_slice1_write_sectors_tex_fb0,13,gpu,raw,MAXWELL_GPU_FBP0,COMPUTE,Cache,L2,UINT64,"Sector writes from TEX to L2 cache in the given slice and FB partition. A sector is 32 bytes."
sm_active_cycles_vsm0,15,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Activity,UINT64,"Number of cycles that this SM has at least one active warp. Increments by 0-1 per cycle per SM."
sm_active_cycles_vsm1,16,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Activity,UINT64,"Number of cycles that this SM has at least one active warp. Increments by 0-1 per cycle per SM."
sm_active_warps_vsm0,17,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Activity,UINT64,"Active warps per cycle on this SM. Increments by 0-64 per cycle per SM."
sm_active_warps_vsm1,18,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Activity,UINT64,"Active warps per cycle on this SM. Increments by 0-64 per cycle per SM."
sm_branches_diverged_vsm0,19,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Divergent branches by this VSM. This counter increments by one if at least one thread in a warp diverges (that is, follows a different execution path) via a data dependent conditional branch. Increments by 0-1 per cycle."
sm_branches_diverged_vsm1,20,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Divergent branches by this VSM. This counter increments by one if at least one thread in a warp diverges (that is, follows a different execution path) via a data dependent conditional branch. Increments by 0-1 per cycle."
sm_branches_executed_vsm0,21,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Branches taken by this VSM. This counter increments by one if at least one thread in a warp takes the branch. Increments by 0-1 per cycle."
sm_branches_executed_vsm1,22,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Branches taken by this VSM. This counter increments by one if at least one thread in a warp takes the branch. Increments by 0-1 per cycle."
sm_branches_taken_vsm0,23,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Increments by one if at least one thread in a warp takes the branch. Increments by 0-4 per cycle."
sm_branches_taken_vsm1,24,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Increments by one if at least one thread in a warp takes the branch. Increments by 0-4 per cycle."
sm_ctas_launched_vsm0,25,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Activity,UINT64,"Thread blocks launched. Increments by 1 per thread block launched."
sm_ctas_launched_vsm1,26,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Activity,UINT64,"Thread blocks launched. Increments by 1 per thread block launched."
sm_inst_executed_vsm0,27,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Instructions executed in this SM, not including replays. Increments by 0-8 per cycle per SM."
sm_inst_executed_vsm1,28,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Instructions executed in this SM, not including replays. Increments by 0-8 per cycle per SM."
sm_inst_executed_generic_loads_vsm0,29,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Generic load instructions executed by this SM."
sm_inst_executed_generic_loads_vsm1,30,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Generic load instructions executed by this SM."
sm_inst_executed_generic_stores_vsm0,31,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Generic store instructions executed by this SM."
sm_inst_executed_generic_stores_vsm1,32,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Generic store instructions executed by this SM."
sm_inst_executed_global_loads_vsm0,33,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Global load instructions executed by this SM."
sm_inst_executed_global_loads_vsm1,34,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Global load instructions executed by this SM."
sm_inst_executed_global_stores_vsm0,35,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Global store instructions executed by this SM."
sm_inst_executed_global_stores_vsm1,36,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Global,UINT64,"Global store instructions executed by this SM."
sm_inst_executed_local_loads_vsm0,37,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Local,UINT64,"Local load instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_local_loads_vsm1,38,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Local,UINT64,"Local load instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_local_stores_vsm0,39,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Local,UINT64,"Local store instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_local_stores_vsm1,40,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Local,UINT64,"Local store instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_shared_loads_vsm0,41,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Shared,UINT64,"Shared load instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_shared_loads_vsm1,42,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Shared,UINT64,"Shared load instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_shared_stores_vsm0,43,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Shared,UINT64,"Shared store instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_shared_stores_vsm1,44,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Shared,UINT64,"Shared store instructions executed by this SM. Increments by 0-1 per cycle per SM."
sm_inst_executed_surface_loads_vsm0,45,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Surface,UINT64,"Surface load instructions executed by this SM."
sm_inst_executed_surface_loads_vsm1,46,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Surface,UINT64,"Surface load instructions executed by this SM."
sm_inst_executed_surface_stores_vsm0,47,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Surface,UINT64,"Surface store instructions executed by this SM."
sm_inst_executed_surface_stores_vsm1,48,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,Memory,Surface,UINT64,"Surface store instructions executed by this SM."
sm_inst_issued_vsm0,49,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Instructions issued by the scheduler, including replays. Increments by 0-8 per cycle per SM."
sm_inst_issued_vsm1,50,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Instructions,UINT64,"Instructions issued by the scheduler, including replays. Increments by 0-8 per cycle per SM."
sm_pmevent_00_vsm0,51,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger00/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_00_vsm1,52,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger00/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_01_vsm0,53,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger01/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_01_vsm1,54,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger01/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_02_vsm0,55,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger02/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_02_vsm1,56,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger02/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_03_vsm0,57,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger03/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_03_vsm1,58,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger03/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_04_vsm0,59,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger04/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_04_vsm1,60,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger04/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_05_vsm0,61,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger05/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_05_vsm1,62,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger05/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_06_vsm0,63,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger06/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_06_vsm1,64,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger06/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_07_vsm0,65,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger07/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction executed."
sm_pmevent_07_vsm1,66,gpu,raw,MAXWELL_GPU_GPC0TPC0,COMPUTE,SM,Triggers,UINT64,"__prof_trigger07/pmevent instructions executed where at least 1 thread is not predicated off. Increments by 0-1 per warp instruction execut...

jachen · July 25, 2016, 5:34am

Hello, surajtn30:
For so long time testing issue, we’d better narrow down the scope first. According to your description, mainly 2 parts are involved, multimedia and CV algorithm. Can you divide your test program and check them seperately?

run multimedia part without CV algorithm.
feed CV algorithm with static data.
During the test program running, keep an eye for ‘top’ command. Generally the resource consumed should be almost a constant in a long period.

For such kind of issue, it may need more dig and testing.

br
ChenJian

surajtn30 · July 25, 2016, 6:59pm

ChenJian,

I have already run those tests. Running without the CV application still caused the issue. By running top, I can see that the gstreamer processes memory usage keeps increasing. Is there something obvious in the pipeline that is causing that memory to increase?

Thank you,
Suraj

linuxdev · July 25, 2016, 7:52pm

If you are using R24.1 you may be running into this memory leak:
[url]https://devtalk.nvidia.com/default/topic/948388/jetson-tx1/memory-leak-on-tx1-/[/url]

jachen · August 1, 2016, 10:24am

Hello, surajtn30:
Can you provide a simple GST pipeline which can reproduce the memory leak in my side, and I will try to help to find out where the issue comes from.

br
ChenJian

surajtn30 · August 5, 2016, 1:30am

ChenJian,

The following pipeline will give show you the memory usage increase.

gst-launch-1.0 -e rtspsrc location='rtsp://10.154.13.219:554/mediainput/h264/stream_1' short-header=true ! rtph264depay ! h264parse ! splitmuxsink  location=test%d.webm max-size-time=100000000000 muxer=qtmux

Thank you,
Suraj

jachen · August 11, 2016, 12:56am

Hello, Suraj:
The pipeline you provided uses common GST plugins. And I have tested in my platform for more than 1 hour (with a 720p RTSP stream) and did not find obvious memory leak issue.
At beginning, top shows:

5547 ubuntu 20 0 1310212 11228 7016 S 10.9 0.3 0:14.44 gst-launch-1.0

and after an hour running:
5547 ubuntu 20 0 1310212 13132 7088 S 4.6 0.3 4:33.72 gst-launch-1.0

We do find memory leak issues in L4T R24.1 64-bit release (refer to https://devtalk.nvidia.com/default/topic/948388/memory-leak-on-tx1-/) and this will be fixed in next release.

But you mentioned that you are using R23 32-bit version, which should have no such issue.

Would you please provide more details about your test, and paste the detailed log here? I can double check with my test result.

br
ChenJian

surajtn30 · August 18, 2016, 12:36am

ChenJian,
The virtual memory doesn’t increase but the resident memory increases, over a period of two three days, the resident memory does not stop to increase and goes upto 5-6times where it started.

Also, I am running the gstreamer using the python API.

I was able to get more logs this time (with the more elaborate pipeline) and it looks like nvdec shows up first and then I see a kernel panic that is causing this.

Aug 17 21:06:39 min25jetson kernel: [609008.503725] ---- mlocks ----
Aug 17 21:06:39 min25jetson kernel: [609008.503771] 
Aug 17 21:06:39 min25jetson kernel: [609008.503776] ---- syncpts ----
Aug 17 21:06:39 min25jetson kernel: [609008.503799] id 5 (gpu.0_506) min 152488 max 152488 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503806] id 6 (gpu.0_505) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503815] id 7 (nvdec_video_record_0) min 6086786 max 6086786 refs 4 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503824] id 8 (msenc_omxh264dec-omxh_0) min 6086761 max 6086761 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503834] id 9 (vic03_video_record_0) min 18260325 max 18260325 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503844] id 11 (vic03_video_record_0) min 18260283 max 18260283 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503851] id 12 (nvdec_video_record_0) min 6087657 max 6087657 refs 4 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503862] id 13 (msenc_omxh264dec-omxh_0) min 6087629 max 6087629 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503871] id 14 (vic03_video_record_0) min 18262899 max 18262899 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503878] id 15 (vic03_video_record_0) min 18262887 max 18262887 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.503888] id 16 (gpu.0_504) min 76326592 max 76326592 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503897] id 17 (gpu.0_503) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503904] id 18 (gpu.0_502) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503911] id 19 (gpu.0_501) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503917] id 20 (gpu.0_500) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503924] id 21 (gpu.0_499) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503930] id 22 (gpu.0_498) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503937] id 23 (gpu.0_497) min 76347520 max 76347520 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503944] id 24 (gpu.0_496) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503951] id 25 (gpu.0_495) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503963] id 28 (gpu.0_494) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503969] id 29 (gpu.0_493) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503978] id 30 (gpu.0_492) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503984] id 31 (gpu.0_491) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503991] id 32 (gpu.0_490) min 76357388 max 76357388 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.503998] id 33 (gpu.0_489) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504005] id 34 (gpu.0_488) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504011] id 35 (gpu.0_487) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504018] id 36 (gpu.0_486) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504025] id 37 (gpu.0_485) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504031] id 38 (gpu.0_484) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504039] id 39 (gpu.0_483) min 76364672 max 76364672 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504045] id 40 (gpu.0_482) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504051] id 41 (gpu.0_481) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504058] id 42 (gpu.0_480) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504065] id 43 (gpu.0_479) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504072] id 44 (gpu.0_478) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504078] id 45 (gpu.0_477) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.504086] id 46 (nvdec_video_record_0) min 7151708 max 7151708 refs 2 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504093] id 47 (msenc_omxh264dec-omxh_0) min 6086630 max 6086630 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504103] id 48 (vic03_video_record_0) min 15726814 max 15726814 refs 10 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504111] id 49 (vic03_video_record_0) min 14661688 max 14661688 refs 2 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504118] id 50 (nvdec_video_record_0) min 6087320 max 6087320 refs 3 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504127] id 51 (msenc_omxh264dec-omxh_0) min 6087250 max 6087250 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504137] id 52 (nvdec_video_record_0) min 6086770 max 6086770 refs 1 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504145] id 53 (nvdec_video_record_0) min 6087593 max 6087593 refs 1 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504152] id 54 (msenc_omxh264dec-omxh_0) min 6086695 max 6086695 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504161] id 55 (msenc_omxh264dec-omxh_0) min 6087516 max 6087516 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504171] id 56 (nvdec_video_record_0) min 6087033 max 6087033 refs 6 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504179] id 57 (msenc_omxh264dec-omxh_0) min 6086988 max 6086988 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504188] id 58 (nvdec_video_record_0) min 8620730 max 8620730 refs 4 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504198] id 59 (msenc_omxh264dec-omxh_0) min 8620686 max 8620686 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504205] id 60 (vic03_video_record_0) min 18259287 max 18259287 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504213] id 61 (vic03_video_record_0) min 18259212 max 18259212 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504221] id 62 (vic03_video_record_0) min 18261924 max 18261924 refs 11 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504228] id 63 (vic03_video_record_0) min 18261756 max 18261756 refs 2 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504236] id 64 (vic03_video_record_0) min 18260238 max 18260238 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504243] id 65 (vic03_video_record_0) min 18262710 max 18262710 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504252] id 66 (vic03_video_record_0) min 18260085 max 18260085 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504262] id 67 (vic03_video_record_0) min 18262548 max 18262548 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504269] id 68 (vic03_video_record_0) min 18261066 max 18261066 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504279] id 69 (vic03_video_record_0) min 18260964 max 18260964 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.504568] 
Aug 17 21:06:39 min25jetson kernel: [609008.504572] ---- channels ----
Aug 17 21:06:39 min25jetson kernel: [609008.504598] 1-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.504607] DMAPUT 800041f8, DMAGET 800041f8, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.504611] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.504620] 
Aug 17 21:06:39 min25jetson kernel: [609008.504620] ffffffc0e15e8000: JOB, syncpt_id=46, syncpt_val=7151708, first_get=800041c0, timeout=10000, num_slots=7, num_handles=11
Aug 17 21:06:39 min25jetson kernel: [609008.504633]     GATHER at 80090000+0f50, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.504637] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00804810 10100002 00000140 00803820 10100002 00000102 0084c000 10100002 00000104 00804820 10100002 00000103 0005a866 10100002 00000109 00804830 10100002 00000105 0082dc00 10100002 00000106 008048c0 10100002 00000108 00803830 10100002 0000010c 00867600 10100002 0000011d 0086a400 10100002 0000010d 00867600 10100002 0000011e 0086a400 10100002 0000010e 00867600 10100002 0000011f 0086a400 10100002 0000010f 00867600 10100002 00000120 0086a400 10100002 00000110 00867600 10100002 00000121 0086a400 10100002 00000111 00867600 10100002 00000122 0086a400 10100002 00000112 00867600 10100002 00000123 0086a400 10100002 00000113 00867600 10100002 00000124 0086a400 10100002 00000114 00867600 10100002 00000125 0086a400 10100002 00000115 00867600 10100002 00000126 0086a400 10100002 00000116 00867600 10100002 00000127 0086a400 10100002 00000117 00867600 10100002 00000128 0086a400 10100002 00000118 00867600 10100002 00000129 0086a400 10100002 00000119 00867600 10100002 0000012a 0086a400 10100002 0000011a 00867600 10100002 0000012b 0086a400 10100002 0000011b 00867600 10100002 0000012c 0086a400 10100002 0000011c 00867600 10100002 0000012d 0086a400 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.504937]     GATHER at 80090000+1178, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.504941] 20000001 0000012e 
Aug 17 21:06:39 min25jetson kernel: [609008.504949] 
Aug 17 21:06:39 min25jetson kernel: [609008.504966] 4-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.504974] DMAPUT 80010c28, DMAGET 80010c28, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.504979] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.504986] 
Aug 17 21:06:39 min25jetson kernel: [609008.504986] ffffffc0f27d2000: JOB, syncpt_id=58, syncpt_val=8620728, first_get=80010b80, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.504997]     GATHER at 80100000+0af0, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.505002] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00804920 10100002 00000140 00804950 10100002 00000102 00886000 10100002 00000104 00804930 10100002 00000103 0004d0c4 10100002 00000109 00804940 10100002 00000105 00883c00 10100002 00000106 0080be00 10100002 00000108 00804960 10100002 0000010c 00889e00 10100002 0000011d 0088a000 10100002 0000010d 0088ce00 10100002 0000011e 0088d000 10100002 0000010e 0088d200 10100002 0000011f 0088d400 10100002 0000010f 00889e00 10100002 00000120 0088a000 10100002 00000110 00889e00 10100002 00000121 0088a000 10100002 00000111 00889e00 10100002 00000122 0088a000 10100002 00000112 00889e00 10100002 00000123 0088a000 10100002 00000113 00889e00 10100002 00000124 0088a000 10100002 00000114 00889e00 10100002 00000125 0088a000 10100002 00000115 00889e00 10100002 00000126 0088a000 10100002 00000116 00889e00 10100002 00000127 0088a000 10100002 00000117 00889e00 10100002 00000128 0088a000 10100002 00000118 00889e00 10100002 00000129 0088a000 10100002 00000119 00889e00 10100002 0000012a 0088a000 10100002 0000011a 00889e00 10100002 0000012b 0088a000 10100002 0000011b 00889e00 10100002 0000012c 0088a000 10100002 0000011c 00889e00 10100002 0000012d 0088a000 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.505297]     GATHER at 80100000+0d18, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.505301] 20000001 0000013a 
Aug 17 21:06:39 min25jetson kernel: [609008.505309] 
Aug 17 21:06:39 min25jetson kernel: [609008.505328] 5-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.505336] DMAPUT 80014a20, DMAGET 80014a20, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.505340] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.505347] 
Aug 17 21:06:39 min25jetson kernel: [609008.505347] ffffffc0cf789000: JOB, syncpt_id=50, syncpt_val=6087319, first_get=800149b0, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.505358]     GATHER at 80078000+2df0, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.505362] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00806f00 10100002 00000140 00806f40 10100002 00000102 00858000 10100002 00000104 00806f80 10100002 00000103 0007185d 10100002 00000109 00806f10 10100002 00000105 00854000 10100002 00000106 0080d100 10100002 00000108 008049f0 10100002 0000010c 00879000 10100002 0000011d 00878800 10100002 0000010d 0086e000 10100002 0000011e 00867800 10100002 0000010e 00875000 10100002 0000011f 00871800 10100002 0000010f 00875000 10100002 00000120 00871800 10100002 00000110 00875000 10100002 00000121 00871800 10100002 00000111 00875000 10100002 00000122 00871800 10100002 00000112 00875000 10100002 00000123 00871800 10100002 00000113 00875000 10100002 00000124 00871800 10100002 00000114 00875000 10100002 00000125 00871800 10100002 00000115 00875000 10100002 00000126 00871800 10100002 00000116 00875000 10100002 00000127 00871800 10100002 00000117 00875000 10100002 00000128 00871800 10100002 00000118 00875000 10100002 00000129 00871800 10100002 00000119 00875000 10100002 0000012a 00871800 10100002 0000011a 00875000 10100002 0000012b 00871800 10100002 0000011b 00875000 10100002 0000012c 00871800 10100002 0000011c 00875000 10100002 0000012d 00871800 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.505656]     GATHER at 80078000+3018, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.505660] 20000001 00000132 
Aug 17 21:06:39 min25jetson kernel: [609008.505667] 
Aug 17 21:06:39 min25jetson kernel: [609008.505689] 9-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.505697] DMAPUT 80024a50, DMAGET 80024a50, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.505701] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.505708] 
Aug 17 21:06:39 min25jetson kernel: [609008.505708] ffffffc0db3d7000: JOB, syncpt_id=7, syncpt_val=6086784, first_get=800249a8, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.505717]     GATHER at 80040000+0690, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.505721] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00800730 10100002 00000140 00800760 10100002 00000102 00801000 10100002 00000104 00800740 10100002 00000103 0004b378 10100002 00000109 00800750 10100002 00000105 00801c00 10100002 00000106 00800780 10100002 00000108 00800770 10100002 0000010c 0080ba00 10100002 0000011d 0080bc00 10100002 0000010d 008e5200 10100002 0000011e 008e5400 10100002 0000010e 00800e00 10100002 0000011f 00801a00 10100002 0000010f 0080ba00 10100002 00000120 0080bc00 10100002 00000110 0080ba00 10100002 00000121 0080bc00 10100002 00000111 0080ba00 10100002 00000122 0080bc00 10100002 00000112 0080ba00 10100002 00000123 0080bc00 10100002 00000113 0080ba00 10100002 00000124 0080bc00 10100002 00000114 0080ba00 10100002 00000125 0080bc00 10100002 00000115 0080ba00 10100002 00000126 0080bc00 10100002 00000116 0080ba00 10100002 00000127 0080bc00 10100002 00000117 0080ba00 10100002 00000128 0080bc00 10100002 00000118 0080ba00 10100002 00000129 0080bc00 10100002 00000119 0080ba00 10100002 0000012a 0080bc00 10100002 0000011a 0080ba00 10100002 0000012b 0080bc00 10100002 0000011b 0080ba00 10100002 0000012c 0080bc00 10100002 0000011c 0080ba00 10100002 0000012d 0080bc00 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.506016]     GATHER at 80040000+08b8, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.506020] 20000001 00000107 
Aug 17 21:06:39 min25jetson kernel: [609008.506027] 
Aug 17 21:06:39 min25jetson kernel: [609008.506051] 10-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.506058] DMAPUT 800281b8, DMAGET 800281b8, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.506062] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.506069] 
Aug 17 21:06:39 min25jetson kernel: [609008.506069] ffffffc0db24a000: JOB, syncpt_id=56, syncpt_val=6087029, first_get=800280a0, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.506078]     GATHER at 80084000+2bc0, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.506082] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00806a90 10100002 00000140 00806a30 10100002 00000102 0080e000 10100002 00000104 00806aa0 10100002 00000103 00042802 10100002 00000109 00806ab0 10100002 00000105 0080cc00 10100002 00000106 00806a40 10100002 00000108 00806a80 10100002 0000010c 00810c00 10100002 0000011d 00810e00 10100002 0000010d 00810400 10100002 0000011e 00810600 10100002 0000010e 00810800 10100002 0000011f 00810a00 10100002 0000010f 00810400 10100002 00000120 00810600 10100002 00000110 00810400 10100002 00000121 00810600 10100002 00000111 00810400 10100002 00000122 00810600 10100002 00000112 00810400 10100002 00000123 00810600 10100002 00000113 00810400 10100002 00000124 00810600 10100002 00000114 00810400 10100002 00000125 00810600 10100002 00000115 00810400 10100002 00000126 00810600 10100002 00000116 00810400 10100002 00000127 00810600 10100002 00000117 00810400 10100002 00000128 00810600 10100002 00000118 00810400 10100002 00000129 00810600 10100002 00000119 00810400 10100002 0000012a 00810600 10100002 0000011a 00810400 10100002 0000012b 00810600 10100002 0000011b 00810400 10100002 0000012c 00810600 10100002 0000011c 00810400 10100002 0000012d 00810600 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.506377]     GATHER at 80084000+2de8, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.506381] 20000001 00000138 
Aug 17 21:06:39 min25jetson kernel: [609008.506388] 
Aug 17 21:06:39 min25jetson kernel: [609008.506411] 11-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.506418] DMAPUT 8002cb80, DMAGET 8002cb80, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.506423] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.506429] 
Aug 17 21:06:39 min25jetson kernel: [609008.506429] ffffffc0308e7000: JOB, syncpt_id=12, syncpt_val=6087655, first_get=8002cad8, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.506439]     GATHER at 80060000+0d20, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.506443] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00800be0 10100002 00000140 00801880 10100002 00000102 00812000 10100002 00000104 00801840 10100002 00000103 00057729 10100002 00000109 00800bf0 10100002 00000105 00814000 10100002 00000106 00803900 10100002 00000108 00800de0 10100002 0000010c 00841000 10100002 0000011d 00842000 10100002 0000010d 00843000 10100002 0000011e 00842800 10100002 0000010e 00829000 10100002 0000011f 00828800 10100002 0000010f 00841000 10100002 00000120 00842000 10100002 00000110 00841000 10100002 00000121 00842000 10100002 00000111 00841000 10100002 00000122 00842000 10100002 00000112 00841000 10100002 00000123 00842000 10100002 00000113 00841000 10100002 00000124 00842000 10100002 00000114 00841000 10100002 00000125 00842000 10100002 00000115 00841000 10100002 00000126 00842000 10100002 00000116 00841000 10100002 00000127 00842000 10100002 00000117 00841000 10100002 00000128 00842000 10100002 00000118 00841000 10100002 00000129 00842000 10100002 00000119 00841000 10100002 0000012a 00842000 10100002 0000011a 00841000 10100002 0000012b 00842000 10100002 0000011b 00841000 10100002 0000012c 00842000 10100002 0000011c 00841000 10100002 0000012d 00842000 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.506734]     GATHER at 80060000+0f48, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.506738] 20000001 0000010c 
Aug 17 21:06:39 min25jetson kernel: [609008.506745] 
Aug 17 21:06:39 min25jetson kernel: [609008.506761] 0-vic03 (21): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.506768] DMAPUT 80000a00, DMAGET 80000a00, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.506773] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.506779] 
Aug 17 21:06:39 min25jetson kernel: [609008.506779] ffffffc0c7446000: JOB, syncpt_id=48, syncpt_val=15726790, first_get=800007c0, timeout=10000, num_slots=8, num_handles=8
Aug 17 21:06:39 min25jetson kernel: [609008.506790]     GATHER at 80074000+3058, 32 words
Aug 17 21:06:39 min25jetson kernel: [609008.506794] 10100002 000001c1 00610110 10100002 000001c2 00804900 10100002 000001cd 00000000 10100002 000001c3 008049c0 10100002 000001cc 00804910 10100002 000001c8 0084da00 10100002 000001c9 0084dc00 10100002 00000100 0084ae00 10100002 00000101 0084ba00 10100002 000000c0 00000100 20000001 00000130 
Aug 17 21:06:39 min25jetson kernel: [609008.506865]     GATHER at 80074000+30d8, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.506870] 20000001 00000130 
Aug 17 21:06:39 min25jetson kernel: [609008.506877] 
Aug 17 21:06:39 min25jetson kernel: [609008.506900] 2-vic03 (21): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.506907] DMAPUT 80008998, DMAGET 80008998, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.506911] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.506918] 
Aug 17 21:06:39 min25jetson kernel: [609008.506918] ffffffc0fa430c00: JOB, syncpt_id=49, syncpt_val=14661688, first_get=80008958, timeout=10000, num_slots=8, num_handles=8
Aug 17 21:06:39 min25jetson kernel: [609008.506928]     GATHER at 80088000+2e38, 32 words
Aug 17 21:06:39 min25jetson kernel: [609008.506932] 10100002 000001c1 00610110 10100002 000001c2 00804980 10100002 000001cd 00000000 10100002 000001c3 00813580 10100002 000001cc 0080e970 10100002 000001c8 0084e200 10100002 000001c9 0084e400 10100002 00000100 0084e600 10100002 00000101 0084f000 10100002 000000c0 00000100 20000001 00000131 
Aug 17 21:06:39 min25jetson kernel: [609008.507003]     GATHER at 80088000+2eb8, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.507007] 20000001 00000131 
Aug 17 21:06:39 min25jetson kernel: [609008.507014] 
Aug 17 21:06:39 min25jetson kernel: [609008.507027] 6-vic03 (21): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.507034] DMAPUT 80018e38, DMAGET 80018e38, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507038] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.507045] 
Aug 17 21:06:39 min25jetson kernel: [609008.507045] ffffffc0160c8800: JOB, syncpt_id=62, syncpt_val=18261897, first_get=80018bb8, timeout=10000, num_slots=8, num_handles=8
Aug 17 21:06:39 min25jetson kernel: [609008.507055]     GATHER at 800a0000+1a90, 32 words
Aug 17 21:06:39 min25jetson kernel: [609008.507059] 10100002 000001c1 00610110 10100002 000001c2 00806fd0 10100002 000001cd 00000000 10100002 000001c3 0080c8c0 10100002 000001cc 0080c810 10100002 000001c8 008a1000 10100002 000001c9 008a0800 10100002 00000100 008fd000 10100002 00000101 008fc800 10100002 000000c0 00000100 20000001 0000013e 
Aug 17 21:06:39 min25jetson kernel: [609008.507131]     GATHER at 800a0000+1b10, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.507135] 20000001 0000013e 
Aug 17 21:06:39 min25jetson kernel: [609008.507142] 
Aug 17 21:06:39 min25jetson kernel: [609008.507161] 7-vic03 (21): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.507168] DMAPUT 8001c7d0, DMAGET 8001c7d0, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507172] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.507179] 
Aug 17 21:06:39 min25jetson kernel: [609008.507179] ffffffc0f339c000: JOB, syncpt_id=63, syncpt_val=18261756, first_get=8001c790, timeout=10000, num_slots=8, num_handles=8
Aug 17 21:06:39 min25jetson kernel: [609008.507189]     GATHER at 800a4000+18f8, 32 words
Aug 17 21:06:39 min25jetson kernel: [609008.507193] 10100002 000001c1 00610110 10100002 000001c2 0080c820 10100002 000001cd 00000000 10100002 000001c3 0080d040 10100002 000001cc 0080c830 10100002 000001c8 008a9000 10100002 000001c9 008aa000 10100002 00000100 008af000 10100002 00000101 008b5800 10100002 000000c0 00000100 20000001 0000013f 
Aug 17 21:06:39 min25jetson kernel: [609008.507264]     GATHER at 800a4000+1978, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.507268] 20000001 0000013f 
Aug 17 21:06:39 min25jetson kernel: [609008.507275] 
Aug 17 21:06:39 min25jetson kernel: [609008.507295] 3-msenc (2): inactive
Aug 17 21:06:39 min25jetson kernel: [609008.507295] 
Aug 17 21:06:39 min25jetson kernel: [609008.507319] 8-msenc (2): inactive
Aug 17 21:06:39 min25jetson kernel: [609008.507319] 
Aug 17 21:06:39 min25jetson kernel: [609008.507327] 
Aug 17 21:06:39 min25jetson kernel: [609008.507327] ---- host general irq ----
Aug 17 21:06:39 min25jetson kernel: [609008.507327] 
Aug 17 21:06:39 min25jetson kernel: [609008.507336] sync_hintmask_ext = 0xc0000000
Aug 17 21:06:39 min25jetson kernel: [609008.507343] sync_hintmask = 0x80002000
Aug 17 21:06:39 min25jetson kernel: [609008.507349] sync_intc0mask = 0x00000001
Aug 17 21:06:39 min25jetson kernel: [609008.507355] sync_intmask = 0x00000011
Aug 17 21:06:39 min25jetson kernel: [609008.507358] 
Aug 17 21:06:39 min25jetson kernel: [609008.507358] ---- host syncpt irq mask ----
Aug 17 21:06:39 min25jetson kernel: [609008.507358] 
Aug 17 21:06:39 min25jetson kernel: [609008.507367] syncpt_thresh_int_mask(0) = 0x00000001
Aug 17 21:06:39 min25jetson kernel: [609008.507373] syncpt_thresh_int_mask(1) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507380] syncpt_thresh_int_mask(2) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507385] syncpt_thresh_int_mask(3) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507391] syncpt_thresh_int_mask(4) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507397] syncpt_thresh_int_mask(5) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507403] syncpt_thresh_int_mask(6) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507410] syncpt_thresh_int_mask(7) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507415] syncpt_thresh_int_mask(8) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507421] syncpt_thresh_int_mask(9) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507427] syncpt_thresh_int_mask(10) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507434] syncpt_thresh_int_mask(11) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507438] 
Aug 17 21:06:39 min25jetson kernel: [609008.507438] ---- host syncpt irq status ----
Aug 17 21:06:39 min25jetson kernel: [609008.507438] 
Aug 17 21:06:39 min25jetson kernel: [609008.507445] syncpt_thresh_cpu0_int_status(0) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507451] syncpt_thresh_cpu0_int_status(1) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507457] syncpt_thresh_cpu0_int_status(2) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507464] syncpt_thresh_cpu0_int_status(3) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507470] syncpt_thresh_cpu0_int_status(4) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507476] syncpt_thresh_cpu0_int_status(5) = 0x00000000
Aug 17 21:06:39 min25jetson kernel: [609008.507480] 
Aug 17 21:06:39 min25jetson kernel: [609008.507480] ---- host syncpt thresh ----
Aug 17 21:06:39 min25jetson kernel: [609008.507480] 
Aug 17 21:06:39 min25jetson kernel: [609008.507491] syncpt_int_thresh_thresh_0(0) = 1
Aug 17 21:06:39 min25jetson kernel: [609008.507995] gpu.0 pbdma 0: 
Aug 17 21:06:39 min25jetson kernel: [609008.508001] id: 0 (tsg), next_id: 0 (tsg) status: invalid
Aug 17 21:06:39 min25jetson kernel: [609008.508012] PUT: 000000200002ab60 GET: 000000200002ab60 FETCH: 000006c0 HEADER: 60400000
Aug 17 21:06:39 min25jetson kernel: [609008.508016] 
Aug 17 21:06:39 min25jetson kernel: [609008.508021] gpu.0 eng 0: 
Aug 17 21:06:39 min25jetson kernel: [609008.508025] id: 0 (tsg), next_id: 0 (tsg), ctx: invalid <6>[609008.508029] 
Aug 17 21:06:39 min25jetson kernel: [609008.508034] gpu.0 eng 1: 
Aug 17 21:06:39 min25jetson kernel: [609008.508038] id: 0 (tsg), next_id: 0 (tsg), ctx: invalid <6>[609008.508042] 
Aug 17 21:06:39 min25jetson kernel: [609008.508045] 
Aug 17 21:06:39 min25jetson kernel: [609008.508487] 477-gpu.0, pid 2091, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508492]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508500] TOP: 8000002000760018 PUT: 0000002000760018 GET: 0000002000760018 FETCH: 0000002000760018
Aug 17 21:06:39 min25jetson kernel: [609008.508500] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508500] SYNCPOINT 00000000 00002d01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508506] 
Aug 17 21:06:39 min25jetson kernel: [609008.508512] 478-gpu.0, pid 2091, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508515]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508523] TOP: 8000002000700018 PUT: 0000002000700018 GET: 0000002000700018 FETCH: 0000002000700018
Aug 17 21:06:39 min25jetson kernel: [609008.508523] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508523] SYNCPOINT 00000000 00002c01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508529] 
Aug 17 21:06:39 min25jetson kernel: [609008.508534] 479-gpu.0, pid 2091, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508537]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508545] TOP: 8000002000580018 PUT: 0000002000580018 GET: 0000002000580018 FETCH: 0000002000580018
Aug 17 21:06:39 min25jetson kernel: [609008.508545] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508545] SYNCPOINT 00000000 00002b01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508551] 
Aug 17 21:06:39 min25jetson kernel: [609008.508556] 480-gpu.0, pid 2091, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508559]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508566] TOP: 80000020003c0018 PUT: 00000020003c0018 GET: 00000020003c0018 FETCH: 00000020003c0018
Aug 17 21:06:39 min25jetson kernel: [609008.508566] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508566] SYNCPOINT 00000000 00002a01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508572] 
Aug 17 21:06:39 min25jetson kernel: [609008.508577] 481-gpu.0, pid 2091, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508580]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508588] TOP: 80000020001e0018 PUT: 00000020001e0018 GET: 00000020001e0018 FETCH: 00000020001e0018
Aug 17 21:06:39 min25jetson kernel: [609008.508588] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508588] SYNCPOINT 00000000 00002901 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508593] 
Aug 17 21:06:39 min25jetson kernel: [609008.508599] 482-gpu.0, pid 2091, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508602]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508609] TOP: 8000002000180030 PUT: 0000002000180030 GET: 0000002000180030 FETCH: 0000002000180030
Aug 17 21:06:39 min25jetson kernel: [609008.508609] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508609] SYNCPOINT 00000000 00002801 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508615] 
Aug 17 21:06:39 min25jetson kernel: [609008.508620] 483-gpu.0, pid 2091, refs: 1122: 
Aug 17 21:06:39 min25jetson kernel: [609008.508623]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508631] TOP: 800000200003a478 PUT: 000000200003a478 GET: 000000200003a478 FETCH: 000000200003a478
Aug 17 21:06:39 min25jetson kernel: [609008.508631] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508631] SYNCPOINT 00000000 00002701 SEMAPHORE 00000001 0000fbd0 00000002 00000004
Aug 17 21:06:39 min25jetson kernel: [609008.508637] 
Aug 17 21:06:39 min25jetson kernel: [609008.508642] 484-gpu.0, pid 2069, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508645]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508653] TOP: 8000002000760018 PUT: 0000002000760018 GET: 0000002000760018 FETCH: 0000002000760018
Aug 17 21:06:39 min25jetson kernel: [609008.508653] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508653] SYNCPOINT 00000000 00002601 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508658] 
Aug 17 21:06:39 min25jetson kernel: [609008.508663] 485-gpu.0, pid 2069, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508667]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508674] TOP: 8000002000700018 PUT: 0000002000700018 GET: 0000002000700018 FETCH: 0000002000700018
Aug 17 21:06:39 min25jetson kernel: [609008.508674] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508674] SYNCPOINT 00000000 00002501 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508680] 
Aug 17 21:06:39 min25jetson kernel: [609008.508687] 486-gpu.0, pid 2069, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508690]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508698] TOP: 8000002000580018 PUT: 0000002000580018 GET: 0000002000580018 FETCH: 0000002000580018
Aug 17 21:06:39 min25jetson kernel: [609008.508698] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508698] SYNCPOINT 00000000 00002401 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508703] 
Aug 17 21:06:39 min25jetson kernel: [609008.508708] 487-gpu.0, pid 2069, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508712]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508719] TOP: 80000020003c0018 PUT: 00000020003c0018 GET: 00000020003c0018 FETCH: 00000020003c0018
Aug 17 21:06:39 min25jetson kernel: [609008.508719] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508719] SYNCPOINT 00000000 00002301 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508724] 
Aug 17 21:06:39 min25jetson kernel: [609008.508729] 488-gpu.0, pid 2069, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508733]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508740] TOP: 80000020001e0018 PUT: 00000020001e0018 GET: 00000020001e0018 FETCH: 00000020001e0018
Aug 17 21:06:39 min25jetson kernel: [609008.508740] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508740] SYNCPOINT 00000000 00002201 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508745] 
Aug 17 21:06:39 min25jetson kernel: [609008.508750] 489-gpu.0, pid 2069, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508753]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508760] TOP: 8000002000180030 PUT: 0000002000180030 GET: 0000002000180030 FETCH: 0000002000180030
Aug 17 21:06:39 min25jetson kernel: [609008.508760] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508760] SYNCPOINT 00000000 00002101 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508766] 
Aug 17 21:06:39 min25jetson kernel: [609008.508771] 490-gpu.0, pid 2069, refs: 1122: 
Aug 17 21:06:39 min25jetson kernel: [609008.508775]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508782] TOP: 8000002000024f08 PUT: 0000002000024f08 GET: 0000002000024f08 FETCH: 0000002000024f08
Aug 17 21:06:39 min25jetson kernel: [609008.508782] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508782] SYNCPOINT 00000000 00002001 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508788] 
Aug 17 21:06:39 min25jetson kernel: [609008.508793] 491-gpu.0, pid 2022, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508796]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508803] TOP: 8000002000760018 PUT: 0000002000760018 GET: 0000002000760018 FETCH: 0000002000760018
Aug 17 21:06:39 min25jetson kernel: [609008.508803] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508803] SYNCPOINT 00000000 00001f01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508809] 
Aug 17 21:06:39 min25jetson kernel: [609008.508814] 492-gpu.0, pid 2022, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508817]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508825] TOP: 8000002000700018 PUT: 0000002000700018 GET: 0000002000700018 FETCH: 0000002000700018
Aug 17 21:06:39 min25jetson kernel: [609008.508825] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508825] SYNCPOINT 00000000 00001e01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508830] 
Aug 17 21:06:39 min25jetson kernel: [609008.508836] 493-gpu.0, pid 2022, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508839]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508847] TOP: 8000002000580018 PUT: 0000002000580018 GET: 0000002000580018 FETCH: 0000002000580018
Aug 17 21:06:39 min25jetson kernel: [609008.508847] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508847] SYNCPOINT 00000000 00001d01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508853] 
Aug 17 21:06:39 min25jetson kernel: [609008.508858] 494-gpu.0, pid 2022, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508861]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508868] TOP: 80000020003c0018 PUT: 00000020003c0018 GET: 00000020003c0018 FETCH: 00000020003c0018
Aug 17 21:06:39 min25jetson kernel: [609008.508868] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508868] SYNCPOINT 00000000 00001c01 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508874] 
Aug 17 21:06:39 min25jetson kernel: [609008.508879] 495-gpu.0, pid 2022, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508882]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508889] TOP: 80000020001e0018 PUT: 00000020001e0018 GET: 00000020001e0018 FETCH: 00000020001e0018
Aug 17 21:06:39 min25jetson kernel: [609008.508889] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508889] SYNCPOINT 00000000 00001901 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508895] 
Aug 17 21:06:39 min25jetson kernel: [609008.508900] 496-gpu.0, pid 2022, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508903]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508910] TOP: 8000002000180030 PUT: 0000002000180030 GET: 0000002000180030 FETCH: 0000002000180030
Aug 17 21:06:39 min25jetson kernel: [609008.508910] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508910] SYNCPOINT 00000000 00001801 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508915] 
Aug 17 21:06:39 min25jetson kernel: [609008.508920] 497-gpu.0, pid 2022, refs: 1122: 
Aug 17 21:06:39 min25jetson kernel: [609008.508924]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508931] TOP: 8000002000028070 PUT: 0000002000028070 GET: 0000002000028070 FETCH: 0000002000028070
Aug 17 21:06:39 min25jetson kernel: [609008.508931] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508931] SYNCPOINT 00000000 00001701 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508936] 
Aug 17 21:06:39 min25jetson kernel: [609008.508942] 498-gpu.0, pid 2012, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508945]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508952] TOP: 8000002000760018 PUT: 0000002000760018 GET: 0000002000760018 FETCH: 0000002000760018
Aug 17 21:06:39 min25jetson kernel: [609008.508952] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508952] SYNCPOINT 00000000 00001601 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508958] 
Aug 17 21:06:39 min25jetson kernel: [609008.508963] 499-gpu.0, pid 2012, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508966]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508974] TOP: 8000002000700018 PUT: 0000002000700018 GET: 0000002000700018 FETCH: 0000002000700018
Aug 17 21:06:39 min25jetson kernel: [609008.508974] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508974] SYNCPOINT 00000000 00001501 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.508979] 
Aug 17 21:06:39 min25jetson kernel: [609008.508984] 500-gpu.0, pid 2012, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.508987]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.508995] TOP: 8000002000580018 PUT: 0000002000580018 GET: 0000002000580018 FETCH: 0000002000580018
Aug 17 21:06:39 min25jetson kernel: [609008.508995] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.508995] SYNCPOINT 00000000 00001401 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509000] 
Aug 17 21:06:39 min25jetson kernel: [609008.509006] 501-gpu.0, pid 2012, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509009]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509016] TOP: 80000020003c0018 PUT: 00000020003c0018 GET: 00000020003c0018 FETCH: 00000020003c0018
Aug 17 21:06:39 min25jetson kernel: [609008.509016] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.509016] SYNCPOINT 00000000 00001301 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509022] 
Aug 17 21:06:39 min25jetson kernel: [609008.509031] 502-gpu.0, pid 2012, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509035]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509042] TOP: 80000020001e0018 PUT: 00000020001e0018 GET: 00000020001e0018 FETCH: 00000020001e0018
Aug 17 21:06:39 min25jetson kernel: [609008.509042] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.509042] SYNCPOINT 00000000 00001201 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509047] 
Aug 17 21:06:39 min25jetson kernel: [609008.509052] 503-gpu.0, pid 2012, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509055]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509062] TOP: 8000002000180030 PUT: 0000002000180030 GET: 0000002000180030 FETCH: 0000002000180030
Aug 17 21:06:39 min25jetson kernel: [609008.509062] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.509062] SYNCPOINT 00000000 00001101 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509068] 
Aug 17 21:06:39 min25jetson kernel: [609008.509073] 504-gpu.0, pid 2012, refs: 1122: 
Aug 17 21:06:39 min25jetson kernel: [609008.509076]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509083] TOP: 800000200002ab60 PUT: 000000200002ab60 GET: 000000200002ab60 FETCH: 000000200002ab60
Aug 17 21:06:39 min25jetson kernel: [609008.509083] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.509083] SYNCPOINT 00000000 00001001 SEMAPHORE 00000001 0000fbd0 00000002 00000004
Aug 17 21:06:39 min25jetson kernel: [609008.509089] 
Aug 17 21:06:39 min25jetson kernel: [609008.509094] 505-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509097]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509104] TOP: 8000002000180040 PUT: 0000002000180040 GET: 0000002000180040 FETCH: 0000002000180040
Aug 17 21:06:39 min25jetson kernel: [609008.509104] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.509104] SYNCPOINT 00000000 00000601 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509110] 
Aug 17 21:06:39 min25jetson kernel: [609008.509118] 506-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509121]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509128] TOP: 8000002000033a80 PUT: 0000002000033a80 GET: 0000002000033a80 FETCH: 0000002000033a80
Aug 17 21:06:39 min25jetson kernel: [609008.509128] HEADER: 60400000 COUNT: 80000000
Aug 17 21:06:39 min25jetson kernel: [609008.509128] SYNCPOINT 00000000 00000501 SEMAPHORE 0000001c 000c0000 00000000 01100002
Aug 17 21:06:39 min25jetson kernel: [609008.509134] 
Aug 17 21:06:39 min25jetson kernel: [609008.509139] 507-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509142]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509149] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Aug 17 21:06:39 min25jetson kernel: [609008.509149] HEADER: 60400000 COUNT: 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509149] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509155] 
Aug 17 21:06:39 min25jetson kernel: [609008.509160] 508-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509164]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509171] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Aug 17 21:06:39 min25jetson kernel: [609008.509171] HEADER: 60400000 COUNT: 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509171] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509176] 
Aug 17 21:06:39 min25jetson kernel: [609008.509182] 509-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509185]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509192] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Aug 17 21:06:39 min25jetson kernel: [609008.509192] HEADER: 60400000 COUNT: 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509192] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509197] 
Aug 17 21:06:39 min25jetson kernel: [609008.509205] 510-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509209]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509216] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Aug 17 21:06:39 min25jetson kernel: [609008.509216] HEADER: 60400000 COUNT: 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509216] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509221] 
Aug 17 21:06:39 min25jetson kernel: [609008.509226] 511-gpu.0, pid 1109, refs: 2: 
Aug 17 21:06:39 min25jetson kernel: [609008.509230]  in use idle not busy
Aug 17 21:06:39 min25jetson kernel: [609008.509236] TOP: 0000000000000000 PUT: 0000000000000000 GET: 0000000000000000 FETCH: 0000000000000000
Aug 17 21:06:39 min25jetson kernel: [609008.509236] HEADER: 60400000 COUNT: 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509236] SYNCPOINT 00000000 00000000 SEMAPHORE 00000000 00000000 00000000 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.509242] 
Aug 17 21:06:39 min25jetson kernel: [609008.509269] ---- mlocks ----
Aug 17 21:06:39 min25jetson kernel: [609008.509313] 
Aug 17 21:06:39 min25jetson kernel: [609008.509318] ---- syncpts ----
Aug 17 21:06:39 min25jetson kernel: [609008.509335] id 5 (gpu.0_506) min 152488 max 152488 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509343] id 6 (gpu.0_505) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509353] id 7 (nvdec_video_record_0) min 6086786 max 6086786 refs 4 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509360] id 8 (msenc_omxh264dec-omxh_0) min 6086761 max 6086761 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509368] id 9 (vic03_video_record_0) min 18260325 max 18260325 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509380] id 11 (vic03_video_record_0) min 18260283 max 18260283 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509387] id 12 (nvdec_video_record_0) min 6087657 max 6087657 refs 4 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509394] id 13 (msenc_omxh264dec-omxh_0) min 6087629 max 6087629 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509403] id 14 (vic03_video_record_0) min 18262899 max 18262899 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509410] id 15 (vic03_video_record_0) min 18262887 max 18262887 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509417] id 16 (gpu.0_504) min 76326592 max 76326592 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509424] id 17 (gpu.0_503) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509430] id 18 (gpu.0_502) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509437] id 19 (gpu.0_501) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509444] id 20 (gpu.0_500) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509450] id 21 (gpu.0_499) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509456] id 22 (gpu.0_498) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509464] id 23 (gpu.0_497) min 76347520 max 76347520 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509470] id 24 (gpu.0_496) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509477] id 25 (gpu.0_495) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509486] id 28 (gpu.0_494) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509494] id 29 (gpu.0_493) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509501] id 30 (gpu.0_492) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509509] id 31 (gpu.0_491) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509517] id 32 (gpu.0_490) min 76357388 max 76357388 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509525] id 33 (gpu.0_489) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509532] id 34 (gpu.0_488) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509538] id 35 (gpu.0_487) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509545] id 36 (gpu.0_486) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509551] id 37 (gpu.0_485) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509559] id 38 (gpu.0_484) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509566] id 39 (gpu.0_483) min 76364672 max 76364672 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509572] id 40 (gpu.0_482) min 4 max 4 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509578] id 41 (gpu.0_481) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509586] id 42 (gpu.0_480) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509593] id 43 (gpu.0_479) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509599] id 44 (gpu.0_478) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509605] id 45 (gpu.0_477) min 2 max 2 refs 1 (previous client : )
Aug 17 21:06:39 min25jetson kernel: [609008.509612] id 46 (nvdec_video_record_0) min 7151708 max 7151708 refs 2 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509620] id 47 (msenc_omxh264dec-omxh_0) min 6086630 max 6086630 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509628] id 48 (vic03_video_record_0) min 15726814 max 15726814 refs 10 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509635] id 49 (vic03_video_record_0) min 14661688 max 14661688 refs 2 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509642] id 50 (nvdec_video_record_0) min 6087320 max 6087320 refs 3 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509651] id 51 (msenc_omxh264dec-omxh_0) min 6087250 max 6087250 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509658] id 52 (nvdec_video_record_0) min 6086770 max 6086770 refs 1 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509665] id 53 (nvdec_video_record_0) min 6087593 max 6087593 refs 1 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509673] id 54 (msenc_omxh264dec-omxh_0) min 6086695 max 6086695 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509682] id 55 (msenc_omxh264dec-omxh_0) min 6087516 max 6087516 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509689] id 56 (nvdec_video_record_0) min 6087033 max 6087033 refs 6 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509696] id 57 (msenc_omxh264dec-omxh_0) min 6086988 max 6086988 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509703] id 58 (nvdec_video_record_0) min 8620730 max 8620730 refs 4 (previous client : nvdec_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509712] id 59 (msenc_omxh264dec-omxh_0) min 8620686 max 8620686 refs 1 (previous client : msenc_omxh264dec-omxh_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509719] id 60 (vic03_video_record_0) min 18259287 max 18259287 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509726] id 61 (vic03_video_record_0) min 18259212 max 18259212 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509733] id 62 (vic03_video_record_0) min 18261924 max 18261924 refs 11 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509740] id 63 (vic03_video_record_0) min 18261756 max 18261756 refs 2 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509749] id 64 (vic03_video_record_0) min 18260238 max 18260238 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509756] id 65 (vic03_video_record_0) min 18262710 max 18262710 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509763] id 66 (vic03_video_record_0) min 18260085 max 18260085 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509770] id 67 (vic03_video_record_0) min 18262548 max 18262548 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509777] id 68 (vic03_video_record_0) min 18261066 max 18261066 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.509784] id 69 (vic03_video_record_0) min 18260964 max 18260964 refs 1 (previous client : vic03_video_record_0)
Aug 17 21:06:39 min25jetson kernel: [609008.510068] 
Aug 17 21:06:39 min25jetson kernel: [609008.510072] ---- channels ----
Aug 17 21:06:39 min25jetson kernel: [609008.510093] 1-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.510102] DMAPUT 800041f8, DMAGET 800041f8, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.510107] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.510114] 
Aug 17 21:06:39 min25jetson kernel: [609008.510114] ffffffc0e15e8000: JOB, syncpt_id=46, syncpt_val=7151708, first_get=800041c0, timeout=10000, num_slots=7, num_handles=11
Aug 17 21:06:39 min25jetson kernel: [609008.510122]     GATHER at 80090000+0f50, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.510126] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00804810 10100002 00000140 00803820 10100002 00000102 0084c000 10100002 00000104 00804820 10100002 00000103 0005a866 10100002 00000109 00804830 10100002 00000105 0082dc00 10100002 00000106 008048c0 10100002 00000108 00803830 10100002 0000010c 00867600 10100002 0000011d 0086a400 10100002 0000010d 00867600 10100002 0000011e 0086a400 10100002 0000010e 00867600 10100002 0000011f 0086a400 10100002 0000010f 00867600 10100002 00000120 0086a400 10100002 00000110 00867600 10100002 00000121 0086a400 10100002 00000111 00867600 10100002 00000122 0086a400 10100002 00000112 00867600 10100002 00000123 0086a400 10100002 00000113 00867600 10100002 00000124 0086a400 10100002 00000114 00867600 10100002 00000125 0086a400 10100002 00000115 00867600 10100002 00000126 0086a400 10100002 00000116 00867600 10100002 00000127 0086a400 10100002 00000117 00867600 10100002 00000128 0086a400 10100002 00000118 00867600 10100002 00000129 0086a400 10100002 00000119 00867600 10100002 0000012a 0086a400 10100002 0000011a 00867600 10100002 0000012b 0086a400 10100002 0000011b 00867600 10100002 0000012c 0086a400 10100002 0000011c 00867600 10100002 0000012d 0086a400 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.510428]     GATHER at 80090000+1178, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.510432] 20000001 0000012e 
Aug 17 21:06:39 min25jetson kernel: [609008.510442] 
Aug 17 21:06:39 min25jetson kernel: [609008.510459] 4-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.510466] DMAPUT 80010c28, DMAGET 80010c28, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.510471] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.510477] 
Aug 17 21:06:39 min25jetson kernel: [609008.510477] ffffffc0f27d2000: JOB, syncpt_id=58, syncpt_val=8620728, first_get=80010b80, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.510483]     GATHER at 80100000+0af0, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.510487] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00804920 10100002 00000140 00804950 10100002 00000102 00886000 10100002 00000104 00804930 10100002 00000103 0004d0c4 10100002 00000109 00804940 10100002 00000105 00883c00 10100002 00000106 0080be00 10100002 00000108 00804960 10100002 0000010c 00889e00 10100002 0000011d 0088a000 10100002 0000010d 0088ce00 10100002 0000011e 0088d000 10100002 0000010e 0088d200 10100002 0000011f 0088d400 10100002 0000010f 00889e00 10100002 00000120 0088a000 10100002 00000110 00889e00 10100002 00000121 0088a000 10100002 00000111 00889e00 10100002 00000122 0088a000 10100002 00000112 00889e00 10100002 00000123 0088a000 10100002 00000113 00889e00 10100002 00000124 0088a000 10100002 00000114 00889e00 10100002 00000125 0088a000 10100002 00000115 00889e00 10100002 00000126 0088a000 10100002 00000116 00889e00 10100002 00000127 0088a000 10100002 00000117 00889e00 10100002 00000128 0088a000 10100002 00000118 00889e00 10100002 00000129 0088a000 10100002 00000119 00889e00 10100002 0000012a 0088a000 10100002 0000011a 00889e00 10100002 0000012b 0088a000 10100002 0000011b 00889e00 10100002 0000012c 0088a000 10100002 0000011c 00889e00 10100002 0000012d 0088a000 10100002 000000c0 00000100 
Aug 17 21:06:39 min25jetson kernel: [609008.510785]     GATHER at 80100000+0d18, 2 words
Aug 17 21:06:39 min25jetson kernel: [609008.510789] 20000001 0000013a 
Aug 17 21:06:39 min25jetson kernel: [609008.510797] 
Aug 17 21:06:39 min25jetson kernel: [609008.510820] 5-nvdec (17): active class 01, offset 0000, val 20000000
Aug 17 21:06:39 min25jetson kernel: [609008.510827] DMAPUT 80014a20, DMAGET 80014a20, DMACTL 00000000
Aug 17 21:06:39 min25jetson kernel: [609008.510832] CBREAD 20000000, CBSTAT 00010000
Aug 17 21:06:39 min25jetson kernel: [609008.510838] 
Aug 17 21:06:39 min25jetson kernel: [609008.510838] ffffffc0cf789000: JOB, syncpt_id=50, syncpt_val=6087319, first_get=800149b0, timeout=10000, num_slots=7, num_handles=15
Aug 17 21:06:39 min25jetson kernel: [609008.510844]     GATHER at 80078000+2df0, 138 words
Aug 17 21:06:39 min25jetson kernel: [609008.510849] 10100002 00000080 00000003 10100002 00000100 00000053 10100002 00000101 00806f00 10100002 00000140 00806f40 10100002 00000102 00858000 10100002 00000104 00806f80 10100002 00000103 0007185d 10100002 00000109 00806f10 10100002 00000105 00854000 10100002 00000106 0080d100 10100002 00000108 008049f0 10100002 0000010c 00879000 10100002 0000011d 00878800 10100002 0000010d 0086e000 10100002 0000011e 00867800 10100002 0000010e 00875000 10100002 0000011f 00871800 10100002 0000010f 00875000 10100002 00000120 00871800 10100002 00000110 00875000 10100002 00000121 00871800 10100002 00000111 00875000 10100002 00000122 00871800 10100002 00000112 00875000 10100002 00000123 00871800 10100002 00000113 00875000 10100002 00000124 00871800 10100002 00000114 00875000 10100002 00000125 00871800 10100002 00000115 00875000 10100002 00000126 00871800 10100002 00000116 00875000 10100002 00000127 00871800 10100002 00000117 00875000 10100002 00000128 00871800 10100002 00000118 00875000 10100002 00000129 00871800 10100002 00000119 00875000 10100002 0000012a 00871800 10100002 0000011a 00875000 10100002 0000012b 00871800 10100002 0000011b 00875000 10100002 0000012c 00871800 10100002 0000011c 00875000 10100002 0000012d 00871800 10100002 000000c0 00000100

jachen · August 22, 2016, 7:33am

Hello, Suraj:
Would you please help me to reproduce the memory leak issue you’ve met in my side? with shortage of memory, we cannot predict what will happen. So we need figure out where memory leak happens first.

The pipe line you provided @ #9 contains no NV-provided plugins. With hours running for that pipeline, I do not see the memory leak.

br
Chenjian

surajtn30 · August 22, 2016, 4:43pm

ChenJian,
Sorry, should have been more specific in my previous post. The pipeline I use is the one mentioned in #1 (more elaborate one), which is

gst-launch-1.0 -e rtspsrc location='rtsp://admin:12345@192.168.1.12:554/mediainput/h264/stream_1' short-header=true ! rtph264depay ! h264parse ! omxh264dec ! videorate ! "video/x-raw,framerate=30/1" ! omxvp8enc ! splitmuxsink      location=test%d.webm max-size-time=100000000000 muxer=qtmux

But I also see a small increase in resident memory even on the smaller pipeline mentioned in #9.

Thank you,

Suraj

jachen · August 24, 2016, 8:04am

Hello, Suraj:
Would you please paste the detailed memory usage between GST pipeline beginning and after 1-day running?
I checked the GST command you provided @ #13, and found no obvious increasing with resident memory.

The SDK I’m using is different from yours. Some issues have been fixed in this version.
So please test it in the coming new release.

br
ChenJian

surajtn30 · August 24, 2016, 7:05pm

ChenJian,
Here is a snapshot of the processes memory usage.

Thank you,
Suraj

jachen · September 5, 2016, 2:13am

Hello, Suraj:
I made another 2-day testing, and the RSS for gst-launch keeps @ 20532 during days.
Please wait for the next release for your side test.

br
ChenJian

marekw63 · January 9, 2017, 8:58pm

I just got the identical error on a gstream stream after running overnight encoding 4 test streams. Any ideas on why. It occurs after hours of running Gstreamer 1.0 . I am on 23.2 with Gstreamer 1.8.3

DaneLLL · January 10, 2017, 2:35am

Hi marekw63,
Can you move to r24.2.1 and give it a try?

marekw63 · January 10, 2017, 6:23pm

Ok. will try that in a week or two. its a bit of time as I will have to rebuild Gstreamer 1.8.3 and all the various dependencies for the plugins that I use that are not part of the standard L4T which will need to be found again. No easy fix for 23.2 likely?

DaneLLL · January 11, 2017, 1:57am

Hi marekw63,
What gst command do you run on r23.2 to reproduce the memory leakage?

BTW, on r24.2.1, it is Gstreamer 1.8.1 and should be less effort to migrate to 1.8,3.

Topic		Replies	Views
Nvidia-desktop kernel: [407343.357549] (NULL device *): nvhost_channelctl: invalid cmd 0x80685600 Jetson AGX Xavier camera , opencv , gstreamer	32	3797	February 17, 2023
Nvivafilter multiple pipelines memory leak Jetson TX1 gstreamer	22	812	April 24, 2024
Possible memory leak in nvvidconv gstreamer plugin or decoder? Jetson Nano gstreamer	4	1856	October 18, 2021
Gstreamer nv3dsink memory gstcaps leaks Jetson Orin NX gstreamer , nvbugs	18	684	March 27, 2024
Tx2-4g R32.3.1 gstreamer "pool" thread using excessive cpu usage Jetson TX2 gstreamer	11	929	October 18, 2021
Memory leak bug Jetson Xavier NX gstreamer	3	1175	April 26, 2022
Gstreamer / DeepStream nvcamerasrc has high CPU usage (around 60 %) Jetson TX1	6	591	October 18, 2021
Memory leaks detected in GStreamer elements (omxh264 or nvvidconv) Jetson TX2	14	5301	October 18, 2021
Using nvv4l2h264enc encoder memory increases when releasing pipeline and setting up a new one Jetson AGX Xavier	23	3921	October 18, 2021
Memory error OpenCV, Gstreamer, and Tensorflow memory error? Jetson TX2	15	2896	October 18, 2021

GPU: Not enough priv cmd buffer space

Related topics