video playing not smooth caused by calling gst_nvvconv_transform() sometimes very time consumed

I have a pipeline decoding 8 IP cameras’ h264 video stream of 1920x1080@25fps simultaneously, and get RGBA video frames via nvvideosink’s EGL stream. Each camera pipeline like this: rtspsrc ! decodebin ! nvvidconv ! nvvideosink.

After profiling with nvvp tool, I find that gst_nvvconv_transform sometimes very time consumed. Normally, it takes within 1ms, but sometimes it takes tens of milliseconds.

By examining the tegrastat’s report, the usages of CPU/GPU/EMC/NVDEC all are not overload.

RAM 4498/7851MB (lfb 5x4MB) cpu [42%@498,off,off,32%@499,40%@499,34%@499] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 13%@522
RAM 4498/7851MB (lfb 5x4MB) cpu [45%@345,off,off,32%@345,34%@348,35%@348] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 31%@420
RAM 4500/7851MB (lfb 5x4MB) cpu [40%@805,off,off,39%@806,48%@807,35%@808] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 13%@318
RAM 4500/7851MB (lfb 5x4MB) cpu [41%@652,off,off,35%@652,36%@653,39%@652] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 74%@522
RAM 4500/7851MB (lfb 5x4MB) cpu [41%@345,off,off,36%@345,35%@344,31%@345] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 46%@420
RAM 4500/7851MB (lfb 5x4MB) cpu [44%@345,off,off,31%@345,37%@345,36%@345] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 51%@420
RAM 4498/7851MB (lfb 5x4MB) cpu [54%@2031,off,off,33%@2035,35%@2030,34%@2034] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 86%@726
RAM 4499/7851MB (lfb 5x4MB) cpu [46%@807,off,off,35%@807,34%@806,32%@807] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 80%@420
RAM 4499/7851MB (lfb 5x4MB) cpu [42%@1420,off,off,35%@1421,31%@1419,36%@1420] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 83%@216
RAM 4499/7851MB (lfb 5x4MB) cpu [38%@806,off,off,40%@806,32%@806,34%@808] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 41%@522
RAM 4498/7851MB (lfb 5x4MB) cpu [43%@960,off,off,33%@960,37%@960,34%@964] EMC 49%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 3%@624
RAM 4498/7851MB (lfb 5x4MB) cpu [43%@653,off,off,32%@652,47%@652,35%@656] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 88%@318
RAM 4498/7851MB (lfb 5x4MB) cpu [40%@805,off,off,31%@806,35%@652,38%@653] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 65%@624
RAM 4499/7851MB (lfb 5x4MB) cpu [43%@499,off,off,33%@499,35%@499,37%@499] EMC 47%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 62%@420
RAM 4464/7851MB (lfb 5x4MB) cpu [47%@806,off,off,31%@806,37%@806,34%@806] EMC 49%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 10%@522
RAM 4500/7851MB (lfb 5x4MB) cpu [54%@653,off,off,36%@652,48%@652,38%@652] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 76%@420
RAM 4500/7851MB (lfb 5x4MB) cpu [45%@345,off,off,35%@345,34%@345,36%@345] EMC 49%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 59%@522

I try running jetson_clocks.sh to maximize the system clock, and the issue disappears. I was wondering is there a problem with the system scheduling. How can i fix this, and any suggestions would be appreciate, thanks.

The attachments are the nvvp profile report, and screenshot of the issue’s timeline.

NewSession1.xml (3.01 MB)

Hi SongjianSu,

Could you also provide the tegrastat’s report by running jetson_clocks.sh to maximize the system clock on your use case again as reference?
And, are you on r28.1?

Thanks

Hi kayccc,

Yes, r28.1, the TX2 was flashed via jetson 3.1.

Below is the tegrastat’s report on maximizing system clock.

RAM 4513/7851MB (lfb 5x4MB) cpu [45%@498,off,off,36%@502,38%@499,34%@502] EMC 53%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 34%@522
RAM 4478/7851MB (lfb 5x4MB) cpu [42%@653,off,off,32%@652,33%@652,32%@653] EMC 50%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 8%@522
RAM 4513/7851MB (lfb 5x4MB) cpu [40%@805,off,off,35%@810,54%@810,35%@810] EMC 50%@1866 APE 150 NVDEC 793 MSENC 217 GR3D 71%@624
RAM 4514/7851MB (lfb 5x4MB) cpu [45%@499,off,off,33%@498,32%@499,39%@498] EMC 49%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 54%@624
//<----- after start runnig jetson_clocks.sh
RAM 4515/7851MB (lfb 5x4MB) cpu [39%@2034,off,off,27%@2034,32%@2035,37%@2035] EMC 48%@1866 APE 150 NVDEC 576 MSENC 217 GR3D 56%@1134
RAM 4515/7851MB (lfb 5x4MB) cpu [21%@2027,off,off,8%@2027,15%@2024,15%@2028] EMC 55%@1600 APE 150 NVDEC 576 MSENC 217 GR3D 61%@1134
RAM 4515/7851MB (lfb 5x4MB) cpu [25%@2023,off,off,14%@2024,31%@2033,14%@2033] EMC 54%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 65%@1134
RAM 4514/7851MB (lfb 5x4MB) cpu [21%@2024,off,off,8%@2022,12%@2016,18%@2020] EMC 53%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 70%@1134
RAM 4514/7851MB (lfb 5x4MB) cpu [23%@2024,off,off,15%@2014,16%@2024,14%@2027] EMC 55%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 74%@1134
RAM 4514/7851MB (lfb 5x4MB) cpu [18%@2031,off,off,6%@2030,17%@2030,15%@2033] EMC 54%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 77%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [23%@2025,off,off,18%@2021,26%@2020,36%@2026] EMC 56%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 72%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [18%@2034,off,off,9%@2035,9%@2034,16%@2025] EMC 54%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 83%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [23%@2027,off,off,13%@2031,12%@2034,18%@2028] EMC 55%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 79%@1134
RAM 4514/7851MB (lfb 5x4MB) cpu [22%@2028,off,off,18%@2031,11%@2031,18%@2029] EMC 55%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 78%@1134
RAM 4514/7851MB (lfb 5x4MB) cpu [30%@2031,off,off,20%@2030,39%@2031,24%@2032] EMC 56%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 58%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [24%@2032,off,off,13%@2032,19%@2031,16%@2008] EMC 55%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 72%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [40%@2031,off,off,51%@2032,56%@2031,35%@2033] EMC 62%@1600 APE 150 NVDEC 576 MSENC 217 GR3D 90%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [47%@2033,off,off,34%@2033,40%@2033,32%@2034] EMC 65%@1600 APE 150 NVDEC 576 MSENC 217 GR3D 82%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [27%@2032,off,off,29%@2034,25%@2033,25%@2032] EMC 63%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 0%@1134
RAM 4514/7851MB (lfb 5x4MB) cpu [31%@2034,off,off,28%@2032,17%@2035,25%@2034] EMC 59%@1600 APE 150 NVDEC 576 MSENC 217 GR3D 0%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [19%@2030,off,off,8%@2034,8%@2034,14%@2034] EMC 56%@1600 APE 150 NVDEC 576 MSENC 217 GR3D 0%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [20%@2030,off,off,11%@2024,15%@2021,10%@2024] EMC 54%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 0%@1134
RAM 4486/7851MB (lfb 5x4MB) cpu [30%@2012,off,off,18%@2030,13%@2027,15%@2023] EMC 54%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 0%@1134
RAM 4513/7851MB (lfb 5x4MB) cpu [24%@2030,off,off,28%@2030,18%@2028,17%@2031] EMC 55%@1600 APE 150 NVDEC 1164 MSENC 217 GR3D 4%@1134

Hi SongjianSu,
It looks to be CPU scheduling issue. You may refer to jetson_clocks.sh and configure one or two CPUs always online.