Has anyone else experienced gstreamer pipelines becoming slower over time? I have a 1080p30 encode I am encoding separately 3 times. It keeps a consistent 30fps encode for each stream for the first few hours.
Then over time (dependent on the number of streams) it drops down to 20-22fps and then stays there. With the 3 stream example I gave it usually starts dropping after 30minutes.
A simple stop and start of the pipeline results in the issue being alleviated but I don’t know why there would be a bottleneck that only arises after such a long period of time.
Any insight would be greatly valuable.
I know nothing of that particular issue, but one thing you might do is to run some form of monitor software for CPU and RAM, and watch if things change over time (not just for that program, but for the system as a whole). For example, htop or xosview (or both).
You might also want to force performance maximum, see [url]http://elinux.org/Jetson/TX1_Controlling_Performance[/url].
Thanks linuxdev for reaching out. I’ve installed htop and will run some tests with it.
I don’t run an xserver on my system so xosview is no go.
Previously i have done the Max Perf scripts nvidia provided and it didn’t improve anything.
I did run regular Top for a cycle. See the attached image where you see the CPU usage goes down by 15% when the slowdown arises. So when running at full 30fps it stays around 60% consistently. When the slowdown happens it stays around 45%.
I don’t know what would be the cause, but I can only speculate that either it was being starved from CPU time by another process, or by a lack of data throughput. I would look closely to see what other processes may have gone up in CPU use as yours drops in CPU use…it wouldn’t necessarily have to be something taking a lot of CPU, but any one or more processes which might push your process away would be suspicious.
If this latter is not the case, then you might be interested memory use by other processes. It could be a case of data transfer slowing if there is a lack of ram suitable for your process. There are different types of memory (and different requirements due to those types), so it wouldn’t necessarily be a lot of use in general (meaning the extra use may not seem like a large increase, but the nature of the ram used by that small case might have a bigger impact than it would appear at first).
If it does look like perhaps there is a memory issue, you might try adding an SD card and formatting it as swap. Enable that as swap, and do the same htop monitoring again; swap should start out unused, and if there is no swap being used at the time when the problem occurs, then it isn’t likely a ram usage issue. If it does start swapping out it may or may not run faster, but you would notice swap usage building up.
I’ll look into it.
Give the CPU is only running at 60% max at any given time i’m inclined to think its not starving anything. I’ll look into if its a memory bandwidth issue.
I did just run a test where I did 3x H.264 (previous test was 2x H.264 + 1x H.265) and that 3x H.264 doesn’t seem to exhibit the problem. So maybe it is something with the NVENC encoding core itself?
Again a restart to the gstreamer pipeline fixes it for another 10 min or so. Is there anyway to reset the encoder?
I’ll check if the queues prior to the encodes are filling up or something. Maybe the 265 core isn’t processing stuff fast enough and the queue is dropping. It is just odd that it would take 10min or so for that to happen…
Ok looks like when i put queues everywhere the only queue which starts holding data is the queue prior to the omxh265enc. Seems like over time it just gets larger and larger very slowly.
So looks like it is the H265encoder (other encodes don’t fill up the queue at all) which is bottle necking.
The queue alleviates this but obviously adds latency. Anyone know of a way to prevent this?
I would prefer to flush the queue and reset the encoder when the queue gets to a certain size. That way it drops like 5 frames and then repicks up. Any ideas?
Hi x1tester62,
We will try to do long run test. Could you please share your pipeline? Does it happen for 1xH265? Or it must be 2xH264+1xH265?
Hi x1tester62,
We don’t observe the issue in running 2xH264 + 1xH265 for 10-hour test (1080p30, num-buffers=1080000):
gst-launch-1.0 videotestsrc num-buffers=1080000 do-timestamp=true is-live=true ! ‘video/x-raw,width=(int)1920,height=(int)1080,framerate=(fraction)30/1,format=(string)I420’ ! omxh264enc ! fakesink & gst-launch-1.0 videotestsrc num-buffers=1080000 do-timestamp=true is-live=true ! ‘video/x-raw,width=(int)1920,height=(int)1080,framerate=(fraction)30/1,format=(string)I420’ ! omxh264enc ! fakesink & gst-launch-1.0 nvcamerasrc num-buffers=1080000 ! ‘video/x-raw(memory:NVMM),width=(int)1920,height=(int)1080,framerate=(fraction)30/1,format=(string)I420’ ! omxh265enc ! fakesink
Could you share your pipeline that hits the issue?
Hi Dane,
When we tried with simultaneous recording of four cameras (one with v4l2src and three with nvcamsrc), we are facing frame drop from 500ms - 1s.
- Is there any way to boost encoder performance?
- Can we monitor encoding process to see if it misses any frame ?
Thanks
Pal
Gstreamer command used:
gst-launch-1.0 nvcamerasrc num-buffers=1400 sensor-id=0 ! ‘video/x-raw(memory:NVMM),width=1280, height=720, framerate=30/1, format=NV12’ ! omxh264enc ! qtmux ! filesink location=cam0.mp4 -ev nvcamerasrc num-buffers=1400 sensor-id=1 ! ‘video/x-raw(memory:NVMM),width=1280, height=720, framerate=30/1, format=NV12’ ! omxh264enc ! qtmux ! filesink location=cam1.mp4 -ev nvcamerasrc num-buffers=1400 sensor-id=2 ! ‘video/x-raw(memory:NVMM),width=1280, height=720, framerate=30/1, format=NV12’ ! omxh264enc ! qtmux ! filesink location=cam2.mp4 -ev v4l2src device=/dev/video0 num-buffers=1400 do-timestamp=true ! ‘video/x-raw , width=(int)1920 , height=(int)1080 , format=(string)UYVY’ ! nvvidconv ! ‘video/x-raw(memory:NVMM) , width=(int)1920 , height=(int)1080 , format=(string)NV12’ ! omxh264enc ! qtmux ! filesink location=cam3.mp4 -ev
Hi Pal1234,
Please try
1 Execute /home/ubuntu/jetson_clocks.sh
2 Use fakesink instead of filesink
TX1 should be able to run three 720p30 and one 1080p30 encoding. Let’s check the result at max performance without FIO.