We are using VisionWorks for a project, and are using Jetson TX2’s. We are having a problem with the load time for image frames from a video file or from a camera.
We are using video with a resolution of 2704x1520. When we test this with VisionWorks using the nvx_sample_player, it is taking 145ms/frame to load and display this video. This is true with .mp4 videos and .avi. We can test the same nvx_sample_player with the cars.mp4 that is included with the VisionWorks samples, and it runs at 32ms/frame or 30 Hz, as you would expect. Our 2704x1520 video is encoded at 24Hz, so there should be nothing in the video encoding that is causing the slow playback time.
Initially, I thought the display time was due to the rendering and display of the video. However, if I comment out all rendering code, so it is only loading frames from the video file and doing nothing with them, the time per frame remains at 145ms.
While you would expect it to take longer to load a larger video frame than a smaller one, I would still expect it to be somewhere in the range of 5-10ms, not 145ms! I can also understand that if the video is encoded at 24Hz, then the video playback may be restricted to only release each new frame after 41.6ms. Again, I can’t see any reason for it to take 145ms.
We have been using the Jetson_clocks.sh script to raise the processing speed on the Jetson. We also tried the nvpmodel -m 0 command, and this speeded it up further. With this command, we now load video frames at 90ms each. This is a substantial speedup, but it should be able to load much faster. We are using a TX2 with Jetpack 3.1.
Can anyone provide any help so we can read in our videos at 24Hz?
Thanks,
Rick
Hi,
Thanks for your feedback.
We are checking this issue and will update information with you later.
Please remember that to execute jetson_clocks.sh after running nvpmodel.
nvpmodel changes the clock range to best performance and jetson_clock fix the clock to the maximal.
sudo nvpmodel -m 0
sudo ./jetson_clocks.sh
Thanks.
Hi,
Car.mp4 is a 720p video rather than 1080p.
We have also tested 1080p and 4K video on our side.
Here is our profiling data for your reference: (under expectation)
- Source size; 3840x2160 (735MB) → Display Time: 170 ~ 171 ms
- Source size: 1920x1080 (435MB) → Display Time: 45 ~ 46 ms
Please remember to set the acceleration command shared in #2 in order.
Thanks.
For a high performance video processing framework, isn’t this rather slow? CPU-only OpenCV is easily able to open these frames at 24Hz on the same machine, and I assume it could actual open them faster, but the frame rate is fixed at 24Hz. I imagine the actual time to decode the video codec and to allocate and set the image data in memory is around 10ms. Why is this high performance video processing framework taking such an exceptionally long time? It seems like a fix should be possible for this.
AastaLLL, I appreciate your checking into this. I didn’t know about the order of the commands for nvpmodel and Jetson_clocks, so that’s definitely useful. Your test results with the 1080p and the 4K show timings consistent with ours. Our 2704x1520 is twice the size of a 1080p picture, and it is taking twice as long once the processor performance is maxxed out.
The thing is, this loading time seems to be fixed regardless of whether the image is rendered, or if other image processing tasks are performed. That makes me think it is actually being loaded faster, and something is making it wait. I don’t think the processors and memory are actually working at max rate for 45ms on the 1080p image or 170ms on the 4K image. I suspect some other factor in the VisionWorks framework is enforcing this unintentional delay.
Hi,
We are checking this issue internally.
Will update information with you later.
Thanks.
Hi,
This issue is feedbacked and will be prioritized internally.
Currently, please check our MMAPI for improvement.
MMAPI can be installed via JetPack directly.
Thanks.