Hello everyone,
I’m interested in achieving 1000FPS with 640x480 (or closely) resolution, monochrome. However, most cameras I’ve found are running on USB3, so they’re limited to protocol restrictions. I guess MIPI/GigE can achive that speed, but most of available specs of those cameras are limited to 100FPS at ~4K resolution. Maybe they can be speed-up while lowering resoltuin, but finding it out is a bit complicated.
So, any ideas would appreciated on how to found these kind of cameras
Thank you!
Hey @xensey
Here, are some USB 3.0 offers, but they are limited to half of the framerate that you want. Someone, seems to have achieved that on MIPI, but limited to 640x80. There are some 1, 2, slomo-targeted sensors use the GigE interface, so maybe those can work for you. However those high speed slomo-targeted sensors, usually only have a certain record window, since they usually use a limited amount of memory or some high speed storage to store the frames and then after the recording frame is done, they dump it over the interface.
Regards,
Andres
Embedded SW Engineer at RidgeRun
Contact us: support@ridgerun.com
Developers wiki: https://developer.ridgerun.com/
Website: www.ridgerun.com
1 Like
Thank you for reply!
Cr450 looks promising. Chronos is too expensive, have limited budget for $1k for video system.
One of the vendors also proposed me to use 2 cameras with shifted synch, think that might be solution too (if jetson will be able to catch all those)
1 Like
In theory at least on the driver level you could control that, have a look here, in order to have two cameras with a shifted clk and simulate one faster camera.
Regards,
Andres
Embedded SW Engineer at RidgeRun
1 Like
Further than driver level, you may better tell what processing you intend to do with Xavier NX.
Be aware that 1000 fps is quite challenging, Driver and/or Argus may take some CPU load… Check in another terminal with sudo tegrastats.
Some quick trial with gstreamer framework (NX devkit with SD Card running R35.2.1):
gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=640,height=480,framerate=1000/1 ! fpsdisplaysink video-sink=fakesink text-overlay=0 -v
...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 334, dropped: 0, current: 666,99, average: 666,99
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 673, dropped: 0, current: 677,29, average: 672,13
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1012, dropped: 0, current: 676,70, average: 673,66
Using a lower resolution for CPU based plugin videotestsrc improves:
gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=320,height=240,framerate=1000/1 ! fpsdisplaysink video-sink=fakesink text-overlay=0 -v
...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 501, dropped: 0, current: 1000,85, average: 1000,85
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1002, dropped: 0, current: 1000,02, average: 1000,44
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1502, dropped: 0, current: 999,99, average: 1000,29
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2003, dropped: 0, current: 999,99, average: 1000,21
That was just showing some CPU limitation at that high framerate.
Now about converting/scaling/cropping, that might be done with Jetson VIC, but it may not support that rate:
gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=320,height=240,framerate=1000/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=640,height=480' ! fpsdisplaysink video-sink=fakesink text-overlay=0 -v
...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 274, dropped: 0, current: 546,68, average: 546,68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 512, dropped: 0, current: 475,50, average: 511,11
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 753, dropped: 0, current: 480,85, average: 501,02
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 994, dropped: 0, current: 480,72, average: 495,94
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1235, dropped: 0, current: 481,21, average: 493,00
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1477, dropped: 0, current: 482,39, average: 491,23
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1716, dropped: 0, current: 476,51, average: 489,13
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1956, dropped: 0, current: 478,22, average: 487,76
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2197, dropped: 0, current: 481,54, average: 487,07
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 2436, dropped: 0, current: 476,33, average: 486,00
Boosting VIC clock may improve a bit:
sudo su
cd /sys/kernel/debug/bpmp/debug/clk/vic/
echo 1 > mrq_rate_locked
cat max_rate > rate
exit
gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=320,height=240,framerate=1000/1 ! nvvidconv ! 'video/x-raw(memory:NVMM),width=640,height=480' ! fpsdisplaysink video-sink=fakesink text-overlay=0 -v
...
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 369, dropped: 0, current: 737,50, average: 737,50
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 740, dropped: 0, current: 741,51, average: 739,50
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1114, dropped: 0, current: 747,16, average: 742,06
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1476, dropped: 0, current: 723,36, average: 737,38
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1845, dropped: 0, current: 737,17, average: 737,34
Not enough… You may also use GPU as backend for nvvidconv at that rate:
gst-launch-1.0 videotestsrc ! video/x-raw,format=NV12,width=320,height=240,framerate=1000/1 ! nvvidconv compute-hw=GPU ! 'video/x-raw(memory:NVMM),width=640,height=480' ! fpsdisplaysink video-sink=fakesink text-overlay=0 -v
....
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = true
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 500, dropped: 0, current: 999,91, average: 999,91
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1000, dropped: 0, current: 999,94, average: 999,92
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 1501, dropped: 0, current: 1000,01, average: 999,95
Thanks for reply!
Sure, 1000FPS is kind of challenging. We run UNET-like network and estimated that we need for current functionality 1-5 TOPS. So I’m also afraid of issues with bottleneck somewhere like transmitting signal from board to jetson.
For now I don’t have actually yet any Jetson - was waiting for Orin Nano, but as it still it’s not available yet decided to look over Xavier NX - as hopefully . Anyway, project is in RnD an’d prototyping stage, so testing different hardware.
Another idea of achieving that FPS is defining region of interest for sensor, since we really need to changes only in 20% of frame. ROI is moving within frame over time, but not that slowly, like checking for changes at 5Hz rate should work. For e.g., Basler (I guess as many others) has an option to feed in ROI,. Not sure will changing cause some lags or not. And to feed in ROI can either take periodically full frame shot and run fast detection, or use another low-speed camera.
I am curious, if you get 1000 f/s, does any computation need to keep up with this, e.g., calculations complete on one frame by the time the next frame arrives? Or even does some processing need to occur, such as decompression, and copy of data to somewhere? Should you achieve 1000 f/s capture rate, it is highly unlikely the actual processing can work with that rate, and that buffering would be a big part of the data pipeline.
Yes, we are trying to compute in one by one frame, but surely expect some delay in output, like 50-100ms.
If we consider the timing of the result of the computation, can I ask if:
- First frame in results in first frame out being mandatory (probably easy to achieve).
- The variation in timing of data arriving is constant, versus can have a certain degree of standard deviation? If some variance in timing is ok, so long as frames are not reordered, is it ok to have a somewhat high variance so long as averages are good? Or is quality of variance in timing important?
If you are ok with averages being good despite timing not being constant, then you are probably ok. However, the closer you approach the scheduling limits (typically 1000 Hz for software IRQs, but some scheduler algorithm for hardware IRQs, which approaches an IRQ starvation if the hardware or scheduler can’t handle it), would be the camera’s limit since it uses a hardware IRQ.
FYI, if you are expecting this to behave like hard realtime (e.g., something in a fighter jet’s autopilot or HUD display), versus something which might be a software realtime (e.g., a self-navigating drone moving at the speed of a walking person or a medium speed automobile), then you are in real trouble as load goes up. Even soft realtime is a pretty big problem when you get close to either hardware IRQ or software IRQ rates. And all of that is within the Linux kernel before you ever get to user space.
The variation of timing can be different, but it’s good to know what it was. 1000FPS is like state-of-art, and competitors use FPGA with some more simple approaches (insted more sophisticated board design with low-level access to sensor) to solve it or ASIC chip (again with custom board and direct sensor access). Of course this is the better way, especially for mass-production/size/power consumption, but that requires much much more budget than we have, so using jetson is looking fine.
This is kind of stretching things, but you could create a custom add-on board with an ARM Cortex-R52 or similar. This could in theory be less than an FPGA cost by quite a bit if an expensive FPGA is used (cost is one reason I have no experience actually using FPGAs other than looking on paper). The add-on board could receive and send messages with something like FreeRTOS. But yes, if this could be achieved with just the Jetson, then it’d be rather nice in a lot of ways. I just don’t know if a Jetson can keep up that performance a sufficient percentage of the time.
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.