Jetson TX2 camera capture delay

jpchae · October 14, 2020, 9:09am

Hello,

I tested to measure by capture delay from custom image sensor to display monitor as shown below picture.

test condition is 24Mp ( 5706x4312) / 30fps.
it took 0.56 seconds.

i want to reduce capture delay to 0.09s.
Please, How to minimize delay time.

JerryChang · October 15, 2020, 2:37am

hello jpchae,

here’s a roughly capture pipeline you may refer to.

[A] issue a frame capture—> [B] rolling shutter—> [C] frame buffering—> [D] rendering display

could you please share the commands you’re using for your 560ms evaluation results.
you may note that [B] depends-on your sensor frame-rate, for example, it’ll take 33ms for your 30-fps sensor modes.
may I know you’re asking to reduce the capture delay to 90ms from [A] to [D].

jpchae · October 15, 2020, 7:13am

Hello JerryChang ,

could you please share the commands you’re using for your 560ms evaluation results.

As you shown above picture, image sensor captured smartphone stop watch app.
at this time , it captured both smartphone display and display monitor by other smartphone camera.
560ms is difference stopwatch time of monitor and stopwatch time of smartphone.
you may note that [B] depends-on your sensor frame-rate, for example, it’ll take 33ms for your 30-fps sensor modes.
I understand
may I know you’re asking to reduce the capture delay to 90ms from [A] to [D].

Yes, I want to reduce until 100ms~90ms from [A] to [D] minimally.

JerryChang · October 15, 2020, 8:24am

hello jpchae,

may I know what’s the commands you’re used to enable camera stream.
thanks

jpchae · October 15, 2020, 8:29am

Hello JerryChang,

here is test commands.

gst-launch-1.0 nvarguscamerasrc sensor-id=0 wbmode=1 aelock=1 exposurecompensation=1 ee-mode=1 ee-strength=0 tnr-strength=0 tnr-mode=1 ! ‘video/x-raw(memory:NVMM), width=5760, height=4312, format=(string)NV12, framerate=30/1’ ! xvimagesink -e

thanks,

JerryChang · October 15, 2020, 8:37am

hello jpchae,

let’s narrow down the issue,
could you please disable camera preview to check the sensor frame-rate through nvarguscamerasrc.
for example,

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=1920, height=1080, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM),format=I420' ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v

jpchae · October 15, 2020, 8:50am

hello JerryChang,

Test result as shown below.

livsmed@livsmed-desktop:~/Documents$ gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! ‘video/x-raw(memory:NVMM),width=5760, height=4312, framerate=30/1, format=NV12’ ! nvvidconv ! ‘video/x-raw(memory:NVMM),format=I420’ ! fpsdisplaysink text-overlay=0 video-sink=fakesink sync=0 -v
Setting pipeline to PAUSED …
Pipeline is live and does not need PREROLL …
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
Setting pipeline to PLAYING …
New clock: GstSystemClock
/GstPipeline:pipeline0/GstNvArgusCameraSrc:nvarguscamerasrc0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:src: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink.GstProxyPad:proxypad0: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0.GstGhostPad:sink: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/GstCapsFilter:capsfilter1.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, framerate=(fraction)30/1, format=(string)I420
/GstPipeline:pipeline0/Gstnvvconv:nvvconv0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, format=(string)NV12, framerate=(fraction)30/1
/GstPipeline:pipeline0/GstCapsFilter:capsfilter0.GstPad:sink: caps = video/x-raw(memory:NVMM), width=(int)5760, height=(int)4312, format=(string)NV12, framerate=(fraction)30/1
GST_ARGUS: Creating output stream
CONSUMER: Waiting until producer is connected…
GST_ARGUS: Available Sensor modes :
GST_ARGUS: 5760 x 4312 FR = 29.999999 fps Duration = 33333334 ; Analog Gain range min 0.000000, max 4.000000; Exposure Range min 34000, max 550385000;

GST_ARGUS: 2880 x 2156 FR = 100.000000 fps Duration = 10000000 ; Analog Gain range min 0.000000, max 4.000000; Exposure Range min 22000, max 358733000;

GST_ARGUS: Running with following settings:
Camera index = 0
Camera mode = 0
Output Stream W = 5760 H = 4312
seconds to Run = 0
Frame Rate = 29.999999
GST_ARGUS: PowerService: requested_clock_Hz=325987200
GST_ARGUS: Setup Complete, Starting captures for 0 seconds
GST_ARGUS: Starting repeat capture requests.
CONSUMER: Producer has connected; continuing.
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0/GstFakeSink:fakesink0: sync = false
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 16, dropped: 0, current: 29.88, average: 29.88
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 29, dropped: 0, current: 25.63, average: 27.81
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 43, dropped: 0, current: 26.09, average: 27.23
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 56, dropped: 0, current: 25.79, average: 26.88
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 69, dropped: 0, current: 25.69, average: 26.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 83, dropped: 0, current: 26.17, average: 26.56
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 97, dropped: 0, current: 27.79, average: 26.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 112, dropped: 0, current: 28.51, average: 26.96
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 126, dropped: 0, current: 27.15, average: 26.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 140, dropped: 0, current: 26.11, average: 26.89
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 153, dropped: 0, current: 25.91, average: 26.80
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 166, dropped: 0, current: 25.99, average: 26.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 179, dropped: 0, current: 25.04, average: 26.61
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 193, dropped: 0, current: 27.24, average: 26.65
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 208, dropped: 0, current: 28.83, average: 26.80
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 222, dropped: 0, current: 27.75, average: 26.86
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 237, dropped: 0, current: 27.99, average: 26.93
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 251, dropped: 0, current: 27.57, average: 26.96
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 265, dropped: 0, current: 26.75, average: 26.95
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 280, dropped: 0, current: 28.66, average: 27.04
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 295, dropped: 0, current: 28.02, average: 27.08
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 309, dropped: 0, current: 27.37, average: 27.10
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 323, dropped: 0, current: 26.79, average: 27.08
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 337, dropped: 0, current: 26.15, average: 27.04
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 350, dropped: 0, current: 25.39, average: 26.98
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 363, dropped: 0, current: 25.67, average: 26.93
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 377, dropped: 0, current: 26.27, average: 26.90
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 390, dropped: 0, current: 25.85, average: 26.87
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 403, dropped: 0, current: 25.97, average: 26.84
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 416, dropped: 0, current: 25.14, average: 26.78
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 429, dropped: 0, current: 25.46, average: 26.74
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 442, dropped: 0, current: 25.66, average: 26.71
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 455, dropped: 0, current: 25.98, average: 26.68
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 468, dropped: 0, current: 25.71, average: 26.66
/GstPipeline:pipeline0/GstFPSDisplaySink:fpsdisplaysink0: last-message = rendered: 481, dropped: 0, current: 25.25, average: 26.62
^Chandling interrupt.
Interrupt: Stopping pipeline …
Execution ended after 0:00:19.620918909
Setting pipeline to PAUSED …
Setting pipeline to READY …
GST_ARGUS: Cleaning up
CONSUMER: Done Success
GST_ARGUS: Done Success
Setting pipeline to NULL …
Freeing pipeline …
GST_ARGUS:
PowerServiceHwVic::cleanupResources

thanks,

JerryChang · October 15, 2020, 8:57am

hello jpchae,

it looks camera stack did not handling this to reach actual sensor frame-rate.
could you please check Supported Modes and Power Efficiency to configure system to maximum performance for testing.

BTW,
would you like to enable camera stream with such high resolution 5760x4312 and display it?
could you please have a try to scale the output frame as 1080p for verification?
thanks

jpchae · October 15, 2020, 9:12am

Hello JerryChang,

Is is right to test command?

gst-launch-1.0 nvarguscamerasrc sensor-id=0 wbmode=1 aelock=1 exposurecompensation=1 ee-mode=1 ee-strength=0 tnr-strength=0 tnr-mode=1 ! ‘video/x-raw(memory:NVMM), width=1920, height=1080, format=(string)NV12, framerate=100/1’ ! nvvidconv flip-method=0 ! nvoverlaysink -e

thanks

JerryChang · October 16, 2020, 12:34am

hello jpchae,

your commands would enable 1920x1080 camera sensor mode, VI would involved for downscaling if there’s no suitable sensor mode;
you may enable your camera sensor at 5760x4312, and downscale the preview frame to 1920x1080 for rendering to display.
for example,

gst-launch-1.0 nvarguscamerasrc sensor-id=0 ! 'video/x-raw(memory:NVMM),width=5760, height=4312, framerate=30/1, format=NV12' ! nvvidconv ! 'video/x-raw(memory:NVMM), format=(string)NV12, width=1920, height=1080' ! nvoverlaysink -ev

jpchae · October 16, 2020, 7:19am

Hello JerryChang,

As your recommand, i tested downscaling preview frame.

5760x4312 -> 1920 x1080 preview : it tooks 190~200ms .
5760x4312 -> 3480x2160 preview : it tooks 230~250ms .

Are there any other tests to review to reduce time?

Thanks,

JerryChang · October 16, 2020, 8:40am

please refer to post #9 to configure system to maximum performance for testing.

nadeemm · October 18, 2021, 6:16pm