Lowest-possible latency MIPI CSI camera to hardware H.264 encode?

[user108834]

Considering how to build an ultra low latency H.264 encoder for a realtime camera. The Jetson Nano platform looks promising. Would be sending frames over Ethernet to a local server.

1. is there actual hardware encoding for H.264 on the Nano?

2. If we used GStreamer and RTP, what would be the lowest latency achievable for HD (1440x1080 to 1920x1080 range)? (and what would be an optimal pipeline to use?)

3. If a custom solution could have lower latency, how do we get access in C language to the camera, send data to the hardware H.264 encoder, and get the output there, from which we could do a custom packet send? (Looking for any links to documentation for this one)

From experience with the Cisco H.264 encoder on a PC, it appears to take about 30 ms to encode H.264 in software, invoked from a single thread. Hoping for that fast in the Nano (plus whatever extra we have to do for packet send).

FYI this application needs very low latency to the server, but the delivered frame rate is 10-11 fps, so should be easy for the hardware. Compressed stream around 1-5 Mbit/s.

Any tips you can provide on where to begin looking, what manuals/URLs, etc. would be much appreciated!

[DaneLLL]

Hi,
We would suggest use jetson_multimedia_api. There is option for printing encoding latency and average frame rate. Please try the commands:

01_video_encode$ gst-launch-1.0 videotestsrc num-buffers=300 ! video/x-raw,width=1920,height=1080 ! filesink location=1080.yuv
01_video_encode$ ./video_encode 1080.yuv 1920 1080 H264 a.264 --stats --max-perf

[user108834]

Thank you @DaneLLL . I just received the Nano Developer Kit last night and was finally able to try your suggestion. The output we’re seeing is as follows:

Creating Encoder in blocking mode 
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
875967048
842091865
H264: Profile = 66, Level = 51 
NVMEDIA_ENC: bBlitMode is set to TRUE 
Could not read complete frame from input file
File read complete.
Got 0 size buffer in capture 
----------- Element = enc0 -----------
Total Profiling time = 10.2596
Average FPS = 29.241
Total units processed = 301
Average latency(usec) = 6860
Minimum latency(usec) = 5350
Maximum latency(usec) = 92925
-------------------------------------
App run was successful

So the average latency is 6.8 milliseconds on a 10 second test at just under 30fps. 29.24fps is a tiny bit slower than 29.97 NTSC (would have to look further before assuming it would work for full speed realtime video) but this speed does look fine for our 11fps use case, for the encoding part of the chain.

More to research re our options further for RTP/RTSP, formats, etc. to get the data off the box and what comm latency occurs, plus the camera front end, but as a first test this is very promising.

Thank you very much for your suggestion!