Deepstream uses nvv4l2h264enc to parse GDR-encoded H264 naked stream How to set profile and level

Hardware Platform (Jetson / GPU) : Orin Nx

DeepStream Version : 6.2

JetPack Version (valid for Jetson only) : 5.1

TensorRT Version: 5.1

NVIDIA GPU Driver Version (valid for GPU only) : CUDA:11.4.315

The Gst framework did not throw an exception when parsing the H264 bare stream for the first time, but the data did not reach the server.

When the nvv4l2h264enc plug-in pushes the stream to rtmp for the first time, it finds that the Selected H264 profile (null) at level (null) is all null, and the data does not reach the server normally, and when it is set from the play state to the pause state, it will return to the play state. get data normally

  1. The record when the program runs to push stream
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:04.325464912 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:776:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> Setting format: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false
0:00:04.325520465 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1019:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Negotiating H264 profile and level.
0:00:04.325564114 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1070:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Selected H264 profile (null) at level (null)
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
0:00:04.328379208 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:834:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> output caps: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false

  1. Record when changing state from GST_STATE_PLAYING to GST_STATE_PAUSED when back to GST_STATE_PLAYING

[1442.129][user]-[Info]-[operator():363) Gst pause state:2 
[1442.129][user]-[Info]-[operator():367) start dji h264 
[1442.135][user]-[Info]-[operator():370) 获取流成功 
[1442.139][user]-[Info]-[operator():374) 请求关键帧成功 
0:23:52.360213862 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 0
0:23:52.360392873 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1416:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Starting encoding thread
0:23:52.360719024 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

H264: Profile = 77, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
0:23:52.375775123 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 1
0:23:52.375826036 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

  1. It can be pushed normally after the second part

Can you provide the complete pipeline?

  1. configuration file
# [group]
# enable=1 
# egl-renderer=false

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/yolov8s.onnx_b1_gpu0_fp32.engine
labelfile-path=/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/cal_trt.bin
force-implicit-batch-dim=1
tensor-meta-pool-size=1
batch-size=1
network-mode=1
num-detected-classes=4
interval=0
gie-unique-id=1
# output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
#scaling-filter=0
#scaling-compute-hw=0
cluster-mode=2
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=libnvdsinfer_custom_impl_Yolo.so
enable-perf-measurement=0
nvbuf-memory-type:0=0

[class-attrs-all]
pre-cluster-threshold=0.2
topk=20
nms-iou-threshold=0.5

[sink0]
enable=1
type=3
enc-type=0
async=1

  1. This code is modified based on DeepStream-test1, the data source is obtained from the memory, and passed to the Appsrc component.
  2. Steps for the problem to occur: After starting the app, enter 2 after a few seconds and switch to GST_STATE_PAUSED. After about ten seconds, a pipeline signal will appear. At this time, if you enter 1 to recreate gst, the problem of H264: Profile = null, Level = null will appear
  3. Pipeline order: appsrc → h264parse → nvv4l2decoder → nvstreammux → nvinfer → nvvideoconvert → nvdsosd → nvvideoconvert → nvv4l2h264enc → h264parse → flvmux → rtmpsink

How did you recreate? For this scenario, the pipeline should be set to NULL state but not PAUSE.

We don’t know about your app. Can you reproduce the problem with DeepStream sample apps?

After pushing the stream, set the pipeline to GST_STATE_PAUSED and wait for about 10s before setting the state to GST_STATE_PLAYING. At this time, the pipeline will receive a bus message callback

Snipaste-2023-06-09-16-27-06.png

Which message? Can you provide the error log?

  1. This is the log when running directly
root@nvidia-desktop:~/develop/PSDK3.5/build# ./dji_sdk_demo_linux_cxx 
[0.003][core]-[Info]-[DjiCore_Init:101) Payload SDK Version : V3.5.0-beta.0-build.1765 
[0.845][adapter]-[Info]-[DjiAccessAdapter_Init:186) Identify aircraft series is Matrice 30 Series 
[0.845][adapter]-[Info]-[DjiAccessAdapter_Init:206) Identify mount position type is Extension Port Type 
[0.848][adapter]-[Info]-[DjiAccessAdapter_Init:279) Now auto reconfigure baudrate from 115200 to 921600 
[0.973][adapter]-[Info]-[DjiAccessAdapter_Init:186) Identify aircraft series is Matrice 30 Series 
[0.973][adapter]-[Info]-[DjiAccessAdapter_Init:206) Identify mount position type is Extension Port Type 
[0.977][adapter]-[Info]-[DjiAccessAdapter_Init:301) Identity uart0 baudrate is 921600 bps 
[2.984][adapter]-[Info]-[DjiPayloadNegotiate_Init:197) Waiting payload negotiate finish. 
[6.026][adapter]-[Info]-[DjiPayloadNegotiate_Init:197) Waiting payload negotiate finish. 
[7.033][core]-[Info]-[DjiIdentityVerify_UpdatePolicy:470) Updating dji sdk policy file... 
[8.033][core]-[Info]-[DjiIdentityVerify_UpdatePolicy:473) Update dji sdk policy file successfully 
[8.044][core]-[Info]-[DjiCore_Init:169) Identify AircraftType = Matrice 30, MountPosition = Extension Port, SdkAdapterType = None 
[8.058][core]-[Info]-[DjiCore_ApplicationStart:239) Start dji sdk application 
[8.058][user]-[Info]-[DjiUser_ApplicationStart:261) Application start. 
Unknown or legacy key specified 'enable-perf-measurement' for group [property]
Unknown or legacy key specified 'nvbuf-memory-type:0' for group [property]
Added elements to bin
0:00:00.295944529  3313 0xffff789c0f30 DEBUG           v4l2videoenc gstv4l2videoenc.c:528:gst_v4l2_video_enc_open:<aaaaaaaaaaaaaa_enc> Opening
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
0:00:00.492494035  3313 0xffff789c0f30 DEBUG           v4l2videoenc gstv4l2videoenc.c:634:gst_v4l2_video_enc_start:<aaaaaaaaaaaaaa_enc> Starting
[12.108][user]-[Info]-[main:321) 获取流成功 
input 1 or 2:
[15.193][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :100 
0:00:06.291117794  3313 0xffff789c0f30 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/yolov8s.onnx_b1_gpu0_fp32.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT input           3x640x640       
1   OUTPUT kFLOAT output          8400x6          

0:00:06.477086776  3313 0xffff789c0f30 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/yolov8s.onnx_b1_gpu0_fp32.engine
0:00:06.508744033  3313 0xffff789c0f30 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/dstest_appsrc_config.txt sucessfully
[17.568][user]-[Info]-[init_ai:610) Running... 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:06.626566281  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:776:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> Setting format: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false
0:00:06.626640041  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1019:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Negotiating H264 profile and level.
0:00:06.626688809  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1070:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Selected H264 profile (null) at level (null)
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
0:00:06.631080838  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:834:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> output caps: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false
[17.722][user]-[Info]-[need_data_callback:178) gst write count :100 
[17.891][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :200 
[17.891][user]-[Info]-[need_data_callback:178) gst write count :200 
[20.591][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :300 
[20.591][user]-[Info]-[need_data_callback:178) gst write count :300 
[23.360][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :400 
[23.361][user]-[Info]-[need_data_callback:178) gst write count :400 
[26.062][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :500 
[26.062][user]-[Info]-[need_data_callback:178) gst write count :500 
  1. Modify the gst pipeline status to PAUSE
[27.321][user]-[Info]-[operator():373) 停止推流成功 
[27.321][user]-[Info]-[operator():379) Gst pause state:3 
  1. Then change the status of the pipeline to PLAY, at this time the H264: Profile = 66, Level = 0 attribute is available, and the server has also received the streaming data
1
[154.376][user]-[Info]-[operator():346) state:0 
[154.377][user]-[Info]-[operator():358) Gst pause state:2 
[154.377][user]-[Info]-[operator():362) start dji h264 
[154.383][user]-[Info]-[operator():365) 获取流成功 
0:02:25.280437731  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 0
0:02:25.280662505  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1416:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Starting encoding thread
0:02:25.280869134  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

H264: Profile = 66, Level = 0 
0:02:25.289653835  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 1
0:02:25.289701996  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

NVMEDIA: Need to set EMC bandwidth : 846000 
NVMEDIA_ENC: bBlitMode is set to TRUE 

What is the gstreamer pipeline state of this stage? The log only shows there is no downstream capability limitation in the negotiation stage. It will not impact anything.

We don’t know anytjing of your DjiCore_Init(), DjiCore_ApplicationStart(), … You’d better debug with your code first.

  1. Dji* related functions just get raw stream data from the device, and finally give me a Buf and len, and transfer the data to the pipeline through Appsrc
  2. The current pipeline state is GST_STATE_PLAYING

From the log, it is not in PLAYING state. Please debug with your code.

OK

  1. I would like to know when the element nvv4l2h264enc will be output for the following two sentences
H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
  1. Whether the element nvv4l2decoder supports GDR-encoded H264 naked stream data

This is hardware encoder log. It is printed when the output buffers are allocated. At this time, the encoder is prepared ready to encoding.

nvv4l2decoder supports any streams compatible to H.264 : Advanced video coding for generic audiovisual services (itu.int).

For the two elements of nvv4l2decoder and nvv4l2h264enc, is there any official documentation? I searched for a long time on Nvidia’s official website and couldn’t find detailed instructions.

What kind of document do you need?

There is document for the plugins Gst-nvvideo4linux2 — DeepStream 6.2 Release documentation

Ok, thanks, I have read this, is there any other related documents

What kind of document do you need?

Detailed instructions for this plugin in GStreamer

The v4l2 plugins usage is as normal as other GStreamer plugins. No special instructions.

They are even open source. You can find the source code here: nv-tegra.nvidia Code Review - tegra/gst-src/gst-nvvideo4linux2.git/summary

ok thanks