Deepstream uses nvv4l2h264enc to parse GDR-encoded H264 naked stream How to set profile and level

5kv7zfxvbc · June 9, 2023, 6:07am

Hardware Platform (Jetson / GPU) : Orin Nx

DeepStream Version : 6.2

JetPack Version (valid for Jetson only) : 5.1

TensorRT Version: 5.1

NVIDIA GPU Driver Version (valid for GPU only) : CUDA:11.4.315

The Gst framework did not throw an exception when parsing the H264 bare stream for the first time, but the data did not reach the server.

When the nvv4l2h264enc plug-in pushes the stream to rtmp for the first time, it finds that the Selected H264 profile (null) at level (null) is all null, and the data does not reach the server normally, and when it is set from the play state to the pause state, it will return to the play state. get data normally

The record when the program runs to push stream

NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:04.325464912 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:776:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> Setting format: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false
0:00:04.325520465 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1019:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Negotiating H264 profile and level.
0:00:04.325564114 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1070:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Selected H264 profile (null) at level (null)
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
0:00:04.328379208 11283 0xaaab072eeb60 DEBUG           v4l2videoenc gstv4l2videoenc.c:834:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> output caps: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false

Record when changing state from GST_STATE_PLAYING to GST_STATE_PAUSED when back to GST_STATE_PLAYING


[1442.129][user]-[Info]-[operator():363) Gst pause state:2 
[1442.129][user]-[Info]-[operator():367) start dji h264 
[1442.135][user]-[Info]-[operator():370) 获取流成功 
[1442.139][user]-[Info]-[operator():374) 请求关键帧成功 
0:23:52.360213862 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 0
0:23:52.360392873 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1416:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Starting encoding thread
0:23:52.360719024 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

H264: Profile = 77, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000 
0:23:52.375775123 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 1
0:23:52.375826036 11283 0xffff8004b520 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

It can be pushed normally after the second part

Fiona.Chen · June 9, 2023, 7:47am

Can you provide the complete pipeline?

5kv7zfxvbc · June 9, 2023, 8:01am

configuration file

# [group]
# enable=1 
# egl-renderer=false

[property]
gpu-id=0
net-scale-factor=0.0039215697906911373
model-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet10.caffemodel
proto-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/resnet10.prototxt
model-engine-file=/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/yolov8s.onnx_b1_gpu0_fp32.engine
labelfile-path=/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/labels.txt
int8-calib-file=/opt/nvidia/deepstream/deepstream/samples/models/Primary_Detector/cal_trt.bin
force-implicit-batch-dim=1
tensor-meta-pool-size=1
batch-size=1
network-mode=1
num-detected-classes=4
interval=0
gie-unique-id=1
# output-blob-names=conv2d_bbox;conv2d_cov/Sigmoid
#scaling-filter=0
#scaling-compute-hw=0
cluster-mode=2
parse-bbox-func-name=NvDsInferParseYolo
custom-lib-path=libnvdsinfer_custom_impl_Yolo.so
enable-perf-measurement=0
nvbuf-memory-type:0=0

[class-attrs-all]
pre-cluster-threshold=0.2
topk=20
nms-iou-threshold=0.5

[sink0]
enable=1
type=3
enc-type=0
async=1

This code is modified based on DeepStream-test1, the data source is obtained from the memory, and passed to the Appsrc component.
Steps for the problem to occur: After starting the app, enter 2 after a few seconds and switch to GST_STATE_PAUSED. After about ten seconds, a pipeline signal will appear. At this time, if you enter 1 to recreate gst, the problem of H264: Profile = null, Level = null will appear
Pipeline order: appsrc → h264parse → nvv4l2decoder → nvstreammux → nvinfer → nvvideoconvert → nvdsosd → nvvideoconvert → nvv4l2h264enc → h264parse → flvmux → rtmpsink

Fiona.Chen · June 9, 2023, 8:15am

How did you recreate? For this scenario, the pipeline should be set to NULL state but not PAUSE.

Fiona.Chen · June 9, 2023, 8:18am

We don’t know about your app. Can you reproduce the problem with DeepStream sample apps?

5kv7zfxvbc · June 9, 2023, 8:29am

After pushing the stream, set the pipeline to GST_STATE_PAUSED and wait for about 10s before setting the state to GST_STATE_PLAYING. At this time, the pipeline will receive a bus message callback

Fiona.Chen · June 9, 2023, 8:36am

Which message? Can you provide the error log?

5kv7zfxvbc · June 9, 2023, 8:44am

This is the log when running directly

root@nvidia-desktop:~/develop/PSDK3.5/build# ./dji_sdk_demo_linux_cxx 
[0.003][core]-[Info]-[DjiCore_Init:101) Payload SDK Version : V3.5.0-beta.0-build.1765 
[0.845][adapter]-[Info]-[DjiAccessAdapter_Init:186) Identify aircraft series is Matrice 30 Series 
[0.845][adapter]-[Info]-[DjiAccessAdapter_Init:206) Identify mount position type is Extension Port Type 
[0.848][adapter]-[Info]-[DjiAccessAdapter_Init:279) Now auto reconfigure baudrate from 115200 to 921600 
[0.973][adapter]-[Info]-[DjiAccessAdapter_Init:186) Identify aircraft series is Matrice 30 Series 
[0.973][adapter]-[Info]-[DjiAccessAdapter_Init:206) Identify mount position type is Extension Port Type 
[0.977][adapter]-[Info]-[DjiAccessAdapter_Init:301) Identity uart0 baudrate is 921600 bps 
[2.984][adapter]-[Info]-[DjiPayloadNegotiate_Init:197) Waiting payload negotiate finish. 
[6.026][adapter]-[Info]-[DjiPayloadNegotiate_Init:197) Waiting payload negotiate finish. 
[7.033][core]-[Info]-[DjiIdentityVerify_UpdatePolicy:470) Updating dji sdk policy file... 
[8.033][core]-[Info]-[DjiIdentityVerify_UpdatePolicy:473) Update dji sdk policy file successfully 
[8.044][core]-[Info]-[DjiCore_Init:169) Identify AircraftType = Matrice 30, MountPosition = Extension Port, SdkAdapterType = None 
[8.058][core]-[Info]-[DjiCore_ApplicationStart:239) Start dji sdk application 
[8.058][user]-[Info]-[DjiUser_ApplicationStart:261) Application start. 
Unknown or legacy key specified 'enable-perf-measurement' for group [property]
Unknown or legacy key specified 'nvbuf-memory-type:0' for group [property]
Added elements to bin
0:00:00.295944529  3313 0xffff789c0f30 DEBUG           v4l2videoenc gstv4l2videoenc.c:528:gst_v4l2_video_enc_open:<aaaaaaaaaaaaaa_enc> Opening
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
0:00:00.492494035  3313 0xffff789c0f30 DEBUG           v4l2videoenc gstv4l2videoenc.c:634:gst_v4l2_video_enc_start:<aaaaaaaaaaaaaa_enc> Starting
[12.108][user]-[Info]-[main:321) 获取流成功 
input 1 or 2:
[15.193][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :100 
0:00:06.291117794  3313 0xffff789c0f30 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/yolov8s.onnx_b1_gpu0_fp32.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT input           3x640x640       
1   OUTPUT kFLOAT output          8400x6          

0:00:06.477086776  3313 0xffff789c0f30 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<primary-nvinference-engine> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/yolov8s.onnx_b1_gpu0_fp32.engine
0:00:06.508744033  3313 0xffff789c0f30 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<primary-nvinference-engine> [UID 1]: Load new model:/root/develop/PSDK3.5/samples/sample_c++/platform/linux/manifold2/application/dstest_appsrc_config.txt sucessfully
[17.568][user]-[Info]-[init_ai:610) Running... 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
0:00:06.626566281  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:776:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> Setting format: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false
0:00:06.626640041  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1019:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Negotiating H264 profile and level.
0:00:06.626688809  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:1070:gst_v4l2_video_enc_negotiate:<aaaaaaaaaaaaaa_enc> Selected H264 profile (null) at level (null)
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
0:00:06.631080838  3313 0xaaaaf5b03b60 DEBUG           v4l2videoenc gstv4l2videoenc.c:834:gst_v4l2_video_enc_set_format:<aaaaaaaaaaaaaa_enc> output caps: video/x-raw(memory:NVMM), width=(int)1920, height=(int)1080, multiview-mode=(string)mono, multiview-flags=(GstVideoMultiviewFlagsSet)0:ffffffff:/right-view-first/left-flipped/left-flopped/right-flipped/right-flopped/half-aspect/mixed-mono, framerate=(fraction)30/1, batch-size=(int)1, num-surfaces-per-frame=(int)1, format=(string)NV12, block-linear=(boolean)false
[17.722][user]-[Info]-[need_data_callback:178) gst write count :100 
[17.891][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :200 
[17.891][user]-[Info]-[need_data_callback:178) gst write count :200 
[20.591][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :300 
[20.591][user]-[Info]-[need_data_callback:178) gst write count :300 
[23.360][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :400 
[23.361][user]-[Info]-[need_data_callback:178) gst write count :400 
[26.062][user]-[Info]-[my_DjiLiveview_H264Callback:117) dji write count :500 
[26.062][user]-[Info]-[need_data_callback:178) gst write count :500

Modify the gst pipeline status to PAUSE

[27.321][user]-[Info]-[operator():373) 停止推流成功 
[27.321][user]-[Info]-[operator():379) Gst pause state:3

Then change the status of the pipeline to PLAY, at this time the H264: Profile = 66, Level = 0 attribute is available, and the server has also received the streaming data

1
[154.376][user]-[Info]-[operator():346) state:0 
[154.377][user]-[Info]-[operator():358) Gst pause state:2 
[154.377][user]-[Info]-[operator():362) start dji h264 
[154.383][user]-[Info]-[operator():365) 获取流成功 
0:02:25.280437731  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 0
0:02:25.280662505  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1416:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Starting encoding thread
0:02:25.280869134  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

H264: Profile = 66, Level = 0 
0:02:25.289653835  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1362:gst_v4l2_video_enc_handle_frame:<aaaaaaaaaaaaaa_enc> Handling frame 1
0:02:25.289701996  3313 0xffff7804bd20 DEBUG           v4l2videoenc gstv4l2videoenc.c:1433:gst_v4l2_video_enc_handle_frame: NO META RETRIEVED BY ENCODER

NVMEDIA: Need to set EMC bandwidth : 846000 
NVMEDIA_ENC: bBlitMode is set to TRUE

Fiona.Chen · June 12, 2023, 8:29am

What is the gstreamer pipeline state of this stage? The log only shows there is no downstream capability limitation in the negotiation stage. It will not impact anything.

We don’t know anytjing of your DjiCore_Init(), DjiCore_ApplicationStart(), … You’d better debug with your code first.

5kv7zfxvbc · June 13, 2023, 10:22am

Dji* related functions just get raw stream data from the device, and finally give me a Buf and len, and transfer the data to the pipeline through Appsrc
The current pipeline state is GST_STATE_PLAYING

Fiona.Chen · June 13, 2023, 10:29am

From the log, it is not in PLAYING state. Please debug with your code.

5kv7zfxvbc · June 14, 2023, 3:41am

OK

I would like to know when the element nvv4l2h264enc will be output for the following two sentences

H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 846000

Whether the element nvv4l2decoder supports GDR-encoded H264 naked stream data

Fiona.Chen · June 14, 2023, 5:52am

This is hardware encoder log. It is printed when the output buffers are allocated. At this time, the encoder is prepared ready to encoding.

nvv4l2decoder supports any streams compatible to H.264 : Advanced video coding for generic audiovisual services (itu.int).

5kv7zfxvbc · June 14, 2023, 8:26am

For the two elements of nvv4l2decoder and nvv4l2h264enc, is there any official documentation? I searched for a long time on Nvidia’s official website and couldn’t find detailed instructions.

Fiona.Chen · June 14, 2023, 8:32am

What kind of document do you need?

There is document for the plugins Gst-nvvideo4linux2 — DeepStream 6.2 Release documentation

5kv7zfxvbc · June 14, 2023, 8:41am

Ok, thanks, I have read this, is there any other related documents

Fiona.Chen · June 14, 2023, 8:42am

What kind of document do you need?

5kv7zfxvbc · June 14, 2023, 8:43am

Detailed instructions for this plugin in GStreamer

Fiona.Chen · June 14, 2023, 8:47am

The v4l2 plugins usage is as normal as other GStreamer plugins. No special instructions.

They are even open source. You can find the source code here: nv-tegra.nvidia Code Review - tegra/gst-src/gst-nvvideo4linux2.git/summary

5kv7zfxvbc · June 14, 2023, 8:49am

ok thanks