How to dynamically modify the resolution after AI inference


filesrc ! h264parse ! nvv4l2decoder  ! nvvideoconvert ! video/x-raw(memory:NVMM),width=1920,height=1080 ! nvstreammux batch-size=1 batched-push-timeout=40000 width=1920 height=1080 ! nvinfer ! nvtracker ! nvvideoconvert ! nvdsosd ! nvvideoconvert ! capsfilter('video/x-raw(memory:NVMM),format=I420,width=1920,height=1080') name=reso ! nvv4l2h264enc ! h264parse ! flvmux ! rtmpsink 

I want to modify the resolution on the filter named reso in front of nvv4l2h264enc, after the modification is completed, the program is abnormal after setting the pipeline to PLAYER state

root@nvidia-desktop:~/shared_develop/develop/gstreamer/build# ./bin/dynamic_resolution 
Working directory changed to: /root/shared_develop/develop/gstreamer/build
With tracker
Using file: ../ai/deep_test2_file/dstest2_config.yml
Opening in BLOCKING MODE 
Opening in BLOCKING MODE 
WARNING: [TRT]: Using an engine plan file across different models of devices is not recommended and is likely to affect performance or even cause errors.
0:00:04.997346099 1680788 0xaaaaffed6240 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<ai_nvinfer> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::deserializeEngineAndBackend() <nvdsinfer_context_impl.cpp:1909> [UID = 1]: deserialized trt engine from :/root/shared_develop/develop/gstreamer/ai/deep_test2_file/yolov8s.onnx_b1_gpu0_fp32.engine
WARNING: [TRT]: The getMaxBatchSize() function should not be used with an engine built from a network created with NetworkDefinitionCreationFlag::kEXPLICIT_BATCH flag. This function will always return 1.
INFO: [Implicit Engine Info]: layers num: 2
0   INPUT  kFLOAT input           3x640x640       
1   OUTPUT kFLOAT output          8400x6          

0:00:05.163530388 1680788 0xaaaaffed6240 INFO                 nvinfer gstnvinfer.cpp:680:gst_nvinfer_logger:<ai_nvinfer> NvDsInferContext[UID 1]: Info from NvDsInferContextImpl::generateBackendContext() <nvdsinfer_context_impl.cpp:2012> [UID = 1]: Use deserialized engine model: /root/shared_develop/develop/gstreamer/ai/deep_test2_file/yolov8s.onnx_b1_gpu0_fp32.engine
0:00:05.192889012 1680788 0xaaaaffed6240 INFO                 nvinfer gstnvinfer_impl.cpp:328:notifyLoadModelStatus:<ai_nvinfer> [UID 1]: Load new model:/root/shared_develop/develop/gstreamer/ai/deep_test2_file/dstest2_pgie_config.yml sucessfully
Running...
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 817800 
NVMEDIA_ENC: bBlitMode is set to TRUE 
7
Modify the resolution to:1280*720
NvMMLiteOpen : Block : BlockType = 4 
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4 
H264: Profile = 66, Level = 0 
NVMEDIA: Need to set EMC bandwidth : 363466 
NVMEDIA_ENC: bBlitMode is set to TRUE 
Segmentation fault (core dumped)

The plugin does not support dynamic resolution changes when it’s running. You need to customize the code yourself, remove the old plugin and add the new plugin dynamiclly.

Hello, since the dynamic deletion and addition will cause the flow to be interrupted, I want to try to optimize the uninterrupted flow. Is there any official suggestion?

If you don’t want to dynamic deletion and addition, you can try to tee multiple streams with different resolutions. You can control the which stream to play on your own.

videotestsrc ! video/x-raw,format=I420,width=1920,height=1080 ! tee name=t \
	t. ! nvvideoconvert ! video/x-raw(memory:NVMM),format=I420,width=1280,height=720 ! nvv4l2h264enc ! h264parse ! flvmux ! rtmpsink
	t. ! nvvideoconvert ! video/x-raw(memory:NVMM),format=I420,width=720,height=480 ! nvv4l2h264enc ! h264parse ! flvmux ! rtmpsink

Hello, do you mean to set the element behind t. into an independent pipeline, and then dynamically modify the state of the corresponding pipeline to achieve the modification resolution?

Yes. The addresses of two RTMPs are different. So you can control which uri to choose at any time on the client.

Thanks, it has been implemented to push the stream to an rtmp, but the player will freeze for about 5s when playing. It is found that the player is stuck when the nvv4l2h264enc element is loaded, and it can be used for the second time. The player can be switched within 1s. I wonder if there is a hot-loading method on the official side? The code below is the record when loading this element. It only needs to be loaded for the first time, and the follow-up will be smooth

Change success.
NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
H264: Profile = 100, Level = 0
NVMEDIA: Need to set EMC bandwidth : 376000
NVMEDIA_ENC: bBlitMode is set to TRUE
Found NAL Unit at offset: 13, Type: 4

Could you attach you whole pipeline now? Do you mean it takes 5s for the nvv4l2h264enc to load for the first time?

videotestsrc ! video/x-raw,format=I420,width=1280,height=720 ! output-selector  !
     nvvideoconvert !  video/x-raw(memory:NVMM),format=I420,width=1080,height=1080  ! nvv4l2h264enc ! h264parse  ! flvmux ! identity !
     nvvideoconvert !  video/x-raw(memory:NVMM),format=I420,width=1080,height=720  ! nvv4l2h264enc ! h264parse  ! flvmux ! identity !
     nvvideoconvert !  video/x-raw(memory:NVMM),format=I420,width=1080,height=480  ! nvv4l2h264enc ! h264parse  ! flvmux ! identity !
input-selector ! rtmpsink

Hello, the pipeline uses output-selector and input-selector input and output selectors to change the three resolutions in the middle, and the activation selector code is attached below

g_object_set(output_selector , "active-pad" , src_pad , NULL);
g_object_set(input_selector , "active-pad" , sink_pad , NULL);

When the corresponding pipeline is activated above, the corresponding nvv4l2h264enc element needs to be initialized, that is, the code below is executed, and then the client freezes for 4~5s, and then the pipeline that has loaded the element will complete the switch within 1s

NvMMLiteOpen : Block : BlockType = 4
===== NVMEDIA: NVENC =====
NvMMLiteBlockCreate : Block : BlockType = 4
H264: Profile = 100, Level = 0
NVMEDIA: Need to set EMC bandwidth : 376000
NVMEDIA_ENC: bBlitMode is set to TRUE
Found NAL Unit at offset: 13, Type: 4

Could you attach your simple demo code that can run? Or you can open more log by adding GST_DEBUG=3 to check the time consumption.

Hello, the code outputs to output and then selects the corresponding Bin to switch, then uses input to control the input data and then pushes it to Rtmp, and finally activates the pad of the selector randomly through a loop. The attachment is the running code and GST_DEBUG=3 is the log

gst.log (7.4 KB)
out_input_selector_test.cpp (14.0 KB)

There are many states of the pipeline that need to be synchronized when switching scenarios. Can you consider the following switching scheme?

         ->nvvideoconvert !  video/x-raw(memory:NVMM),format=I420,width=1080,height=720! nvv4l2h264enc ! h264parse  ! flvmux ! identity ! rtmpsink0
src->tee->nvvideoconvert !  video/x-raw(memory:NVMM),format=I420,width=1920,height=1080  ! nvv4l2h264enc ! h264parse  ! flvmux ! identity ! rtmpsink1
        ->nvvideoconvert !  video/x-raw(memory:NVMM),format=I420,width=640,height=480! nvv4l2h264enc ! h264parse  ! flvmux ! identity ! rtmpsink2

This can simultaneously generate three different resolution rtmp streams, allowing the client to choose which one they need. This can support seamless stream switching.

Hello, this multi-branch will cause extra traffic overhead, right?

Yes. But it can support seamless stream switching. If you want to save the resources, you need to accept the issue of time-consuming switching.

Thank you for your support, I will consider it in the future.

The final solution is this method. The playback end will be stuck for 1~3 seconds. There is currently no better solution.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.