Custom new model segmentation in DeepStream

Can you generate engine with tao-converter tool? TAO Converter | NVIDIA NGC

Yes. I will try.

Sorry, but I got error “tao-converter: command not found”

Please download the binary tool according to your platform. E.G. for x86 dGPU platform:

wget --content-disposition 'https://api.ngc.nvidia.com/v2/resources/nvidia/tao/tao-converter/versions/v3.22.05_trt8.4_x86/files/tao-converter'

Here after I run your command

@pesuyn444 According to Linux OS rules, you need to run “./tao-converter -h” or add the path to the system execution path list. Please do not raise such issue in this forum, this is just DeepStream forum.

Sorry but i want to ask one more question about TAO-Converter, I have problem running it

@pesuyn444 Please refer to the command in Deploying to Deepstream — TAO Toolkit 4.0 documentation . You need to use “-p” parameter.

Yes, it worked. Thanks you so much.

One more question. Can I use any Pre-trained Unet model and deploy it to Deepstream, for example I have file .h5. So what do I need to do?

I think I’ve post the types of models which DeepStream can support:

DeepStream can support the following types of models:

  • Caffe Model and Caffe Prototxt
  • ONNX
  • UFF file
  • TAO Encoded Model and Key

Hi,
I run successful Unet in deepstream_tao_apps with 1 file video. So now I want to run Unet with my webcam, So what do I need to do?
Thanks

Just input your webcam stream as the normal RTSP source. There are lots of samples in DeepStream SDK. E.G. deepstream-app DeepStream Reference Application - deepstream-app — DeepStream 6.1.1 Release documentation

Please read the document and try the samples.

And please make sure you are familiar with GStreamer( GStreamer: open source multimedia framework) before you start with DeepStream since you want to do some customization work.

1.

I run example peopleSemSegNetVanilla in deepstream_tao_apps with my video mp4 by command:
./apps/tao_segmentation/ds-tao-segmentation -c configs/peopleSemSegNet_tao/vanilla/pgie_peopleSemSegVanillaUnet_tao_config.txt -i file:///media/anlab/data/DeepStream/Sample/samples/streams/sample_720p.mp4

  • My file pgie_peopleSemSegVanillaUnet_tao_config.txt:

[property]
gpu-id=0
net-scale-factor=0.007843

model-color-format=1
offsets=127.5;127.5;127.5
labelfile-path=./peopleSemSegVanillaUnet_labels.txt

##Replace following path to your model file

model-engine-file=…/…/…/models/peopleSemSegNet/vanilla/peoplesemsegnet_vanilla_unet_dynamic_etlt_int8_fp16.etlt_b1_gpu0_fp16.engine

#DS5.x cannot parse onnx etlt model, so you need to
#convert the etlt model to TensoRT engine first use tao-convert

int8-calib-file=…/…/…/models/peopleSemSegNet/vanilla/peoplesemsegnet_vanilla_unet_dynamic_etlt_int8.cache
tlt-encoded-model=…/…/…/models/peopleSemSegNet/vanilla/peoplesemsegnet_vanilla_unet_dynamic_etlt_int8_fp16.etlt

#tlt-encoded-model=…/…/…/models/peopleSemSegNet/vanilla/peoplesemsegnet_vanilla_unet_dynamic_etlt_fp32.etlt

tlt-model-key=tlt_encode

infer-dims=3;544;960
batch-size=1

#0=FP32, 1=INT8, 2=FP16 mode

network-mode=2
num-detected-classes=2
interval=0
gie-unique-id=1
network-type=100
output-blob-names=argmax_1
segmentation-threshold=0.0
output-tensor-meta=1

##specify the output tensor order, 0(default value) for CHW and 1 for HWC

segmentation-output-order=1

[class-attrs-all]
roi-top-offset=0
roi-bottom-offset=0
detected-min-w=0
detected-min-h=0
detected-max-w=0
detected-max-h=0

2.

I want run with my webcam so I run example “deepstream_app_source1_segmentation.txt” and change source to webcam and run command:

sudo deepstream-app -c deepstream_app_source1_segmentation.txt -t

  • File deepstream_app_source1_segmentation.txt:

[application]
enable-perf-measurement=1
perf-measurement-interval-sec=1

[tiled-display]
enable=1
rows=1
columns=1
width=960
height=540
gpu-id=0

[source0]
enable=1
type=1
camera-width=1280
camera-height=960
camera-fps-n=5
camera-fps-d=1
camera-v4l2-dev-node=0
#uri=file:/media/anlab/data/DeepStream-Yolo/data/test8.mp4
num-sources=1
gpu-id=0
cudadec-memtype=0

[streammux]
gpu-id=0
batch-size=1
batched-push-timeout=40000
#set muxer output width and height
width=960
height=540

[sink0]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File
type=2
sync=0
source-id=0
gpu-id=0

[osd]
enable=1
gpu-id=0
border-width=3
text-size=15
text-color=1;1;1;1;
text-bg-color=0.3;0.3;0.3;1
font=Serif
display-mask=1
display-bbox=1
display-text=1

[primary-gie]
enable=1
gpu-id=0
#Modify as necessary
batch-size=1
#Required by the app for OSD, not a plugin property
bbox-border-color0=1;0;0;1
bbox-border-color1=0;1;1;1
bbox-border-color2=0;0;1;1
bbox-border-color3=0;1;0;1
gie-unique-id=1
#Replace the infer primary config file when you need to
#use other detection models
#model-engine-file=…/…/models/tao_pretrained_models/mrcnn/mask_rcnn_resnet50.etlt_b1_gpu0_int8.engine
config-file=config_infer_primary_peopleSegNet.txt

[sink1]
enable=0
type=3
#1=mp4 2=mkv
container=1
#1=h264 2=h265 3=mpeg4
codec=1
encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=2000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
output-file=out.mp4
source-id=0

[sink2]
enable=0
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=4
#1=h264 2=h265
codec=1
encoder type 0=Hardware 1=Software
enc-type=0
sync=0
bitrate=4000000
#H264 Profile - 0=Baseline 2=Main 4=High
#H265 Profile - 0=Main 1=Main10
profile=0
set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400

[tracker]
enable=1
#For NvDCF and DeepSORT tracker, tracker-width and tracker-height must be a multiple of 32, respectively
tracker-width=640
tracker-height=384
ll-lib-file=/opt/nvidia/deepstream/deepstream/lib/libnvds_nvmultiobjecttracker.so
#ll-config-file required to set different tracker types
#ll-config-file=…/deepstream-app/config_tracker_IOU.yml
ll-config-file=…/deepstream-app/config_tracker_NvDCF_perf.yml
#ll-config-file=…/deepstream-app/config_tracker_NvDCF_accuracy.yml
#ll-config-file=…/deepstream-app/config_tracker_DeepSORT.yml
gpu-id=0
enable-batch-process=1
enable-past-frame=1
display-tracking-id=1

[tests]
file-loop=0

  • File config_infer_primary_peopleSegNet.txt:

[property]
gpu-id=0
net-scale-factor=0.017507
offsets=123.675;116.280;103.53
model-color-format=1
tlt-model-key=nvidia_tlt
tlt-encoded-model=/media/anlab/data/deepstream_tao_apps/models/peopleSegNet/V2/peoplesegnet_resnet50.etlt
model-engine-file=/media/anlab/data/deepstream_tao_apps/models/peopleSegNet/V2/peoplesegnet_resnet50.etlt_b1_gpu0_int8.engine
network-type=3 ## 3 is for instance segmentation network
labelfile-path=./peopleSegNet_labels.txt
int8-calib-file=/media/anlab/data/deepstream_tao_apps/models/peopleSegNet/V2/peoplesegnet_resnet50_int8.txt
infer-dims=3;576;960
num-detected-classes=2
uff-input-blob-name=Input
batch-size=1
##0=FP32, 1=INT8, 2=FP16 mode
network-mode=1
interval=0
gie-unique-id=1
#no cluster
##1=DBSCAN, 2=NMS, 3= DBSCAN+NMS Hybrid, 4 = None(No clustering)
##MRCNN supports only cluster-mode=4; Clustering is done by the model itself
cluster-mode=4
output-instance-mask=1
output-blob-names=generate_detections;mask_fcn_logits/BiasAdd
parse-bbox-instance-mask-func-name=NvDsInferParseCustomMrcnnTLTV2
custom-lib-path=/opt/nvidia/deepstream/deepstream/lib/libnvds_infercustomparser.so

[class-attrs-all]
pre-cluster-threshold=0.8

3

I want to change model peoplesegnet_resnet50.etlt to model peoplesemsegnet_vanilla_unet_dynamic_etlt_int8_fp16.etlt. So I copy all config in file pgie_peopleSemSegVanillaUnet_tao_config.txt and paste to file config_infer_primary_peopleSegNet.txt. And run file deepstream_app_source1_segmentation.txt. Run successful but not segmentation:

The peoplesegnet is an instance segmentation model ( CTSE-AI_Computing / DeepStream / deepstream_tao_apps · GitLab (nvidia.com)), so you can not use your segmentation model as peoplesegnet model. Just refer to the v4l2 camera source part in the deepstream-app source code and modify your code according to it.

Please make sure you are familiar with GStreamer( GStreamer: open source multimedia framework) before you start with DeepStream since you want to do some customization work.

Means I need to change the input file mp4 in command : ./apps/tao_segmentation/ds-tao-segmentation -c configs/peopleSemSegNet_tao/vanilla/pgie_peopleSemSegVanillaUnet_tao_config.txt -i file:///media/anlab/data/DeepStream/Sample/samples/streams/sample_720p.mp4, change to webcam?

Please read the source code deepstream_tao_apps/deepstream_seg_app.c at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub, this app only support uridecodebin source, you need to change the code to v4l2src source. The v4l2src sample code can refer to the create_camera_source_bin() function in /opt/nvidia/deepstream/deepstream/sources/apps/apps-common/src/deepstream_source_bin.c

DeepStream is a SDK, we have provided lots of sample codes to show how to use DeepStream APIs.

Please make sure you are familiar with GStreamer( GStreamer: open source multimedia framework) before you start with DeepStream since you want to do some customization work.

I copy create_camera_source_bin() function in /opt/nvidia/deepstream/deepstream/sources/apps/apps-common/src/deepstream_source_bin.c and paste in deepstream_tao_apps/apps/tao_segmentation/deepstream_seg_app.c and edit function create_source_bin() but when I run “make” error:

Could you guide me in detail? Please.
Thanks

There is no update from you for a period, assuming this is not an issue anymore. Hence we are closing this topic. If need further support, please open a new one. Thanks

Please learn how to write and debug c/c++ code by yourself. Please make sure you are familiar with GStreamer knowledge and coding skills ( GStreamer: open source multimedia framework) before you start with DeepStream since you want to do some customization work.

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.