Using multiple pipelines in TAO CV

May I know that if you are using default(without any change) gaze application(./samples/tao_cv/demo_gaze/gaze) to test?

In my tests, I used the original gaze_demo example.
I temporarily do not use the program from the topic header, since the problem was noticed precisely in the processing of the gaze, and therefore I use gaze_demo.
The tests, the results of which were marked with success, I interrupted with Ctrl + C to go to the next one.

Thanks for the info. I will take time to try to reproduce.
More, the inference pipeline(including this gaze_demo) will not be supported in next release(very soon) .
For gaze, please run inference with deepstream_tao_apps/apps/tao_others/deepstream-gaze-app at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

May I ask if you managed to reproduce the error mentioned above, and is it possible to somehow fix it?

I would also like to clarify the details of the capabilities of the software you recommend:
1.Software stability compared to TAO Toolkit Computer Vision Inference Pipeline for L4T ( TAO Toolkit CV Inference Pipeline L4T | NVIDIA NGC ) - detectors of gaze, emotions, gestures, etc .:
1) the gaze detector crashed due to a mismatch in the number of detected faces and detected landmarks, the emotion detector is the same, but a little less often;
2) the gesture detector classified any of the gestures listed in the documentation as a “random gesture” (regardless of the value of the binding to the right hand flag).
2. compatible types of cameras (webcam / CSI camera), supported pixel formats, etc.
3.Compatible JetPack firmware versions, for example:
1) Gaze Demo Container for Jetson ( Gaze Demo for Jetson/L4T | NVIDIA NGC ) required only one separate version of JetPack - 4.4 Developer Preview (DP);
2) TAO Toolkit Computer Vision Inference Pipeline for L4T ( TAO Toolkit CV Inference Pipeline L4T | NVIDIA NGC ) only started on JetPack 4.5.1 , didn’t work on JP 4.6.
4. the ability to update the OpenCV libraries built into the container by sharing the version of OpenCV installed on the host.

Sorry, I’m still checking. From your latest comment, the deepstream_tao_apps/apps/tao_others/deepstream-gaze-app at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub will also meet random crash, right?

I’m just rushing you to answer, because here on the forum, the topic is usually automatically closed after 2 weeks.
I previously wanted to clarify information about your proposed DeepStream software and its requirements for hardware and software.
Are there any conflicts with the already installed “TAO Toolkit Computer Vision Inference Pipeline for L4T”?
p.s. The day before yesterday I added BodyPose processing to the program from the topic header using the “TAO Toolkit Computer Vision Inference Pipeline for L4T”, but I also received an error of inconsistency between the number of detected faces and the number of landmarks (the program works for about 10 … 30 seconds and then exits with an error).
p.s. p.s. Sorry for my poor English - I have to use Google’s machine translation.

This is just marked by top moderators. We can open it if there is still issue.

I just ran deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub and did not reproduce similar error.

Please follow deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub 's readme to set up.

You can run it with static images or mp4 video files.

Suggest you install JP4.6 because it will install Deepstream 6.0 GA.

For inference pipeline, I’m still checking whether you can use an old version of it.

I upgraded Jetson to JP4.6, installed DeepStrream 6.0, then followed the instructions on the page: deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub .
An example of a gaze detector accidentally crashes at the first start, but at the next start, the results of the work of the detector are not visible - empty files *.264 / *.jpg are created only (I also tried passing different values for the first argument 1,2 and 3.):

./deepstream-gaze-app 2 …/…/…/configs/facial_tao/sample_faciallandmarks_config.txt file:////home/professorx/gaze_video.mp4 ./gaze

./deepstream-gaze-app 1 …/…/…/configs/facial_tao/sample_faciallandmarks_config.txt file:////home/professorx/gaze.png ./gaze

System info

NVIDIA Jetson Xavier NX (Developer Kit Version)
L4T 32.6.1 [ JetPack 4.6 ]
Ubuntu 18.04.6 LTS
Kernel Version: 4.9.253-tegra
CUDA 10.2.300
CUDA Architecture: 7.2
OpenCV version: 4.1.1
OpenCV Cuda: NO
Vision Works:
VPI: ii libnvvpi1 1.1.15 arm64 NVIDIA Vision Programming Interface library
Vulcan: 1.2.70


ds_gaze_app_output.txt (6.8 KB)

May I know if it is full log?

Did you follow the steps in deepstream_tao_apps/apps/tao_others/deepstream-gaze-app at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub as well?

Yes, I followed the instructions indicated.
The output of the application in the attached file above is all output, after this output the application does nothing, as if it were hanging.
For convenience, I transferred the command lines to scripts, files are attached.
Perhaps there is a problem with the model, or am I doing something wrong?
p.s. I have not edited the configs in ‘deepstream_tao_apps/configs’. (456 Bytes) (377 Bytes)
sample_gazenet_model_config.txt (228 Bytes)

I can run it well on my Xavier.
The part log is as below. So, you can check whether gazenet_facegrid.etlt_b8_gpu0_fp16.engine is generated on your side.

In cb_newpad
###Decodebin pick nvidia decoder plugin.
In cb_newpad
Deserializing engine from: ./gazeinfer_impl/../../../../models/gazenet/gazenet_facegrid.etlt_b8_gpu0_fp16.engineThe logger passed into createInferRuntime differs from one already assigned, 0x55c6961eb0, logger not updated.

Gaze: -6.359375 -26.625000 16.859375 0.023422 0.079285
Gaze: -6.828125 -27.656250 16.859375 0.023438 0.079712
Frame Number = 0 Face Count = 1
1 Like

Unfortunately, this file is missing:

root@x-mansion:~/ssd/DeepStream# find ./ -name "*gazenet*"
root@x-mansion:~/ssd/DeepStream# find ./ -name "*.engine"

I called the script for downloading models with a preliminary indication of the full path to the tao-converter, according to the instructions in point #2 of the Download paragraph deepstream_tao_apps/apps/tao_others at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

export TAO_CONVERTER=/home/professorx/ssd/DeepStream/tao-converter-jp46-trt8.0.1.6/tao-converter
cd /home/professorx/ssd/DeepStream/deepstream_tao_apps

Could you try again without any script?

I re-executed the script, added the export of the path to the tao-converter to it, it worked.
Perhaps when you start the application for the first time, it is important that the path to the tao-converter is exported to the environment?

export CUDA_VER=10.2
export TAO_CONVERTER=/home/professorx/ssd/DeepStream/tao-converter-jp46-trt8.0.1.6/tao-converter
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/nvidia/deepstream/deepstream/lib/cvcore_libs
cd /home/professorx/ssd/DeepStream/deepstream_tao_apps/apps/tao_others/deepstream-gaze-app
./deepstream-gaze-app 3 ../../../configs/facial_tao/sample_faciallandmarks_config.txt file:///home/professorx/gaze_video.mp4 ./gaze
#./deepstream-gaze-app 3 ../../../configs/facial_tao/sample_faciallandmarks_config.txt file:///dev/video0 ./gaze

Yes, suggest to do that. I can also see it is mentioned in the github.

export TAO_CONVERTER=the file path of tao-converter
1 Like

Ok, thanks.

I additionally have two questions:

  1. Is it possible in these examples (registration of gaze, emotions, body position and gestures) to use video capture from a webcam? As far as I understood from the source code of the examples, when processing a video file, the H.264 format is expected, and the webcam provides YUV and MJPG formats.
    When I tried to “take off the swoop” - i.e. specifying ‘/dev/video0’ as the source, then I received an error stating that the device could not be opened.
  2. When combining several detectors into a single application (as I already tried to do similar manipulations in the application for “TAO CV Pipeline”, the source code of which was attached in the first message of the topic), you will also have to use detectors in parallel (capture video and transfer it sequentially to each of the detectors used instead)?
    For example, I transfer video to the gaze detector, from which I can “pick up” the registered image of the face, and then transfer it to the emotion detector, in order to avoid the situation when these two detectors do the same job and unnecessarily load the CPU and GPU of Jetson ( similarly for body position and gesture detectors).

For 1) , refer to using rtsp mentioned in The LPD and LPR models in the TAO tool do not work well - #22 by Morganh
For 2), do you mean the pipeline runs the gaze detector firstly and then runs emotion detector?

  1. Sorry, I didn’t have time to test your proposed code for streaming via the RTSP protocol.
  2. I wanted to merge the paperlines as shown in the diagram in the attached image (gaze and emotion detectors for example).
    p.s. I need to test the detectors on recorded videos on a webcam.
    p.s. p.s. Perhaps it would be better if I create a separate topic for discussion and questions on deepstream_tao_apps?

Yes, please create a new topic about the discussion.

1 Like

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.