How to visualise the 3d gaze vector output of the GazeNet model?

I suggest you to have a try to visualize your own images with the existing parameters in notebook.

More, you can also download the official gaze model to run inference against your own images.See https://docs.nvidia.com/metropolis/TLT/tlt-user-guide/text/gaze_estimation/gaze_estimation.html#run-inference-on-the-model and
Gaze Estimation | NVIDIA NGC

No, there is no more. The info only locates in def visualize_frame().

Hi Morganh,

Thanks for your answer.

When I run the inference on one image:

tao gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/Data/test0001.png -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY

I get the error:

NotADirectoryError: [Errno 20] Not a directory: '/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors'
Traceback (most recent call last):
  File "/usr/local/bin/gazenet", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.

It looks like the inference step does not accept a single file as in the model document.

When I run the inference on one directory:

tao gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset -k $KEY

I get the error:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 121, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 107, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_data_manager.py", line 58, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 60, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 65, in _extract_json
  File "/usr/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/usr/local/bin/gazenet", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.
2022-01-14 22:27:00,494 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

My input directory has the following structure:

The Config folder is copied from the inference-set dataset. Because the test-dataset has new images, I don’t have the json file as in the inference-set generated by the notebook
image

In short, do you have any suggestion to run the model inference on new images that are not from the training dataset?

Thanks,
Tin

For the error above, I am afraid there is something in the folder mapping.
What is your ~/.tao_mounts.json ?

I suggest you run all the steps in terminal instead of notebook.
Then, login the docker directly.
$ tao gazenet run /bin/bash

Then inside the docker, try to run
# gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/Data/test0001.png -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY

If the path is not available or test001.png is not available, please try to find it inside the docker.

Hi Morganh,

Here is the content of ~/.tao_mounts.json on my machine:

{
    "Mounts": [
        {
            "source": "/home/ubuntu/sources/tao/cv_samples_v1.3.0",
            "destination": "/home/ubuntu/sources/tao/cv_samples_v1.3.0"
        },
        {
            "source": "/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/specs",
            "destination": "/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/specs"
        }
    ]
}

I login to the docker and check all that paths and I am sure they are available:

but when I run the following command

gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/Data/test0001.png -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY

I get the same error

NotADirectoryError: [Errno 20] Not a directory: '/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors'
Traceback (most recent call last):
  File "/usr/local/bin/gazenet", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.

Looking at the error message, it says it assumes the following path /home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors is a directory because it only accepts the input folder instead of a specific file. This is confirmed because when I run the inference step on an input folder

gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/ -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY

the error message is different when running on a file:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 121, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 107, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_data_manager.py", line 58, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 60, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 65, in _extract_json
  File "/usr/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/usr/local/bin/gazenet", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.

The error happens when it tries to execute the script custom_jsonlabels_strategy.py to parse the json file.
I also confirm that I could run the inference step successfully on the validation dataset sample-dataset/inference-set generated from training dataset with the following command:

tao gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml \
                       -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/sample-dataset/inference-set \
                       -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt \
                       -o $USER_EXPERIMENT_DIR/experiment_result/exp1 \
                       -k $KEY

so there should be no problem with the folder mapping.
It seems that the inference step can only accept an input folder and that input folder must have the additional json files in json_datafactory_v2 (those can only be generated from the training dataset).

Can you run the inference step on your machine with the input of a single image or a new image folder to validate that hypothesis?

Thanks,
Tin

I can reproduce your error. I am working on a solution for you.

1 Like

Hi,
There are three approaches to visualize the gaze vector.

  1. Use “gazenet inference”.

In this approach, please leverage the existing data/MPIIFaceGaze/sample-dataset/inference-set/json_datafactory_v2/p01_day03.json
Each image has the “annotations”. Its format follows https://docs.nvidia.com/tao/tao-toolkit/text/data_annotation_format.html#emotionnet-fpenet-gazenet-json-label-data-format

Need to generate a json file for your custom dataset.
Then run “gaze inerfence” to get the result.txt. And use section 7 (Visualize Inference) of https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/resources/cv_samples/version/v1.3.0/files/gazenet/gazenet.ipynb to visualize the gaze vector.

  1. Use deepstream-gaze-app deepstream_tao_apps/apps/tao_others/deepstream-gaze-app at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub

For example, below command will

  • generate an image(gazenet.jpg) with 80 facial landmarks points on it.
  • output the gaze info in the log

./deepstream-gaze-app 1 ../../../configs/facial_tao/sample_faciallandmarks_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/configs/tao_pretrained_models/deepstream_tao_apps/apps/tao_others/deepstream-gaze-app/gaze_test.png ./gazenet

Please dump the facial landmarks points value from the “output array” in deepstream_tao_apps/deepstream_gaze_app.cpp at master · NVIDIA-AI-IOT/deepstream_tao_apps · GitHub
Then leverage the section 7 (Visualize Inference) of https://catalog.ngc.nvidia.com/orgs/nvidia/teams/tao/resources/cv_samples/version/v1.3.0/files/gazenet/gazenet.ipynb to visualize the gaze vector.

BTW, the algorithm is mentioned in utils_gazeviz.py
( wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/tao/cv_samples/versions/v1.3.0/files/gazenet/utils_gazeviz.py )

  1. Use an inference pipeline which is mentioned in an old version of user guide. See https://docs.nvidia.com/tao/archive/tlt-30/text/tlt_cv_inf_pipeline/quick_start_scripts.html and
    https://docs.nvidia.com/tao/archive/tlt-30/text/tlt_cv_inf_pipeline/running_samples.html#running-the-gaze-estimation-sample
    But please note that this approach is already deprecated in latest user guide because we will not maintain it in future.

Hi Morganh,

Thanks for your reply.
I have some comments on each approach as below:

  1. Use “gazenet inference”.
    When looking at the inference-set/json_datafactory_v2/ p01_day03.json file, I see that the annotations are different for each file, i.e. for each image file, we need to know the location of faces, land marks, etc. So, I don’t think I can reuse them for a new image.

  2. Use deepstream-gaze-app
    The app can generate 80 facial landmarks, but the utils_gazeviz.py requires 104 landmarks. So, I guess they are not compatible.

  3. Use an old inference pipeline
    I am not clear how to process with this.

Did any approach above work for you?
Finally, my main goal is to visualise the gaze within the deepstream-gaze-app. Do you have any suggestion to do that directly in the deepstream-gaze-app?

Thanks,
Tin

Actually approach 3 is verified previously. The this example video from NVIDIA is running with it.
For utils_gazeviz.py, I will check if it works after reducing 104 to 80.

Just share a workaround for approach2 which is running inference with deepstream-gaze-app. Please follow the README in attached demo.zip.
It will dump the 80 facial landmarks points in order to visualize the cropped face along with the gaze vector.
demo.zip (372.6 KB)

Main change:

cp deepstream_gaze_app.cpp  bak_deepstream_gaze_app.cpp
cp deepstream_gaze_app_update.cpp  deepstream_gaze_app.cpp

make clean && make

./deepstream-gaze-app 1 ../../../configs/facial_tao/sample_faciallandmarks_config.txt file:///opt/nvidia/deepstream/deepstream-6.0/samples/configs/tao_pretrained_models/deepstream_tao_apps/apps/tao_others/deepstream-gaze-app/frame_816_571_0.png ./gazenet  |tee log.txt

python vis.py

result

Thanks Morganh. I will give it a try.

Cheers,
Tin

Hi Morganh,

I have changed the lines 419 and 423 to:

    "softargmax/strided_slice:0") == 0) {
    "softargmax/strided_slice_1:0") == 0) {

so that the pipeline could work with the TAO gazenet model on jetson nano.
I could visualise the gaze on some new images and the results did not look very accurate when a person was looking straight but the arrows pointed to different directions.
Looking at the utils_gazeviz.py script, I see that the script uses a 3D face model and the intrinsic camera parameters from a public training dataset. Are these camera parameters the same as those of the dataset used to train the TAO gazenet model. If they are different, is that the reason why the visualisation is not accurate? And where is the 3D face model from?

Thanks,
Tin

Firstly, may I know if you have trained a new model against your own dataset?

Hi Morganh,

No, I haven’t trained a new model. I use this gaze model from NVIDIA.

Cheers,
Tin

Thanks for the info. I am afraid the different data distribution between your own images and training images from gaze model may will result in inaccurate inference result.
If possible, could you try to run training against your own dataset?

To narrow down, with gaze model from NVIDIA, can you run “tao inference” against your own dataset to check if it can work? .
For json file, you can leverage the facial points dumped from the deepstream-app.

is this (the example photo) an accurate prediction for where you were looking ?

Hi @Morganh,

I am not sure why we need to run “tao inference” on new images as we have everything we need to visualise from the deepstream-app.
I cannot train the model on my new images as I don’t have labels for them. Besides, I also don’t want to train the model as I just want to use it on new images.
In addition, what does the gazenet model do if it cannot be used on new images that are not from the training dataset?

Cheers,
Tin

To run “tao inference” on a test image is in order to check whether it can work. This will help narrow down the issue. If it works, the gaze model has not problem. Then we need to find the gap between “tao inference” and deepstream-app.
I will run “tao inference” against your test image.

On your side, could you please resize your test image to 1280x720 and try again with deepstream-app?

Hi,
Please run above-mentioned approach 3 for better result.
Step:
Refer to 3.21.08 doc https://docs.nvidia.com/tao/tao-toolkit-archive/tao-30-2108/text/tao_cv_inf_pipeline/requirements_and_installation.html#download-the-tao-toolkit-cv-inference-pipeline-quick-start to download scripts via TAO Computer Vision Inference Pipeline | NVIDIA NGC or
ngc registry resource download-version "nvidia/tao/tao_cv_inference_pipeline_quick_start:v0.3-ga"

Setup server

$ cd tao_cv_inference_pipeline_quick_start_vv0.3-ga/scripts
$ bash tao_cv_init.sh 
$ bash tao_cv_start_server.sh

Open another terminal to run client.

$ export DISPLAY=:0
$ cd tao_cv_inference_pipeline_quick_start_vv0.3-ga/scripts
$ bash tao_cv_start_client.sh

Modify several lines in samples/tao_cv/demo_gaze/demo.conf
root@xx:/workspace/tao_cv-pkg# vim samples/tao_cv/demo_gaze/demo.conf     
            video_path=  /tmp/yourtest.mp4 
            fps = yourvideo_fps
            is_video_path_file= true
            resolution_whc= 640,480,3

root@xx:/workspace/tao_cv-pkg# ./samples/tao_cv/demo_gaze/gaze samples/tao_cv/demo_gaze/demo.conf

BTW, run “docker cp” to cp your testvideo.mp4 to the client.
$ docker cp yourtest.mp4 image_tao_cv_client:/tmp/