How to visualise the 3d gaze vector output of the GazeNet model?

Accelerated Computing Intelligent Video Analytics TAO Toolkit
11

Hi Morganh,

Here is the content of ~/.tao_mounts.json on my machine:

{
    "Mounts": [
        {
            "source": "/home/ubuntu/sources/tao/cv_samples_v1.3.0",
            "destination": "/home/ubuntu/sources/tao/cv_samples_v1.3.0"
        },
        {
            "source": "/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/specs",
            "destination": "/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/specs"
        }
    ]
}

I login to the docker and check all that paths and I am sure they are available:

image
image2334×656 171 KB

but when I run the following command

gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/Data/test0001.png -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY

I get the same error

NotADirectoryError: [Errno 20] Not a directory: '/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors'
Traceback (most recent call last):
  File "/usr/local/bin/gazenet", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.

Looking at the error message, it says it assumes the following path /home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors is a directory because it only accepts the input folder instead of a specific file. This is confirmed because when I run the inference step on an input folder

gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/ -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY

the error message is different when running on a file:

Traceback (most recent call last):
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 121, in <module>
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 107, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_data_manager.py", line 58, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 60, in __init__
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 65, in _extract_json
  File "/usr/lib/python3.6/posixpath.py", line 80, in join
    a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
  File "/usr/local/bin/gazenet", line 8, in <module>
    sys.exit(main())
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
  File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.

The error happens when it tries to execute the script custom_jsonlabels_strategy.py to parse the json file.
I also confirm that I could run the inference step successfully on the validation dataset sample-dataset/inference-set generated from training dataset with the following command:

tao gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml \
                       -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/sample-dataset/inference-set \
                       -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt \
                       -o $USER_EXPERIMENT_DIR/experiment_result/exp1 \
                       -k $KEY

so there should be no problem with the folder mapping.
It seems that the inference step can only accept an input folder and that input folder must have the additional json files in json_datafactory_v2 (those can only be generated from the training dataset).

Can you run the inference step on your machine with the input of a single image or a new image folder to validate that hypothesis?

Thanks,
Tin