Hi Morganh,
Here is the content of ~/.tao_mounts.json on my machine:
{
"Mounts": [
{
"source": "/home/ubuntu/sources/tao/cv_samples_v1.3.0",
"destination": "/home/ubuntu/sources/tao/cv_samples_v1.3.0"
},
{
"source": "/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/specs",
"destination": "/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/specs"
}
]
}
I login to the docker and check all that paths and I am sure they are available:
but when I run the following command
gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/Data/test0001.png -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY
I get the same error
NotADirectoryError: [Errno 20] Not a directory: '/home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors'
Traceback (most recent call last):
File "/usr/local/bin/gazenet", line 8, in <module>
sys.exit(main())
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.
Looking at the error message, it says it assumes the following path /home/ubuntu/sources/tao/cv_samples_v1.3.0/gazenet/data/MPIIFaceGaze/test-dataset/Data/test0001.png/errors is a directory because it only accepts the input folder instead of a specific file. This is confirmed because when I run the inference step on an input folder
gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml -i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/test-dataset/ -m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt -o $USER_EXPERIMENT_DIR/experiment_result/test-dataset/ -k $KEY
the error message is different when running on a file:
Traceback (most recent call last):
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 121, in <module>
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/scripts/inference.py", line 107, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_data_manager.py", line 58, in __init__
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 60, in __init__
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/dataio/custom_jsonlabels_strategy.py", line 65, in _extract_json
File "/usr/lib/python3.6/posixpath.py", line 80, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
Traceback (most recent call last):
File "/usr/local/bin/gazenet", line 8, in <module>
sys.exit(main())
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/mitgazenet/entrypoint/gazenet.py", line 13, in main
File "/root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/driveix/build_wheel.runfiles/ai_infra/driveix/common/entrypoint/entrypoint.py", line 300, in launch_job
AssertionError: Process run failed.
The error happens when it tries to execute the script custom_jsonlabels_strategy.py to parse the json file.
I also confirm that I could run the inference step successfully on the validation dataset sample-dataset/inference-set generated from training dataset with the following command:
tao gazenet inference -e $SPECS_DIR/gazenet_tlt_pretrain.yaml \
-i $DATA_DOWNLOAD_DIR/MPIIFaceGaze/sample-dataset/inference-set \
-m $USER_EXPERIMENT_DIR/experiment_result/exp1/model.tlt \
-o $USER_EXPERIMENT_DIR/experiment_result/exp1 \
-k $KEY
so there should be no problem with the folder mapping.
It seems that the inference step can only accept an input folder and that input folder must have the additional json files in json_datafactory_v2 (those can only be generated from the training dataset).
Can you run the inference step on your machine with the input of a single image or a new image folder to validate that hypothesis?
Thanks,
Tin