Try to Run tao detectnet_v2 command inside of docker and fork tao toolkit tf

Accelerated Computing Intelligent Video Analytics TAO Toolkit
1

Please provide the following information when requesting support.

• Hardware (T4/V100/Xavier/Nano/etc)
• Network Type Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here) = 3.22.05

Although I have the nvidia-tao python library installed and am attempting to run the tao command in a docker container, tao is unable to mount the directory for the container. It displays a file not found:
OSError: Specfile not found at: /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt

3

Usually this kind of error is due to wrong setting in ~/.tao_mounts.json.
Please double check.

A simple way is that you can set destination to the same as source.

4

tao mount.json was also examined. Furthermore, it is the right one.

5

You can access to the tao docker directly to check if the file exists.
$ tao detectnet_v2 run /bin/bash

Then run something as below.
ls /workspace/tao-experiments/

6

I’m trying this and below directory

logs
logs1344×481 96.1 KB

7

Please share your ~/.tao_monuts.json.

8

{
“Mounts”: [
{
“source”: “/app/backend_codes/…/workspace/mangesh123/test2134/dataset”,
“destination”: “/workspace/tao-experiments”
},
{
“source”: “/app/backend_codes/…/workspace/mangesh123/test2134/models”,
“destination”: “/workspace/tao-experiments/detectnet_v2/specs”
}
],
“DockerOptions”: {
“shm_size”: “16G”,
“ulimits”: {
“memlock”: -1,
“stack”: 67108864
},
“user”: “0:0”,
“ports”: {
“8888”: 8888
}
}

9

They may conflict.

You can just set only one.
{
“source”: “/app/backend_codes/…/workspace/mangesh123/test2134”,
“destination”: “/workspace/tao-experiments”
}

10

I am referee this format cv_sample

drive_map = {
    "Mounts": [
            # Mapping the data directory
            {
                "source": os.environ["LOCAL_PROJECT_DIR"],
                "destination": "/workspace/tao-experiments"
            },
            # Mapping the specs directory.
            {
                "source": os.environ["LOCAL_SPECS_DIR"],
                "destination": os.environ["SPECS_DIR"]
            },
        ],
    "DockerOptions": {
        "user": "{}:{}".format(os.getuid(), os.getgid())
    }
}
11

You can set to
{
“source”: “/app/backend_codes/…/workspace/mangesh123/test2134”,
“destination”: “/workspace/tao-experiments”
}

or

{
“source”: “/app/backend_codes/…/workspace/mangesh123/test2134/dataset”,
“destination”: “/workspace/tao-experiments/detectnet_v2/dataset”
},
{
“source”: “/app/backend_codes/…/workspace/mangesh123/test2134/models”,
“destination”: “/workspace/tao-experiments/detectnet_v2/specs”
}

12

I tried both solutions, but they were unsuccessful.

OSError: Specfile not found at: /workspace/tao-experiments/detectnet_v2/specs/detectnet_v2_train_resnet18_kitti.txt

13

Why the result folder (experiment_dir_unpruned) locate in your spec folder?
Is it expected? Can you share the full training command line ?

14

tao detectnet_v2 train -e $SPECS_DIR/detectnet_v2_train_resnet18_kitti.txt -r $SPECS_DIR/experiment_dir_unpruned -k $KEY -n resnet18_detector --gpus $NUM_GPUS

inside docker Im also try this command with full path but showing same error

15

Please change to
-r experiment_dir_unpruned.

More, can you share your latest ~/.tao_mounts.json?

16

First try this,

"Mounts": [
    {
        "source": "/app/backend_codes/../workspace/diycam/tews1323/dataset",
        "destination": "/workspace/tao-experiments/detectnet_v2/dataset"
    },
    {
        "source": "/app/backend_codes/../workspace/diycam/tews1323/models",
        "destination": "/workspace/tao-experiments/detectnet_v2/specs"
    }
],
"DockerOptions": {
    "shm_size": "16G",
    "ulimits": {
        "memlock": -1,
        "stack": 67108864
    },
    "user": "0:0"
}

second try this:
{
“Mounts”: [
{
“source”: “/app/backend_codes/…/workspace/diycam/tews1323”,
“destination”: “/workspace/tao-experiments”
}
],
“DockerOptions”: {
“shm_size”: “16G”,
“ulimits”: {
“memlock”: -1,
“stack”: 67108864
},
“user”: “0:0”
}

17

Are you sure the spec files are available locally?
$ ls /app/backend_codes/…/workspace/diycam/tews1323

$ find /app/backend_codes/…/workspace/diycam/tews1323
-name detectnet_v2_train_resnet18_kitti.txt

18

find workspace/diycam/tews1323/models/detectnet_v2_train_resnet18_kitti.txt
this right path

19

So, when run the second tao_mounts.json, can you find the spec file via below command?
$ tao detectnet_v2 run ls /workspace/tao-experiments/models/detectnet_v2_train_resnet18_kitti.txt

20

Screenshot from 2022-10-13 12-05-22
Screenshot from 2022-10-13 12-05-221328×173 36 KB

21

All the path inside the docker is defined according to “destination” in tao_mount.json.

So, if you set 2nd
“source”: “/app/backend_codes/…/workspace/diycam/tews1323”,
“destination”: “/workspace/tao-experiments”

Please run
$ tao detectnet_v2 run ls /workspace/tao-experiments/models/detectnet_v2_train_resnet18_kitti.txt