Large number of samples for dataset conversion for PointPillars

Accelerated Computing Intelligent Video Analytics TAO Toolkit
22

Done these and tried to run the docker again, but still got the same result.

23

Can you run $ cat /etc/docker/daemon.json again?
If there is still empty, please vim it as below.

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    },
    "default-runtime": "nvidia"
}

Then, update the docker.

sudo pkill -SIGHUP dockerd
24

It worked and I entered the docker as a root.
What should I do next?

25

OK, please run below. Note that local-yourname should be modified.
$ sudo chown local-yourname:docker /var/run/docker.sock
$ sudo usermod -a -G docker local-yourname

26

Do you mean you can run below command successfully, right?
$ docker run --runtime=nvidia -it --rm --shm-size 32G nvcr.io/nvidia/tao/tao-toolkit:5.2.0-pyt2.1.0 /bin/bash

If yes, please run below command inside the docker. Note that there is not tao in the beginning.
# pointpillars dataset_convert xxx

27

image
image798×211 7.09 KB

28

OK, please run below to check if there is still memory issue.
# pointpillars dataset_convert xxx

29
  1. Does “yourname” refer to my account on the system?
  2. It seems that “sudo” doesn’t exist in this environment.
30

Yes.

Ignore doing this now. You already run inside the docker successfully now.

Could you share $nvidia-smi as well?

31

Couldn’t locate the yaml file when I’m in the docker.

image
image1170×80 7.7 KB

32

Please add -v in the command line to map your_local_path to docker path.
For example,

$ docker run --runtime=nvidia -it --rm --shm-size 32G -v /home/morganh/localfolder:/workspace/tao-experiments/
nvcr.io/nvidia/tao/tao-toolkit:5.2.0-pyt2.1.0 /bin/bash`

33

image

As for the virtual memory consumption, it’s around 11.2G and there’s no sign of it increasing till full consumption over time.

image
image1160×171 10.9 KB

So it’s strange.
Running dataset_convert directly from terminal or docker looks fine, but memory consumption may increase over time if I do the same thing via .ipynb from firefox.

34

In the terminal, please let it continue to run. Please check if it can run successfully in the end.

35

Successfully, and the peak memory consumption is 11.7 GB.

36

OK, so can you share ~/.tao_mounts.json？

Please refer to TAO Toolkit Launcher - NVIDIA Docs to set shm_size as well in it.
Then run again to check if it is successful in notebook.

37

Currently the content is as shown below:
image

38

Refer to TAO Toolkit Launcher - NVIDIA Docs, you can add

    "DockerOptions": {
        "shm_size": "32G",
        "ulimits": {
            "memlock": -1,
            "stack": 67108864
        },
        "ports": {
            "8888": 8888
        }
    }
39

Added this part to the mount file and the port I used to run .ipynb on firefox was exactly 8888. I modified the port in the DockerOptions to some other port like 8890 so that dataset_convert could start running.

However, the mem consumption still acts like how it used to. Obviously increasing.

40

Is it successful or “crashed” ?

41

It’s still running, with memory consumption obviously increasing as well.
So far near 10K out of the 45K point cloud data have been processed in dataset_convert and the memory consumption has reached 26 GB so far. I’m expecting that the tab would just crash again when it reaches full memory consumption once again during the process.