Cannot generate samples data from Docker Isaac Object Detection Training

sofiahradzi · January 4, 2021, 1:13am

Hi I tried to run example in jupyter from this link
https://docs.nvidia.com/isaac/isaac/doc/tutorials/training_in_docker.html

Have tried so many times but no data been generated and cannot access http://localhost:3000/

Creating data folder /workspace/tlt-experiments/data WARNING: flashing images!! ***** Don’t visit the following URL if you are sensitive to flashing lights ******* Go to http://localhost:3000 see the generated images being generated
Generated 0 samples
Generated 0 samples
Generated 0 samples
Generated 0 samples
Generated 0 samples

sofiahradzi · January 7, 2021, 8:17am

Im currently using Isaac SDK 2020.2 If I tried this link (https://docs.nvidia.com/isaac/isaac/packages/detect_net/doc/detect_net.html) I managed to simulate data the dolly objects using my host PC. My aim is to try generate on custom dataset other other object than dolly. The documentation is unclear for me.

But when try from jupyter for simulation from docker following from link(https://docs.nvidia.com/isaac/isaac/doc/tutorials/training_in_docker.html) , it doesnt show any error but only shows generate 0 samples endlessly. In the docker training link, it says I can change to my own .fbx file to customized.

Here the result if I run the command
docker run --gpus all nvidia/cuda:10.0-base nvidia-smi ,
here from my screen.

hemals · January 11, 2021, 11:04pm

Could you show me a listing of the files in your ~/workspace/isaac-experiments directory and the contents of ~/isaac-experiments/start.sh from “First Run” in “Training Object Detection from Simulation in Docker” doc, please?

If you can, can you modify start.sh and add set -x at the top right after the license agreement? This will echo all of the shell commands to console so we can see what was being invoked.

While running, could you check the contents of ~/isaac-experiments/.ipcas well? This will tell us what the PIDs of the containerized Isaac Sim process and the KITTI dataset generator process. If either of these are blank, something must have gone wrong and.

The error line you are seeing is with “Generated 0 samples” is just the script polling the file system for the generated assets written out by background process of Isaac Sim-Unity3d which doesn’t seem to be happening.

Finally, I had suspected port 3000 open an issue with just exposing the port, but the scripts should run the container with ports 8888 and 3000 exposed to host, so should not be a problem.

sofiahradzi · January 18, 2021, 7:41am

@hemals

Here after I put set -x in the start.sh

./start.sh
++ docker -v
++ grep 19
++ wc -l

IS_19=0
[[ 0 == 1 ]]
NV_FLAG=‘–runtime=nvidia -e CUDA_VISIBLE_DEVICES=all’
VIDEO_OPTS=
[[ -e /dev/video0 ]]
echo ‘We will need your ROOT password to enable access to your display from inside Docker’
We will need your ROOT password to enable access to your display from inside Docker
echo ‘’
DISPLAY=:1
sudo xhost +
access control disabled, clients can connect from any host
echo ‘Starting tutorial use Ctrl-C to stop.’
Starting tutorial use Ctrl-C to stop.
docker login nvcr.io
Authenticating with existing credentials…
WARNING! Your password will be stored unencrypted in /home/sofiah/.docker/config.json.
Configure a credential helper to remove this warning. See
docker login | Docker Documentation

Login Succeeded
++ id -u
++ id -u

docker run --runtime=nvidia -e CUDA_VISIBLE_DEVICES=all -e DISPLAY=:1 -e USER=1000 -v /tmp/.X11-unix:/tmp/.X11-unix:rw -v /home/sofiah/isaac-experiments:/workspace -p 3000:3000 -p 8888:8888 -u 1000:root -it nvcr.io/nvidia/isaac-ml-training:2020.1
- Open your browser and go to: *
- http://localhost:8888/notebooks/object_detection_from_sim.ipynb?token=object-detection-from-sim *
- ```
                                                                                            *
```
Press Ctrl-C twice to exit