Training Object Detection from Simulation in Docker on Ubuntu 20.04

Autonomous Machines Isaac Isaac SDK
#1

Hi all,

is it possible to run the object detection tutorial
https://docs.nvidia.com/isaac/isaac/doc/tutorials/training_in_docker.html
on a system with Ubuntu 20.04?

I could successfully create and execute the start.sh script opening the jupyter notebook. When running the first cells an Unity 3D window pops up constantly flashing generated training images. However, the notebook responses with “Generated Samples: 0” (see appended picture) while localhost:3000 is not accessible.

Hope you can help!

error_snap
error_snap1531×1095 349 KB

Cannot generate samples data from Docker Isaac Object Detection Training
#2

Hi I got the same problem on Ubuntu 18.
This is the same problem Cannot generate samples data from Docker Isaac Object Detection Training

1 Like
#3

Could you verify that GPU support in Docker is working correctly with the following command from the tutorial?

docker run --gpus all nvidia/cuda:10.0-base nvidia-smi

We have seen this issue when GPUs are not available properly in the container before at least.

#4

Looks good to me:

jesser@9700-FJLTC63-Ubuntu:~/dev-isaac$ docker run --gpus all nvidia/cuda:10.0-base nvidia-smi
Fri Feb 19 13:49:01 2021       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce RTX 206...  Off  | 00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8     4W /  N/A |    528MiB /  5934MiB |      5%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+

The outputs are similiar to Cannot generate samples data from Docker Isaac Object Detection Training - #5 by sofiahradzi

Any suggestions from your side on how to debug this issue?

#5

Hi, I got my solutions on Ubuntu 18.

Modify the start.sh like this

#if [[ $IS_19 == 1 ]]; then
NV_FLAG="--gpus=all"
#else
#    NV_FLAG="--runtime=nvidia -e CUDA_VISIBLE_DEVICES=all"
#fi

Because I found my docker’s version is 20.
It will use NV_FLAG="--runtime=nvidia -e CUDA_VISIBLE_DEVICES=all", but this will cause the problem.
So I change to NV_FLAG="--gpus=all"

Cannot generate samples data from Docker Isaac Object Detection Training
#7

This actually solved my problem. my docker version was 20 as well
I checked it by

docker version

Thank you.

#8

I had made the script change. Running it seems that it’s stuck - I saw roughly 30 rows of:

Generated 0 samples

over a few minutes

before it started making progress. Then it worked to completion.