TLT-Train DetectNetv2 ResNet18 always give mAP 0% at target class

aliqammar92 · October 22, 2022, 5:18pm

Please provide the following information when requesting support.

• Hardware: RTX 2080 SUPER
• Network Type: Detectnet_v2
• TLT Version: TLT2.0 tlt-streamanalytics:v2.0_py3
• Training spec file(If have, please share here): detectnet_v2_train_resnet18_kitti.txt (2.9 KB)

Hello, I am trying custom data to train using detectnet_v2 resnet 18 but it always give 0% mAP. Training specs file is attached for reference.

The output from training log is

> Epoch 50/200
> Validation cost: -0.000010
> Mean average_precision (in %): 0.0000
> 
> class name      average precision (in %)
> ------------  --------------------------
> TARGET                                 0
> 
> Median Inference Time: 0.020825

Morganh · October 22, 2022, 5:32pm

Please change all “TARGET” to target.

aliqammar92 · October 26, 2022, 2:03am

I have changed all “TARGET” to “target” but still gives mAP 0%.

Morganh · October 26, 2022, 2:10am

Did you ever keep the full log when you generate the tfrecord files?
If yes, please share here.

aliqammar92 · October 26, 2022, 6:31am

Converting Tfrecords for kitti trainval dataset
2022-10-26 06:29:12.620037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Using TensorFlow backend.
2022-10-26 06:29:24,381 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2022-10-26 06:29:24,393 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 1836	Val: 203
2022-10-26 06:29:24,393 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-10-26 06:29:24,395 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2022-10-26 06:29:24,395 - tensorflow - WARNING - From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66592.txt"
2022-10-26 06:29:24,830 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66500.txt"
2022-10-26 06:29:25,173 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7970.txt"
2022-10-26 06:29:25,420 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame8000.txt"
2022-10-26 06:29:25,780 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
2022-10-26 06:29:26,062 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11960.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66400.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65084.txt"
2022-10-26 06:29:26,312 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65040.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame122.txt"
2022-10-26 06:29:26,522 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame24022.txt"
2022-10-26 06:29:26,769 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3628.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23666.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7606.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19482.txt"
2022-10-26 06:29:27,067 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23676.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1764.txt"
2022-10-26 06:29:27,399 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - 
Wrote the following numbers of objects:
b'target': 187

2022-10-26 06:29:27,399 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame18276.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23686.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame12630.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66394.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1628.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame112.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2688.txt"
2022-10-26 06:29:30,221 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66572.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66612.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66602.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame126.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7990.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame4086.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65500.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66510.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame116.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66610.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1766.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2716.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11930.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74940.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74960.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65050.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2690.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame26282.txt"
2022-10-26 06:29:32,734 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame24012.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55490.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame26272.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66404.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66530.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7596.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13316.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7960.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65104.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15326.txt"
2022-10-26 06:29:35,448 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7824.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15306.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3832.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame118.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11940.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2718.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66580.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame18256.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame394.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74932.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6688.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19492.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66582.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13306.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3626.txt"
2022-10-26 06:29:38,279 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65124.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3622.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66590.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1632.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65020.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6128.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7980.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame56182.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6690.txt"
2022-10-26 06:29:40,840 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame12640.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66490.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65510.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74942.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66390.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1762.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65030.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66402.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65094.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3834.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66600.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1626.txt"
2022-10-26 06:29:43,449 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame56180.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13336.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13346.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66520.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3624.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6698.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame4084.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame4088.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65520.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7816.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74950.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23656.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1624.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66570.txt"
2022-10-26 06:29:46,430 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1748.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55480.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6700.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame114.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3862.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame124.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74962.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame120.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame56178.txt"
2022-10-26 06:29:49,496 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19502.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15336.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11950.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55470.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74930.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame12620.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66392.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65530.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6100.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19472.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1630.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame26292.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame21594.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65114.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3860.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6072.txt"
2022-10-26 06:29:52,304 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15316.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame130.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55500.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13326.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74952.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame18266.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame128.txt"
2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - 
Wrote the following numbers of objects:
b'target': 1719

2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - 
Wrote the following numbers of objects:
b'target': 1906

2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map. 
Label in GT: Label in tfrecords file 
b'target': b'target'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-10-26 06:29:55,341 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.

Morganh · October 26, 2022, 6:46am

Could you attach your latest spec file?

aliqammar92 · October 26, 2022, 9:20am

detectnet_v2_tfrecords_kitti_trainval.txt

kitti_config {
  root_directory_path: "/workspace/tlt-experiments/data/training"
  image_dir_name: "image_2"
  label_dir_name: "label_2"
  image_extension: ".jpg"
  partition_mode: "random"
  num_partitions: 2
  val_split: 10
  num_shards: 10
}
image_directory_path: "/workspace/tlt-experiments/data/training"

detectnet_v2_train_resnet18_kitti.txt

random_seed: 42
dataset_config {
  data_sources {
    tfrecords_path: "/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*"
    image_directory_path: "/workspace/tlt-experiments/data/training"
  }
  image_extension: "jpg"
  target_class_mapping {
    key: "target"
    value: "target"
  }
  validation_fold: 0
}
augmentation_config {
  preprocessing {
    output_image_width: 1280
    output_image_height: 720
    min_bbox_width: 1.0
    min_bbox_height: 1.0
    output_image_channel: 3
  }
  spatial_augmentation {
    hflip_probability: 0.5
    zoom_min: 1.0
    zoom_max: 1.0
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298023224
    contrast_scale_max: 0.10000000149011612
    contrast_center: 0.5
  }
}
postprocessing_config {
  target_class_config {
    key: "target"
    value {
      clustering_config {
        coverage_threshold: 0.004999999888241291
        dbscan_eps: 0.15000000596046448
        dbscan_min_samples: 0.05000000074505806
        minimum_bounding_box_height: 10
      }
    }
  }
}
model_config {
  pretrained_model_file: "/workspace/tlt-experiments/detectnet_v2/pretrained_resnet18/tlt_pretrained_detectnet_v2_vresnet18/resnet18.hdf5"
  num_layers: 18
  use_batch_norm: true
  objective_set {
    bbox {
      scale: 35.0
      offset: 0.5
    }
    cov {
    }
  }
  training_precision {
    backend_floatx: FLOAT32
  }
  arch: "resnet"
}
evaluation_config {
  validation_period_during_training: 10
  first_validation_epoch: 10
  minimum_detection_ground_truth_overlap {
    key: "target"
    value: 0.699999988079071
  }
  evaluation_box_config {
    key: "target"
    value {
      minimum_height: 10
      maximum_height: 9999
      minimum_width: 10
      maximum_width: 9999
    }
  }
  average_precision_mode: INTEGRATE
}
cost_function_config {
  target_classes {
    name: "target"
    class_weight: 1.0
    coverage_foreground_weight: 0.05000000074505806
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 10.0
    }
  }
  enable_autoweighting: true
  max_objective_weight: 0.9998999834060669
  min_objective_weight: 9.999999747378752e-05
}
training_config {
  batch_size_per_gpu: 1
  num_epochs: 200
  learning_rate {
    soft_start_annealing_schedule {
      min_learning_rate: 4.999999873689376e-06
      max_learning_rate: 0.0005000000237487257
      soft_start: 0.10000000149011612
      annealing: 0.699999988079071
    }
  }
  regularizer {
    type: L1
    weight: 3.000000026176508e-09
  }
  optimizer {
    adam {
      epsilon: 9.99999993922529e-09
      beta1: 0.8999999761581421
      beta2: 0.9990000128746033
    }
  }
  cost_scaling {
    initial_exponent: 20.0
    increment: 0.005
    decrement: 1.0
  }
  checkpoint_interval: 10
}
bbox_rasterizer_config {
  target_class_config {
    key: "target"
    value {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.4000000059604645
      cov_radius_y: 0.4000000059604645
      bbox_min_radius: 1.0
    }
  }
  deadzone_radius: 0.4000001549720764
}

Morganh · October 26, 2022, 9:36am

I find that you are still using an old version of TLT(currently, it is named TAO).
Any reason why you do not use the latest TAO?

May I know if your training images are all the same resolutions?

aliqammar92 · October 26, 2022, 11:48am

As i have mentioned before, the hardware i am using is RTX 2080 SUPER which is not listed in the supported hardware on TOA Toolkit documentation. (TAO Toolkit Quick Start Guide — TAO Toolkit 3.22.05 documentation)

All images used are of same resolution 1280 x 720.

Morganh · October 26, 2022, 2:30pm

RTX2080 can be used. The tao user guide just mentions several examples of GPUs but it does not mean other GPUs cannot be used.

Morganh · October 26, 2022, 2:32pm

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Can you share an example of the label file?
More, is the target object too small? How about its average size?

system · November 9, 2022, 2:33pm

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.