TLT-Train DetectNetv2 ResNet18 always give mAP 0% at target class

Please provide the following information when requesting support.

• Hardware: RTX 2080 SUPER
• Network Type: Detectnet_v2
• TLT Version: TLT2.0 tlt-streamanalytics:v2.0_py3
• Training spec file(If have, please share here): detectnet_v2_train_resnet18_kitti.txt (2.9 KB)

Hello, I am trying custom data to train using detectnet_v2 resnet 18 but it always give 0% mAP. Training specs file is attached for reference.

The output from training log is

> Epoch 50/200
> Validation cost: -0.000010
> Mean average_precision (in %): 0.0000
> 
> class name      average precision (in %)
> ------------  --------------------------
> TARGET                                 0
> 
> Median Inference Time: 0.020825

Please change all “TARGET” to target.

I have changed all “TARGET” to “target” but still gives mAP 0%.

Did you ever keep the full log when you generate the tfrecord files?
If yes, please share here.

Converting Tfrecords for kitti trainval dataset
2022-10-26 06:29:12.620037: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
Using TensorFlow backend.
2022-10-26 06:29:24,381 - iva.detectnet_v2.dataio.build_converter - INFO - Instantiating a kitti converter
2022-10-26 06:29:24,393 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Num images in
Train: 1836	Val: 203
2022-10-26 06:29:24,393 - iva.detectnet_v2.dataio.kitti_converter_lib - INFO - Validation data in partition 0. Hence, while choosing the validationset during training choose validation_fold 0.
2022-10-26 06:29:24,395 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 0
WARNING:tensorflow:From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

2022-10-26 06:29:24,395 - tensorflow - WARNING - From /home/vpraveen/.cache/dazel/_dazel_vpraveen/715c8bafe7816f3bb6f309cd506049bb/execroot/ai_infra/bazel-out/k8-py3-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/dataio/dataset_converter_lib.py:142: The name tf.python_io.TFRecordWriter is deprecated. Please use tf.io.TFRecordWriter instead.

/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: VisibleDeprecationWarning: Reading unicode strings without specifying the encoding argument is deprecated. Set the encoding, use None for the system default.
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66592.txt"
2022-10-26 06:29:24,830 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 1
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66500.txt"
2022-10-26 06:29:25,173 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 2
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7970.txt"
2022-10-26 06:29:25,420 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 3
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame8000.txt"
2022-10-26 06:29:25,780 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 4
2022-10-26 06:29:26,062 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 5
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11960.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66400.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65084.txt"
2022-10-26 06:29:26,312 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 6
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65040.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame122.txt"
2022-10-26 06:29:26,522 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 7
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame24022.txt"
2022-10-26 06:29:26,769 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 8
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3628.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23666.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7606.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19482.txt"
2022-10-26 06:29:27,067 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 0, shard 9
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23676.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1764.txt"
2022-10-26 06:29:27,399 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - 
Wrote the following numbers of objects:
b'target': 187

2022-10-26 06:29:27,399 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 0
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame18276.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23686.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame12630.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66394.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1628.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame112.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2688.txt"
2022-10-26 06:29:30,221 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 1
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66572.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66612.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66602.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame126.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7990.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame4086.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65500.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66510.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame116.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66610.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1766.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2716.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11930.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74940.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74960.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65050.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2690.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame26282.txt"
2022-10-26 06:29:32,734 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 2
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame24012.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55490.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame26272.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66404.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66530.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7596.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13316.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7960.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65104.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15326.txt"
2022-10-26 06:29:35,448 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 3
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7824.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15306.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3832.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame118.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11940.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame2718.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66580.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame18256.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame394.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74932.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6688.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19492.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66582.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13306.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3626.txt"
2022-10-26 06:29:38,279 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 4
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65124.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3622.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66590.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1632.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65020.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6128.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7980.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame56182.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6690.txt"
2022-10-26 06:29:40,840 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 5
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame12640.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66490.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65510.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74942.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66390.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1762.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65030.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66402.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65094.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3834.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66600.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1626.txt"
2022-10-26 06:29:43,449 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 6
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame56180.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13336.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13346.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66520.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3624.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6698.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame4084.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame4088.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65520.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame7816.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74950.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame23656.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1624.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66570.txt"
2022-10-26 06:29:46,430 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 7
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1748.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55480.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6700.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame114.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3862.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame124.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74962.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame120.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame56178.txt"
2022-10-26 06:29:49,496 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 8
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19502.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15336.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame11950.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55470.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74930.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame12620.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame66392.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65530.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6100.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame19472.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame1630.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame26292.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame21594.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame65114.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame3860.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame6072.txt"
2022-10-26 06:29:52,304 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Writing partition 1, shard 9
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame15316.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame130.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame55500.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame13326.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame74952.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame18266.txt"
/usr/local/lib/python3.6/dist-packages/iva/detectnet_v2/dataio/kitti_converter_lib.py:273: UserWarning: genfromtxt: Empty input file: "/workspace/tlt-experiments/data/training/label_2/frame128.txt"
2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - 
Wrote the following numbers of objects:
b'target': 1719

2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Cumulative object statistics
2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - 
Wrote the following numbers of objects:
b'target': 1906

2022-10-26 06:29:55,340 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Class map. 
Label in GT: Label in tfrecords file 
b'target': b'target'
For the dataset_config in the experiment_spec, please use labels in the tfrecords file, while writing the classmap.

2022-10-26 06:29:55,341 - iva.detectnet_v2.dataio.dataset_converter_lib - INFO - Tfrecords generation complete.

Could you attach your latest spec file?

detectnet_v2_tfrecords_kitti_trainval.txt

kitti_config {
  root_directory_path: "/workspace/tlt-experiments/data/training"
  image_dir_name: "image_2"
  label_dir_name: "label_2"
  image_extension: ".jpg"
  partition_mode: "random"
  num_partitions: 2
  val_split: 10
  num_shards: 10
}
image_directory_path: "/workspace/tlt-experiments/data/training"

detectnet_v2_train_resnet18_kitti.txt

random_seed: 42
dataset_config {
  data_sources {
    tfrecords_path: "/workspace/tlt-experiments/data/tfrecords/kitti_trainval/*"
    image_directory_path: "/workspace/tlt-experiments/data/training"
  }
  image_extension: "jpg"
  target_class_mapping {
    key: "target"
    value: "target"
  }
  validation_fold: 0
}
augmentation_config {
  preprocessing {
    output_image_width: 1280
    output_image_height: 720
    min_bbox_width: 1.0
    min_bbox_height: 1.0
    output_image_channel: 3
  }
  spatial_augmentation {
    hflip_probability: 0.5
    zoom_min: 1.0
    zoom_max: 1.0
    translate_max_x: 8.0
    translate_max_y: 8.0
  }
  color_augmentation {
    hue_rotation_max: 25.0
    saturation_shift_max: 0.20000000298023224
    contrast_scale_max: 0.10000000149011612
    contrast_center: 0.5
  }
}
postprocessing_config {
  target_class_config {
    key: "target"
    value {
      clustering_config {
        coverage_threshold: 0.004999999888241291
        dbscan_eps: 0.15000000596046448
        dbscan_min_samples: 0.05000000074505806
        minimum_bounding_box_height: 10
      }
    }
  }
}
model_config {
  pretrained_model_file: "/workspace/tlt-experiments/detectnet_v2/pretrained_resnet18/tlt_pretrained_detectnet_v2_vresnet18/resnet18.hdf5"
  num_layers: 18
  use_batch_norm: true
  objective_set {
    bbox {
      scale: 35.0
      offset: 0.5
    }
    cov {
    }
  }
  training_precision {
    backend_floatx: FLOAT32
  }
  arch: "resnet"
}
evaluation_config {
  validation_period_during_training: 10
  first_validation_epoch: 10
  minimum_detection_ground_truth_overlap {
    key: "target"
    value: 0.699999988079071
  }
  evaluation_box_config {
    key: "target"
    value {
      minimum_height: 10
      maximum_height: 9999
      minimum_width: 10
      maximum_width: 9999
    }
  }
  average_precision_mode: INTEGRATE
}
cost_function_config {
  target_classes {
    name: "target"
    class_weight: 1.0
    coverage_foreground_weight: 0.05000000074505806
    objectives {
      name: "cov"
      initial_weight: 1.0
      weight_target: 1.0
    }
    objectives {
      name: "bbox"
      initial_weight: 10.0
      weight_target: 10.0
    }
  }
  enable_autoweighting: true
  max_objective_weight: 0.9998999834060669
  min_objective_weight: 9.999999747378752e-05
}
training_config {
  batch_size_per_gpu: 1
  num_epochs: 200
  learning_rate {
    soft_start_annealing_schedule {
      min_learning_rate: 4.999999873689376e-06
      max_learning_rate: 0.0005000000237487257
      soft_start: 0.10000000149011612
      annealing: 0.699999988079071
    }
  }
  regularizer {
    type: L1
    weight: 3.000000026176508e-09
  }
  optimizer {
    adam {
      epsilon: 9.99999993922529e-09
      beta1: 0.8999999761581421
      beta2: 0.9990000128746033
    }
  }
  cost_scaling {
    initial_exponent: 20.0
    increment: 0.005
    decrement: 1.0
  }
  checkpoint_interval: 10
}
bbox_rasterizer_config {
  target_class_config {
    key: "target"
    value {
      cov_center_x: 0.5
      cov_center_y: 0.5
      cov_radius_x: 0.4000000059604645
      cov_radius_y: 0.4000000059604645
      bbox_min_radius: 1.0
    }
  }
  deadzone_radius: 0.4000001549720764
}

I find that you are still using an old version of TLT(currently, it is named TAO).
Any reason why you do not use the latest TAO?

May I know if your training images are all the same resolutions?

As i have mentioned before, the hardware i am using is RTX 2080 SUPER which is not listed in the supported hardware on TOA Toolkit documentation. (TAO Toolkit Quick Start Guide — TAO Toolkit 3.22.05 documentation)

All images used are of same resolution 1280 x 720.

RTX2080 can be used. The tao user guide just mentions several examples of GPUs but it does not mean other GPUs cannot be used.

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Can you share an example of the label file?
More, is the target object too small? How about its average size?

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.