TrafficCamNet inference error

[user150815]

Please provide the following information when requesting support.

• Hardware (GTX 1060 6GB)
• Network Type (TrafficCamNet)
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(
detectnet_v2_inference_kitti_tlt.txt (2.3 KB)
)
• How to reproduce the issue ? My pipeline
my.ipynb (15.9 KB)

I took the pre-trained TrafficCamNet and want to run inference using

!tao detectnet_v2 inference -e $SPECS_DIR/detectnet_v2_inference_kitti_tlt.txt \
                            -o $USER_EXPERIMENT_DIR/tlt_infer_testing \
                            -i $DATA_DOWNLOAD_DIR/data \
                            -k $KEY

I have structure:

where
TrafficCamNet is LOCAL_PROJECT_DIR
TrafficCamNet/data is LOCAL_DATA_DIR
TrafficCamNet/specs is LOCAL_SPECS_DIR
TrafficCamNet/tlt_trafficcamnet_vunpruned_v1.0 is downloaded local path for pre-train TrafficCamNet

And i get this error:

/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_1 (InputLayer)         (None, 3, 384, 1248)      0         
_________________________________________________________________
model_1 (Model)              multiple                  11558548  
=================================================================
Total params: 11,558,548
Trainable params: 11,546,900
Non-trainable params: 11,648
_________________________________________________________________
2022-01-25 16:40:18,236 [INFO] __main__: Initialized model
2022-01-25 16:40:18,239 [INFO] __main__: Commencing inference
  0%|                                                    | 0/94 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/inference.py", line 210, in <module>
  File "/opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/inference.py", line 206, in main
  File "/opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/inference.py", line 159, in inference_wrapper_batch
  File "/opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/postprocessor/bbox_handler.py", line 245, in bbox_preprocessing
  File "/opt/tlt/.cache/dazel/_dazel_tlt/75913d2aee35770fa76c4a63d877f3aa/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/postprocessor/bbox_handler.py", line 271, in abs_bbox_converter
  File "/usr/local/lib/python3.6/dist-packages/addict/addict.py", line 64, in __getitem__
    if name not in self:
TypeError: unhashable type: 'slice'

2022-01-25 18:40:24,460 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

I think I made a lot of mistakes. I will be grateful for any help

[Morganh]

Did you generate tfrecords for your test set?

[user150815]

Not sure about this. How can i do this?

[Morganh]

Ignore above way.
Please check another culprit.

Please run below and share result.
tao detectnet_v2 run ls -rltsh $DATA_DOWNLOAD_DIR/data

[user150815]

my output:

2022-01-26 12:59:14,487 [INFO] root: Registry: ['nvcr.io']
2022-01-26 12:59:14,536 [INFO] tlt.components.instance_handler.local_instance: Running command in container: nvcr.io/nvidia/tao/tao-toolkit-tf:v3.21.11-tf1.15.4-py3
2022-01-26 12:59:14,589 [WARNING] tlt.components.docker_handler.docker_handler: 
Docker will run the commands as root. If you would like to retain your
local host permissions, please add the "user":"UID:GID" in the
DockerOptions portion of the "/home/NIX/dronov/.tao_mounts.json" file. You can obtain your
users UID and GID by using the "id -u" and "id -g" commands on the
terminal.
total 510M
 472K -rw-rw-r-- 1 1001 1001  472K Jan 21 16:39 95ca4b1736e2ba52.jpg
 188K -rw-rw-r-- 1 1001 1001  188K Jan 21 16:39 f89e86b54b69518a.jpg
 216K -rw-rw-r-- 1 1001 1001  214K Jan 21 16:39 4f7c4926f12a2f2f.jpg
 396K -rw-rw-r-- 1 1001 1001  393K Jan 21 16:39 56cda7ee10a0a298.jpg
 260K -rw-rw-r-- 1 1001 1001  259K Jan 21 16:39 3bbdab1c07455377.jpg
 376K -rw-rw-r-- 1 1001 1001  376K Jan 21 16:39 bdccdd859d99f6ae.jpg
 608K -rw-rw-r-- 1 1001 1001  608K Jan 21 16:39 807d04f0cc56b1d7.jpg
 320K -rw-rw-r-- 1 1001 1001  320K Jan 21 16:39 5707a4ab4a9f60d6.jpg
 220K -rw-rw-r-- 1 1001 1001  220K Jan 21 16:39 bfb8cae92bdb2304.jpg
 196K -rw-rw-r-- 1 1001 1001  194K Jan 21 16:39 2c7412840229a88c.jpg
 132K -rw-rw-r-- 1 1001 1001  131K Jan 21 16:39 28af01e0b7f49943.jpg
  80K -rw-rw-r-- 1 1001 1001   79K Jan 21 16:39 4a00804fdf4cfe65.jpg
 580K -rw-rw-r-- 1 1001 1001  580K Jan 21 16:39 05576e20f464534c.jpg
 196K -rw-rw-r-- 1 1001 1001  193K Jan 21 16:39 06eb204b70032b18.jpg
 996K -rw-rw-r-- 1 1001 1001  995K Jan 21 16:39 ea240e927bc85e6a.jpg
 964K -rw-rw-r-- 1 1001 1001  963K Jan 21 16:39 b011f185cf5a40d5.jpg
 204K -rw-rw-r-- 1 1001 1001  201K Jan 21 16:39 c50e43b2b9c50bd4.jpg
  92K -rw-rw-r-- 1 1001 1001   92K Jan 21 16:39 a57d500ff5355b1f.jpg
 196K -rw-rw-r-- 1 1001 1001  196K Jan 21 16:39 26a20bc8abeb09f8.jpg
 352K -rw-rw-r-- 1 1001 1001  350K Jan 21 16:39 2f4741c196f34b59.jpg
 484K -rw-rw-r-- 1 1001 1001  482K Jan 21 16:39 76bb1ab71edaa07b.jpg
 888K -rw-rw-r-- 1 1001 1001  886K Jan 21 16:39 bdd868fdeb92922b.jpg
 136K -rw-rw-r-- 1 1001 1001  134K Jan 21 16:39 a5ca2a0005ff6af6.jpg
 248K -rw-rw-r-- 1 1001 1001  245K Jan 21 16:39 7e12345a39e7b07d.jpg
 292K -rw-rw-r-- 1 1001 1001  292K Jan 21 16:39 c5f7a81b3a553ab6.jpg
 248K -rw-rw-r-- 1 1001 1001  245K Jan 21 16:39 76af98a3b5040a14.jpg
 504K -rw-rw-r-- 1 1001 1001  503K Jan 21 16:39 b9fa4b6fd11c820a.jpg
 280K -rw-rw-r-- 1 1001 1001  278K Jan 21 16:39 71c0ce2682d02343.jpg
 276K -rw-rw-r-- 1 1001 1001  273K Jan 21 16:39 04698ba478065006.jpg
 144K -rw-rw-r-- 1 1001 1001  144K Jan 21 16:39 850e36a383ef0700.jpg
  36K -rw-rw-r-- 1 1001 1001   34K Jan 21 16:39 b7e4836997c8955f.jpg
 836K -rw-rw-r-- 1 1001 1001  833K Jan 21 16:39 90b4068959e4edf8.jpg
 140K -rw-rw-r-- 1 1001 1001  138K Jan 21 16:39 e86e8de83f1c1785.jpg
 424K -rw-rw-r-- 1 1001 1001  423K Jan 21 16:39 55adfbf18fd2a1d9.jpg
 720K -rw-rw-r-- 1 1001 1001  720K Jan 21 16:39 0c004fa1557a1d49.jpg
 112K -rw-rw-r-- 1 1001 1001  111K Jan 21 16:39 edd823f2e065a36e.jpg
 696K -rw-rw-r-- 1 1001 1001  695K Jan 21 16:39 906ccf5312abe6b0.jpg
 860K -rw-rw-r-- 1 1001 1001  858K Jan 21 16:39 7e3c2d3b43224318.jpg
 104K -rw-rw-r-- 1 1001 1001  101K Jan 21 16:39 48a2b3babd24a6e6.jpg
 252K -rw-rw-r-- 1 1001 1001  249K Jan 21 16:39 730fe880f8354a62.jpg
 464K -rw-rw-r-- 1 1001 1001  461K Jan 21 16:39 f4fc4b8a9818b7b8.jpg
 136K -rw-rw-r-- 1 1001 1001  133K Jan 21 16:39 a9396bb1a3330d8c.jpg
 356K -rw-rw-r-- 1 1001 1001  353K Jan 21 16:39 9f4703190706c5b6.jpg
  92K -rw-rw-r-- 1 1001 1001   89K Jan 21 16:39 d3c8ddfaffa6216a.jpg
 824K -rw-rw-r-- 1 1001 1001  821K Jan 21 16:39 ce388037d85ec521.jpg
 376K -rw-rw-r-- 1 1001 1001  373K Jan 21 16:39 e4fb88f13e63e1d8.jpg
 444K -rw-rw-r-- 1 1001 1001  442K Jan 21 16:39 f6c7e48bea0e5980.jpg
 616K -rw-rw-r-- 1 1001 1001  613K Jan 21 16:39 c00c38463e106074.jpg
 336K -rw-rw-r-- 1 1001 1001  336K Jan 21 16:39 f6ee63ee4e378fdf.jpg
 404K -rw-rw-r-- 1 1001 1001  402K Jan 21 16:39 b0f803f7dcd6a935.jpg
 108K -rw-rw-r-- 1 1001 1001  107K Jan 21 16:39 f605f073c52b6d5f.jpg
 372K -rw-rw-r-- 1 1001 1001  372K Jan 21 16:39 985cd6e06088350a.jpg
 108K -rw-rw-r-- 1 1001 1001  106K Jan 21 16:39 dd38525f62671b18.jpg
 268K -rw-rw-r-- 1 1001 1001  268K Jan 21 16:39 7cf44c8c0513bc50.jpg
 536K -rw-rw-r-- 1 1001 1001  533K Jan 21 16:39 082954914d9ed6f6.jpg
 252K -rw-rw-r-- 1 1001 1001  252K Jan 21 16:39 bb46793a36f90450.jpg
 456K -rw-rw-r-- 1 1001 1001  454K Jan 21 16:39 93fede79cc49aeb6.jpg
 388K -rw-rw-r-- 1 1001 1001  388K Jan 21 16:39 2205e8865a2394b9.jpg
 216K -rw-rw-r-- 1 1001 1001  213K Jan 21 16:39 c0a80c09d471c0fe.jpg
 108K -rw-rw-r-- 1 1001 1001  106K Jan 21 16:39 60c1248109e5ac8d.jpg
 136K -rw-rw-r-- 1 1001 1001  135K Jan 21 16:39 71a2695cc86deb7d.jpg
 688K -rw-rw-r-- 1 1001 1001  686K Jan 21 16:39 1e99443397799e2e.jpg
 616K -rw-rw-r-- 1 1001 1001  615K Jan 21 16:39 3d024b1573083ec4.jpg
 444K -rw-rw-r-- 1 1001 1001  441K Jan 21 16:39 c690abe5c7e842b8.jpg
 316K -rw-rw-r-- 1 1001 1001  313K Jan 21 16:39 cbec1d66c947bf99.jpg
 364K -rw-rw-r-- 1 1001 1001  363K Jan 21 16:39 8a522694b1a77d11.jpg
 412K -rw-rw-r-- 1 1001 1001  410K Jan 21 16:39 a1a26eaf3e17fa9a.jpg
 976K -rw-rw-r-- 1 1001 1001  974K Jan 21 16:39 701fbc8f0ea714f5.jpg
 788K -rw-rw-r-- 1 1001 1001  787K Jan 21 16:39 9daa2f492531c146.jpg
 464K -rw-rw-r-- 1 1001 1001  464K Jan 21 16:39 c516945d52e85605.jpg
 132K -rw-rw-r-- 1 1001 1001  130K Jan 21 16:39 ddadb1b2d24ca577.jpg
 220K -rw-rw-r-- 1 1001 1001  219K Jan 21 16:39 2afc9e7fa5601e2f.jpg
 128K -rw-rw-r-- 1 1001 1001  127K Jan 21 16:39 3c0a2020db4e0365.jpg
 156K -rw-rw-r-- 1 1001 1001  155K Jan 21 16:39 329dbe0e5a23a5ec.jpg
 228K -rw-rw-r-- 1 1001 1001  226K Jan 21 16:39 48b2498103345f21.jpg
 148K -rw-rw-r-- 1 1001 1001  145K Jan 21 16:39 b9d427c95bae8f9d.jpg
...
916K -rw-rw-r-- 1 1001 1001  914K Jan 21 16:40 322663588a415c05.jpg
 360K -rw-rw-r-- 1 1001 1001  360K Jan 21 16:40 edb0e042c1a8b0ba.jpg
 120K -rw-rw-r-- 1 1001 1001  119K Jan 21 16:40 c9be0d5bfbb54fb5.jpg
 296K -rw-rw-r-- 1 1001 1001  296K Jan 21 16:40 072872a2c1cc5b8c.jpg
 372K -rw-rw-r-- 1 1001 1001  370K Jan 21 16:40 d316c04fb7d4880a.jpg

2022-01-26 12:59:15,059 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.

[Morganh]

Can you share your training spec?
I remember that you only train for only one class “car”,right?
If yes, need to modify detectnet_v2_inference_kitti_tlt.txt accordingly.

[user150815]

I didn’t train the model, I downloaded the pretrained one TrafficCamNet | NVIDIA NGC

[Morganh]

OK, I see.
For TrafficCamNet | NVIDIA NGC , there are four classes.
Car
Bicycle
Person
Roadsign

So you need to modify detectnet_v2_inference_kitti_tlt.txt accordingly.

[user150815]

Thank you very much the script really worked, but the results are pretty bad. I think the problem is my bad detectnet_v2_inference_kitti_tlt.txt setting. What do you think?
Is there some kind of manual or document in which it is written about each configuration parameter?

[Morganh]

Can you share latest detectnet_v2_inference_kitti_tlt.txt ?

[Morganh]

For inference spec setting, please refer to DetectNet_v2 — TAO Toolkit 3.22.05 documentation.

[user150815]

detectnet_v2_inference_kitti_tlt.txt (2.4 KB)

[Morganh]

Can you share an example of the inferenced result?

[user150815]

Yes, sure

[Morganh]

Please resize to 960x544 and retry.

[user150815]

I don’t feel much of a difference

[Morganh]

Could you change to a larger threshold ?

[user150815]

Ok, I increased dbscan_confidence_threshold to 0.9 and dbscan_eps to 0.2
But I hoped that results will be better. I believe that it is not the limit.

[Morganh]

For better result, need to retrain your own dataset.

TrafficCamNet v1.0 model was trained on a proprietary dataset with more than 3 million objects for car class. Most of the training dataset was collected and labeled in-house from several traffic cameras in a city in the US. The dataset also contains 40,000 images from a variety of dashcam to help with generalization and discrimination across classes. This content was chosen to improve accuracy of the object detection for images from a traffic cam at a traffic intersection.

[user150815]

Thanks a lot for your help