Please provide the following information when requesting support.
• Hardware (T4/V100/Xavier/Nano/etc) T4
• Network Type (Detectnet_v2/Faster_rcnn/Yolo_v4/LPRnet/Mask_rcnn/Classification/etc) Detectnet_v2
• TLT Version (Please run “tlt info --verbose” and share “docker_tag” here)
• Training spec file(If have, please share here)detectnet_v2.ipynb
• How to reproduce the issue ? (This is for errors. Please share the command line and the detailed log here.)
When I used the detectnet_v2 algorithm to train my kitti format data set, I replaced the pretrained_resnet18 model with the dashcamnet model. My own data set categories are pedestrian and car. The following results appeared during training. There is only the accuracy of the pedestrian category, and the accuracy results of the other two categories are missing.
class name average precision (in %)
------------ --------------------------
car 0
cyclist 0
pedestrian 12.7891
Here are the detailed results.Please help me.
/usr/local/lib/python3.6/dist-packages/keras/engine/saving.py:292: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
warnings.warn('No training configuration found in save file: '
_________________________________________________________
2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 48, io threads: 96, compute threads: 48, buffered batches: 4
2021-12-28 07:30:06,779 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 114, number of sources: 1, batch size per gpu: 4, steps: 29
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
2021-12-28 07:30:06,831 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/tensorflow_core/python/autograph/converters/directives.py:119: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.
WARNING:tensorflow:Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:06,893 [WARNING] tensorflow: Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30fd0>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:06,922 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2021-12-28 07:30:07,230 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: True - shard 0 of 1
2021-12-28 07:30:07,238 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2021-12-28 07:30:07,238 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
WARNING:tensorflow:Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:07,259 [WARNING] tensorflow: Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09b02cfc50>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:07,692 [INFO] __main__: Found 114 samples in training set
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.
2021-12-28 07:30:07,832 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/rasterizers/bbox_rasterizer.py:347: The name tf.bincount is deprecated. Please use tf.math.bincount instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
2021-12-28 07:30:07,981 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:89: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
2021-12-28 07:30:08,000 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/training_proto_utilities.py:36: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.
2021-12-28 07:30:08,214 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_functions.py:17: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
2021-12-28 07:30:08,262 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/cost_function/cost_auto_weight_hook.py:235: The name tf.assign_add is deprecated. Please use tf.compat.v1.assign_add instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
2021-12-28 07:30:08,273 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/model/detectnet_model.py:587: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.
2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Serial augmentation enabled = False
2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Pseudo sharding enabled = False
2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: Max Image Dimensions (all sources): (0, 0)
2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: number of cpus: 48, io threads: 96, compute threads: 48, buffered batches: 4
2021-12-28 07:30:10,495 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: total dataset size 18, number of sources: 1, batch size per gpu: 4, steps: 5
WARNING:tensorflow:Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:10,512 [WARNING] tensorflow: Entity <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method DriveNetTFRecordsParser.__call__ of <iva.detectnet_v2.dataloader.drivenet_dataloader.DriveNetTFRecordsParser object at 0x7f09cbe30828>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:10,542 [INFO] iva.detectnet_v2.dataloader.default_dataloader: Bounding box coordinates were detected in the input specification! Bboxes will be automatically converted to polygon coordinates.
2021-12-28 07:30:10,823 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: shuffle: False - shard 0 of 1
2021-12-28 07:30:10,829 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: sampling 1 datasets with weights:
2021-12-28 07:30:10,829 [INFO] modulus.blocks.data_loaders.multi_source_loader.data_loader: source: 0 weight: 1.000000
WARNING:tensorflow:Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:10,845 [WARNING] tensorflow: Entity <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>> could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output. Cause: Unable to locate the source code of <bound method Processor.__call__ of <modulus.blocks.data_loaders.multi_source_loader.processors.asset_loader.AssetLoader object at 0x7f09583e23c8>>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code
2021-12-28 07:30:11,125 [INFO] __main__: Found 18 samples in validation set
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead.
2021-12-28 07:30:11,861 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/validation_hook.py:40: The name tf.summary.FileWriterCache is deprecated. Please use tf.compat.v1.summary.FileWriterCache instead.
2021-12-28 07:30:13,292 [INFO] __main__: Checkpoint interval: 10
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.
2021-12-28 07:30:13,293 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/scripts/train.py:108: The name tf.train.Scaffold is deprecated. Please use tf.compat.v1.train.Scaffold instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.
2021-12-28 07:30:13,293 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:14: The name tf.local_variables_initializer is deprecated. Please use tf.compat.v1.local_variables_initializer instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.
2021-12-28 07:30:13,294 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:15: The name tf.tables_initializer is deprecated. Please use tf.compat.v1.tables_initializer instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
2021-12-28 07:30:13,295 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/common/graph/initializers.py:16: The name tf.get_collection is deprecated. Please use tf.compat.v1.get_collection instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.
2021-12-28 07:30:13,298 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:59: The name tf.train.LoggingTensorHook is deprecated. Please use tf.estimator.LoggingTensorHook instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.
2021-12-28 07:30:13,298 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:60: The name tf.train.StopAtStepHook is deprecated. Please use tf.estimator.StopAtStepHook instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.
2021-12-28 07:30:13,298 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:73: The name tf.train.StepCounterHook is deprecated. Please use tf.estimator.StepCounterHook instead.
INFO:tensorflow:Create CheckpointSaverHook.
2021-12-28 07:30:13,299 [INFO] tensorflow: Create CheckpointSaverHook.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.
2021-12-28 07:30:13,299 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/tfhooks/utils.py:99: The name tf.train.SummarySaverHook is deprecated. Please use tf.estimator.SummarySaverHook instead.
WARNING:tensorflow:From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.
2021-12-28 07:30:13,300 [WARNING] tensorflow: From /root/.cache/bazel/_bazel_root/ed34e6d125608f91724fda23656f1726/execroot/ai_infra/bazel-out/k8-fastbuild/bin/magnet/packages/iva/build_wheel.runfiles/ai_infra/iva/detectnet_v2/training/utilities.py:140: The name tf.train.SingularMonitoredSession is deprecated. Please use tf.compat.v1.train.SingularMonitoredSession instead.
INFO:tensorflow:Graph was finalized.
2021-12-28 07:30:14,536 [INFO] tensorflow: Graph was finalized.
INFO:tensorflow:Running local_init_op.
2021-12-28 07:30:16,823 [INFO] tensorflow: Running local_init_op.
INFO:tensorflow:Done running local_init_op.
2021-12-28 07:30:17,613 [INFO] tensorflow: Done running local_init_op.
INFO:tensorflow:Saving checkpoints for step-0.
2021-12-28 07:30:26,523 [INFO] tensorflow: Saving checkpoints for step-0.
INFO:tensorflow:epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.09131467, step = 0
2021-12-28 07:30:57,713 [INFO] tensorflow: epoch = 0.0, learning_rate = 4.9999994e-06, loss = 0.09131467, step = 0
2021-12-28 07:30:57,722 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 0/120: loss: 0.09131 learning rate: 0.00000 Time taken: 0:00:00 ETA: 0:00:00
2021-12-28 07:30:57,722 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 0.439
INFO:tensorflow:global_step/sec: 0.772853
2021-12-28 07:31:00,306 [INFO] tensorflow: global_step/sec: 0.772853
INFO:tensorflow:global_step/sec: 10.915
2021-12-28 07:31:00,489 [INFO] tensorflow: global_step/sec: 10.915
INFO:tensorflow:global_step/sec: 10.8179
2021-12-28 07:31:00,674 [INFO] tensorflow: global_step/sec: 10.8179
INFO:tensorflow:global_step/sec: 10.6503
2021-12-28 07:31:00,862 [INFO] tensorflow: global_step/sec: 10.6503
INFO:tensorflow:global_step/sec: 11.1249
2021-12-28 07:31:01,042 [INFO] tensorflow: global_step/sec: 11.1249
INFO:tensorflow:global_step/sec: 11.0209
2021-12-28 07:31:01,223 [INFO] tensorflow: global_step/sec: 11.0209
INFO:tensorflow:global_step/sec: 10.5862
2021-12-28 07:31:01,412 [INFO] tensorflow: global_step/sec: 10.5862
INFO:tensorflow:global_step/sec: 10.0341
2021-12-28 07:31:01,611 [INFO] tensorflow: global_step/sec: 10.0341
INFO:tensorflow:global_step/sec: 11.1387
2021-12-28 07:31:01,791 [INFO] tensorflow: global_step/sec: 11.1387
INFO:tensorflow:global_step/sec: 9.6168
2021-12-28 07:31:01,999 [INFO] tensorflow: global_step/sec: 9.6168
INFO:tensorflow:global_step/sec: 9.13294
2021-12-28 07:31:02,218 [INFO] tensorflow: global_step/sec: 9.13294
INFO:tensorflow:global_step/sec: 10.1474
2021-12-28 07:31:02,415 [INFO] tensorflow: global_step/sec: 10.1474
2021-12-28 07:31:02,416 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 7.246
INFO:tensorflow:global_step/sec: 10.1296
2021-12-28 07:31:02,612 [INFO] tensorflow: global_step/sec: 10.1296
INFO:tensorflow:epoch = 0.9655172413793103, learning_rate = 7.242517e-06, loss = 0.06743895, step = 28 (5.111 sec)
2021-12-28 07:31:02,824 [INFO] tensorflow: epoch = 0.9655172413793103, learning_rate = 7.242517e-06, loss = 0.06743895, step = 28 (5.111 sec)
INFO:tensorflow:global_step/sec: 9.23476
2021-12-28 07:31:02,829 [INFO] tensorflow: global_step/sec: 9.23476
49da551de758:116:224 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.7<0>
49da551de758:116:224 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
49da551de758:116:224 [0] NCCL INFO NET/IB : No device found.
49da551de758:116:224 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1<0> [1]eth0:172.17.0.7<0>
49da551de758:116:224 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda11.1
49da551de758:116:224 [0] NCCL INFO Channel 00/32 : 0
49da551de758:116:224 [0] NCCL INFO Channel 01/32 : 0
49da551de758:116:224 [0] NCCL INFO Channel 02/32 : 0
....
Median Inference Time: 0.008003
INFO:tensorflow:epoch = 110.0, learning_rate = 1.7969065e-05, loss = 8.230345e-05, step = 3190 (9.965 sec)
2021-12-28 07:37:44,296 [INFO] tensorflow: epoch = 110.0, learning_rate = 1.7969065e-05, loss = 8.230345e-05, step = 3190 (9.965 sec)
INFO:tensorflow:global_step/sec: 0.216576
2021-12-28 07:37:44,298 [INFO] tensorflow: global_step/sec: 0.216576
2021-12-28 07:37:44,302 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 110/120: loss: 0.00008 learning rate: 0.00002 Time taken: 0:00:11.694104 ETA: 0:01:56.941044
INFO:tensorflow:global_step/sec: 10.2147
....
INFO:tensorflow:global_step/sec: 10.9742
2021-12-28 07:38:07,158 [INFO] tensorflow: global_step/sec: 10.9742
INFO:tensorflow:global_step/sec: 11.0977
2021-12-28 07:38:07,339 [INFO] tensorflow: global_step/sec: 11.0977
INFO:tensorflow:global_step/sec: 10.3461
2021-12-28 07:38:07,532 [INFO] tensorflow: global_step/sec: 10.3461
INFO:tensorflow:global_step/sec: 11.9319
2021-12-28 07:38:07,700 [INFO] tensorflow: global_step/sec: 11.9319
INFO:tensorflow:global_step/sec: 10.8791
2021-12-28 07:38:07,883 [INFO] tensorflow: global_step/sec: 10.8791
2021-12-28 07:38:07,975 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 43.965
INFO:tensorflow:global_step/sec: 10.9462
2021-12-28 07:38:08,066 [INFO] tensorflow: global_step/sec: 10.9462
2021-12-28 07:38:08,166 [INFO] iva.detectnet_v2.tfhooks.task_progress_monitor_hook: Epoch 119/120: loss: 0.00007 learning rate: 0.00001 Time taken: 0:00:02.648905 ETA: 0:00:02.648905
INFO:tensorflow:global_step/sec: 10.6033
2021-12-28 07:38:08,255 [INFO] tensorflow: global_step/sec: 10.6033
INFO:tensorflow:global_step/sec: 10.9142
2021-12-28 07:38:08,438 [INFO] tensorflow: global_step/sec: 10.9142
INFO:tensorflow:global_step/sec: 10.5721
2021-12-28 07:38:08,627 [INFO] tensorflow: global_step/sec: 10.5721
INFO:tensorflow:global_step/sec: 11.1025
2021-12-28 07:38:08,807 [INFO] tensorflow: global_step/sec: 11.1025
INFO:tensorflow:global_step/sec: 11.317
2021-12-28 07:38:08,984 [INFO] tensorflow: global_step/sec: 11.317
INFO:tensorflow:global_step/sec: 10.2896
2021-12-28 07:38:09,178 [INFO] tensorflow: global_step/sec: 10.2896
INFO:tensorflow:global_step/sec: 10.7394
2021-12-28 07:38:09,365 [INFO] tensorflow: global_step/sec: 10.7394
INFO:tensorflow:global_step/sec: 11.306
2021-12-28 07:38:09,542 [INFO] tensorflow: global_step/sec: 11.306
INFO:tensorflow:global_step/sec: 10.4033
2021-12-28 07:38:09,734 [INFO] tensorflow: global_step/sec: 10.4033
INFO:tensorflow:epoch = 119.6551724137931, learning_rate = 5.225487e-06, loss = 0.00011281592, step = 3470 (5.146 sec)
2021-12-28 07:38:09,919 [INFO] tensorflow: epoch = 119.6551724137931, learning_rate = 5.225487e-06, loss = 0.00011281592, step = 3470 (5.146 sec)
INFO:tensorflow:global_step/sec: 10.5473
2021-12-28 07:38:09,924 [INFO] tensorflow: global_step/sec: 10.5473
INFO:tensorflow:global_step/sec: 11.345
2021-12-28 07:38:10,100 [INFO] tensorflow: global_step/sec: 11.345
INFO:tensorflow:global_step/sec: 11.0311
2021-12-28 07:38:10,281 [INFO] tensorflow: global_step/sec: 11.0311
2021-12-28 07:38:10,284 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 43.326
INFO:tensorflow:global_step/sec: 11.2454
2021-12-28 07:38:10,459 [INFO] tensorflow: global_step/sec: 11.2454
INFO:tensorflow:global_step/sec: 11.3238
2021-12-28 07:38:10,636 [INFO] tensorflow: global_step/sec: 11.3238
INFO:tensorflow:Saving checkpoints for step-3480.
2021-12-28 07:38:10,736 [INFO] tensorflow: Saving checkpoints for step-3480.
WARNING:tensorflow:Ignoring: /tmp/tmpi08w9yct; No such file or directory
2021-12-28 07:38:10,979 [WARNING] tensorflow: Ignoring: /tmp/tmpi08w9yct; No such file or directory
2021-12-28 07:38:15,057 [INFO] iva.detectnet_v2.evaluation.evaluation: step 0 / 4, 0.00s/step
Matching predictions to ground truth, class 1/3.: 100%|█| 3282/3282 [00:00<00:00, 24675.21it/s]
Matching predictions to ground truth, class 3/3.: 100%|█| 1341/1341 [00:00<00:00, 15211.67it/s]
Epoch 120/120
=========================
Validation cost: 0.000036
Mean average_precision (in %): 4.2630
class name average precision (in %)
------------ --------------------------
car 0
cyclist 0
pedestrian 12.7891
Median Inference Time: 0.007898
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.
2021-12-28 07:38:20,386 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:95: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.
WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
2021-12-28 07:38:20,386 [WARNING] tensorflow: From /usr/local/lib/python3.6/dist-packages/keras/backend/tensorflow_backend.py:98: The name tf.placeholder_with_default is deprecated. Please use tf.compat.v1.placeholder_with_default instead.
2021-12-28 07:38:20,390 [INFO] modulus.hooks.sample_counter_hook: Train Samples / sec: 43.326
Time taken to run __main__:main: 0:08:27.855154.
2021-12-28 15:38:26,442 [INFO] tlt.components.docker_handler.docker_handler: Stopping container.
