Slow GPU workaround for NHWC error when training

[Morganh]

Hi @digitaldave
Please check your hardware and software meet the requirement mentioned in tlt user guide. https://docs.nvidia.com/metropolis/TLT/tlt-getting-started-guide/text/requirements_and_installation.html#hardware-requirements

For the error, refer to

TLT Detectnet TrafficCamNet training not working

All of my images are in RGB and in the kitti-format as recommended by TLT for training a detectnet on a custom dataset. I used tlt-dataset-convert -d conversion_spec_file_trainval.txt -o /workspace/kitti_data/train/train_tfrecords as instructed here The conversion_spec_file_trainval.txt is : conversion_spec_file_trainval.txt (230 Bytes) All the training images are saved in the RGB format (544,960,3) in kitti_data/train/images/. I see that the tfrecords I create through this command only poin…

Training on Custom Dataset using TLT

I have generated KITTI format dataset for 1 class and modified the specs files for single class. Dataset converted successfully into tfrecords but when running the training snippet, i am facing this error. This machine is amzon ec2 instance p2.xlarge with NVIDIA K-80 GPU. tensorflow.python.framework.errors_impl.InvalidArgumentError: Conv2DCustomBackpropInputOp only supports NHWC Then i tried on my local machine with nvidia gtx 1060 and training and everything works well. But I cannot train it…