Pointpillars training error

[rainaider]

dataset_convert and data split is no error
when I use my own data for training, error happen bellow:,

File “/opt/conda/lib/python3.8/site-packages/pointcloud/pointpillars/pcdet/models/init.py”, line 142, in model_func
ret_dict, tb_dict, disp_dict = model(batch_dict)
File “/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl
return forward_call(*input, **kwargs)
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/pointcloud/pointpillars/pcdet/models/detectors/pointpillar.py”, line 16, in forward
File “/opt/conda/lib/python3.8/site-packages/torch/nn/modules/module.py”, line 1110, in _call_impl
return forward_call(*input, **kwargs)
File “/home/jenkins/agent/workspace/tlt-pytorch-main-nightly/pointcloud/pointpillars/pcdet/models/backbones_2d/base_bev_backbone.py”, line 107, in forward
RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 127 but got size 128 for tensor number 1 in the list.

[Morganh]

Can you share the training spec file? If possible, could you please share the smallest part of dataset which can easily reproduce the error? Thanks.

[Morganh]

Still checking, will update to you if any. Thanks.

[Morganh]

BTW, did you run the default notebook successfully?

[rainaider]

Thanks

[rainaider]

use kitti data ,no problem

[Morganh]

OK. Indeed, with your dataset, I can also reproduce now. Will check further.

[Morganh]

It is related to POINT_CLOUD_RANGE.
The point cloud coordinates range in [xmin, ymin, zmin, xmax, ymax, zmax] format .
Please set compatible range according to your dataset.

After checking, below can work with your attached dataset.
For example,
POINT_CLOUD_RANGE: [0, -39.68, -3, 69.12, 39.68, 1]
or
POINT_CLOUD_RANGE: [0, -30.68, -2, 50.00, 30.68, 1]

[rainaider]

How to set this range?is lidar sensor range? my dataset range is 360. new problem happen

[Morganh]

For CUDNN error, please exit the container and retry again.

[etshen]

Thanks for the reply, how do we get the compatible coordinates range of dataset [xmin, ymin, zmin, xmax, ymax, zmax], do we need to iterate through the dataset and find these values?

[rainaider]

another problem when I retrain.

[Morganh]

There is no update from you for a period, assuming this is not an issue anymore.
Hence we are closing this topic. If need further support, please open a new one.
Thanks

Please check nvidia-smi.
You can share its result here.