I am building my own model according to https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v2.0/ with a self-made data loader+transformer and model.
When I run train.sh, only event files are created in the folder “models” but no checkpoint files.
The error message in the console is
Exception: <class ‘ValueError’>: Cannot feed value of shape (1, 384, 384) for Tensor ‘NV_MODEL_INPUT:0’, which has shape ‘(?, 1, 384, 384, 1)’
- My input data are numpy arrays shaped (384, 384). Thus, in the config_train.json I used shape “HW”.
I checked the shape after import data = np.load(file_name, allow_pickle=True).astype(self._dtype) where type is ‘f4’ and it shows (384,384).
- To get (1, 384, 384), I used output_batch_size=1 in the image pipeline.
Do you have any advice which step I am missing? Data format should be ‘NCDHW’ (3D) or ‘NCHW’ (2D)