# UBLAS\_STATUS\_NOT\_SUPPORTED when calling cublasGemmStridedBatchedEx" During Grounding DINO Training

**URL:** <https://forums.developer.nvidia.com/t/ublas-status-not-supported-when-calling-cublasgemmstridedbatchedex-during-grounding-dino-training/319946>\
**Category:** Container: CUDA\
**Tags:** tao\
**Created:** [January 14, 2025, 2:33pm UTC](https://forums.developer.nvidia.com/t/ublas-status-not-supported-when-calling-cublasgemmstridedbatchedex-during-grounding-dino-training/319946 "2025-01-14T14:33:36Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![sameer.afzal](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@sameer.afzal](https://forums.developer.nvidia.com/u/sameer.afzal)\
**Post date:** [January 14, 2025, 2:33pm UTC](https://forums.developer.nvidia.com/t/ublas-status-not-supported-when-calling-cublasgemmstridedbatchedex-during-grounding-dino-training/319946/1 "2025-01-14T14:33:36Z")

</div>

I am encountering an issue while using the NVIDIA TAO Toolkit version 5.5. I followed the instructions in the Grounding DINO sample notebook available at the official NVIDIA

GitHub repository: `[Grounding DINO Notebook.](https://github.com/NVIDIA/tao_tutorials/blob/main/notebooks/tao_launcher_starter_kit/grounding_dino/finetune_grounding_dino.ipynb)`

Steps to Reproduce:

1. Installed NVIDIA TAO Toolkit 5.5.

2. Configured the train.yaml file for training Grounding DINO.

3. Ran the following training command:  
!tao model grounding\_dino train   
-e $SPECS\_DIR/train.yaml   
train.num\_gpus=$NUM\_TRAIN\_GPUS   
results\_dir=$RESULTS\_DIR

4. During training, the following error was raised:  
RuntimeError: CUDA error: CUBLAS\_STATUS\_NOT\_SUPPORTED when calling cublasGemmStridedBatchedEx(…)

System Details:

- GPU: NVIDIA GeForce GTX TITAN X
- Driver Version: 555.42.06
- CUDA Version: 12.5 (nvidia-smi)
- CUDA Version: 12.4 (miniconda-environment)
- PyTorch Version: 2.5.1+cu124 (miniconda-environment)

Additional Notes:

1. Using Automatic Mixed Precision (AMP) with bfloat16.
2. Reducing the batch size and disabling AMP partially mitigated the issue but training still fails.
3. Input size and dataset configurations follow the default settings in the sample notebook.

I suspect the issue may be related to:

- Incompatibility of bfloat16 precision with my GPU.
- A mismatch in CUDA/PyTorch versions.

Any guidance or suggestions would be greatly appreciated!