NVIDIA CUDA cannot work properly

Hello guys:
I try to run a machine learning program with NVIDIA RTX A6000 graphical cards and Pytorch, but confronted with the following problem:
if I try to print the loss object with spyder, it will print the following error:

Traceback (most recent call last):

File “C:\Users\hp\anaconda3\lib\site-packages\IPython\core\formatters.py”, line 702, in **call**

File “C:\Users\hp\anaconda3\lib\site-packages\IPython\lib\pretty.py”, line 394, in pretty
return _repr_pprint(obj, self, cycle)

File “C:\Users\hp\anaconda3\lib\site-packages\IPython\lib\pretty.py”, line 700, in _repr_pprint
output = repr(obj)

File “C:\Users\hp\anaconda3\lib\site-packages\torch\tensor.py”, line 193, in **repr**
return torch._tensor_str._str(self)

File “C:\Users\hp\anaconda3\lib\site-packages\torch_tensor_str.py”, line 383, in _str
return _str_intern(self)

File “C:\Users\hp\anaconda3\lib\site-packages\torch_tensor_str.py”, line 358, in _str_intern
tensor_str = _tensor_str(self, indent)

File “C:\Users\hp\anaconda3\lib\site-packages\torch_tensor_str.py”, line 242, in _tensor_str
formatter = _Formatter(get_summarized_data(self) if summarize else self)

File “C:\Users\hp\anaconda3\lib\site-packages\torch_tensor_str.py”, line 90, in **init**
nonzero_finite_vals = torch.masked_select(tensor_view, torch.isfinite(tensor_view) & tensor_view.ne(0))

RuntimeError: CUDA error: an illegal memory access was encountered

After some times, I installed cupy packages and run some linear algebra calculations on the same server, it seems that python can smoothly run some caluclations, but the same error will appear unexpectedly.
The NVIDIA driver version is 472.47, CUDA version is cuda 11.4, I am using NVIDIA RTX A6000 GPU on an server with windows 10.
Note that we also tried running Pytorch machine learning program on the same server with Pytorch on ubuntu 16.04LTS environment, but no errors will appear. As a result, I think this is a problem caused by driver and cuda, not because of inproper installation of Pytorch/Cupy.

Please tell me if you have any effective solutions, thank you!!!

The error prompted when using cupy looks like follows:

Traceback (most recent call last):

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\spyder_kernels\py3compat.py", line 356, in compat_exec
    exec(code, globals, locals)

  File "h:\jktong\2d_water_test\train_cupy_all_format_2d_zarr.py", line 358, in <module>
    Rt_Zarr[:] = cp.asnumpy(Rt_gpu)

  File "C:\Users\hp\.conda\envs\CUPY\lib\site-packages\cupy\__init__.py", line 816, in asnumpy
    return a.get(stream=stream, order=order, out=out)

  File "cupy\_core\core.pyx", line 1755, in cupy._core.core._ndarray_base.get

  File "cupy\_core\core.pyx", line 1840, in cupy._core.core._ndarray_base.get

  File "cupy\cuda\memory.pyx", line 562, in cupy.cuda.memory.MemoryPointer.copy_to_host

  File "cupy_backends\cuda\api\runtime.pyx", line 580, in cupy_backends.cuda.api.runtime.memcpy

  File "cupy_backends\cuda\api\runtime.pyx", line 143, in cupy_backends.cuda.api.runtime.check_status

CUDARuntimeError: cudaErrorIllegalAddress: an illegal memory access was encountered

I performed some linear algebra calculations on the GPU using using CUPY.