I am trying to use project MusicGPT - https://github.com/gabotechs/MusicGPT
I am using Ubuntu 22.04.4 LTS with 6.5.0-35-generic kernel.
I have nvidia-driver-555 with cuda-toolkit NVIDIA Container Runtime Hook version 1.17.1
installed. (Driver Version: 555.58.02 CUDA Version: 12.5)
From following tutorial -
(https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#configuring-docker)
I am getting following error when I try to generate any length of music, using MusicGPT -
Audio Generation Failed
Failed to run inference on model: Non-zero status code returned while running Slice node. Name:‘/decoder/layers.12/Slice’ Status Message: /onnxruntime_src/onnxruntime/core/framework/bfc_arena.cc:376 void* onnxruntime::BFCArena::AllocateRawInternal(size_t, bool, onnxruntime::Stream*, bool, onnxruntime::WaitNotificationFn) Failed to allocate memory for requested buffer of size 81428480
Failed to run inference on model: Non-zero status code returned while running LSTM node. Name:‘/decoder/layers.1/lstm/LSTM’ Status Message: CUDNN failure 4: CUDNN_STATUS_INTERNAL_ERROR ; GPU=0 ; hostname=402880b502d3 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/rnn/cudnn_rnn_base.cc ; line=352 ; expr=cudnnRNNForward(GetCudnnHandle(ctx), rnn_desc, CUDNN_FWD_MODE_INFERENCE, sequence_lens_buffer.GpuPtr(), x_desc1, x_data_input, y_desc1, y_data, hx_desc, hx_data, y_h_data, cx_desc, cx_data, y_c_data, weight_cached_ ? w_data_cache_size_in_bytes_ : w_data_size_in_bytes, weight_cached_ ? w_data_cache_.get() : w_data.get(), workspace_bytes, workspace_cuda.get(), reservespace_bytes, reservespace_cuda.get());
I think size 81428480 in bites is like 10 megabytes or something, I have 2 GB space left on the hard drive.