"Health ready check failed."error when run riva_start.sh

I hope someone can help me solve this problem.

Device:Jetson AGX Orin 32G

SDK:JetPack5.1.4
Hardware - CPU:ARM 64
Operating System:Ubuntu 20.04.6 LTS
Riva Version:2.24.0-l4t-aarch64

I followed the steps in the following link.

https://github.com/nvidia-riva/python-clients?tab=readme-ov-file

After the Installation is complete, I then follow the steps in the following link.

https://docs.nvidia.com/deeplearning/riva/user-guide/docs/quick-start-guide.html

Then everything worked until I ran the riva_start.sh.

The error:

evoc@tegra-ubuntu:~/riva_quickstart_arm64_2.24.0$ sudo bash riva_start.sh
Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech

Then I try this way but it still doesn’t work.

riva_speech_logs.txt (1.2 KB)

config.txt (19.6 KB)

riva_start.txt (5.2 KB)

Thanks!

My command and error:

evoc@tegra-ubuntu:~/riva_quickstart_arm64_2.24.0$ sudo bash riva_start.sh
Starting Riva Speech Services. This may take several minutes depending on the number of models deployed.
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Waiting for Riva server to load all models...retrying in 10 seconds
Health ready check failed.
Check Riva logs with: docker logs riva-speech
evoc@tegra-ubuntu:~/riva_quickstart_arm64_2.24.0$ sudo docker logs riva-speech

==========================
=== Riva Speech Skills ===
==========================

NVIDIA Release 25.12 (build 242137302)

Copyright (c) 2018-2025, NVIDIA CORPORATION & AFFILIATES. All rights reserved.

Various files include modifications (c) NVIDIA CORPORATION & AFFILIATES.  All rights reserved.

GOVERNING TERMS: The software and materials are governed by the NVIDIA Software License Agreement
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-software-license-agreement/)
and the Product-Specific Terms for NVIDIA AI Products
(found at https://www.nvidia.com/en-us/agreements/enterprise-software/product-specific-terms-for-ai-products/).

NVIDIA Tegra driver detected.

  > Riva waiting for Triton server to load all models...retrying in 1 second
W0122 08:13:54.227710 144 pinned_memory_manager.cc:273] "Unable to allocate pinned system memory, pinned memory pool will not be available: CUDA driver version is insufficient for CUDA runtime version"
E0122 08:13:54.267891 144 model_lifecycle.cc:654] "failed to load 'magpie_tts-Magpie-Multilingual' version 1: Invalid argument: instance group magpie_tts-Magpie-Multilingual_0 of model magpie_tts-Magpie-Multilingual has kind KIND_GPU but no GPUs are available"
E0122 08:13:54.347809 144 model_lifecycle.cc:654] "failed to load 'riva-trt-parakeet-1.1b-en-US-asr-streaming-am-streaming' version 1: Invalid argument: instance group riva-trt-parakeet-1.1b-en-US-asr-streaming-am-streaming_0 of model riva-trt-parakeet-1.1b-en-US-asr-streaming-am-streaming has kind KIND_GPU but no GPUs are available"
E0122 08:13:54.347800 144 model_lifecycle.cc:654] "failed to load 'riva-trt-riva-punctuation-en-US-nn-bert-base-uncased' version 1: Invalid argument: instance group riva-trt-riva-punctuation-en-US-nn-bert-base-uncased_0 of model riva-trt-riva-punctuation-en-US-nn-bert-base-uncased has kind KIND_GPU but no GPUs are available"
E0122 08:13:55.110052 144 model_lifecycle.cc:654] "failed to load 'tts_preprocessor-Magpie-Multilingual' version 1: Invalid argument: instance group tts_preprocessor-Magpie-Multilingual_0 of model tts_preprocessor-Magpie-Multilingual has kind KIND_GPU but no GPUs are available"
Found yaml file: /data/models/parakeet-1.1b-en-US-asr-streaming/1/riva_bls_config.yaml
  > Riva waiting for Triton server to load all models...retrying in 1 second
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister( pinned_host_punct_buffer_.data(), pinned_host_punct_buffer_.size() * sizeof(float), 0)' in fileriva/nlp/pipeline/punctuator/punctuator.cc line 176'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister( pinned_host_capit_buffer_.data(), pinned_host_capit_buffer_.size() * sizeof(float), 0)' in fileriva/nlp/pipeline/punctuator/punctuator.cc line 178'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaSetDevice(device_id)' in file./riva/pipeline/pipeline.h line 52'
  > Riva waiting for Triton server to load all models...retrying in 1 second
cublasStatus_t 1 : "CUBLAS_STATUS_NOT_INITIALIZED" returned from 'cublasCreate(&cublas_handle_)'cublasStatus_t 1 : "CUBLAS_STATUS_NOT_INITIALIZED" returned from 'cublasSetStream(cublas_handle_, cudaStreamPerThread)'curandStatus_t 203 : "CURAND_STATUS_INITIALIZATION_FAILED" returned from 'curandCreateGenerator( &curand_handle_, CURAND_RNG_PSEUDO_DEFAULT)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetGeneratorOrdering( curand_handle_, CURAND_ORDERING_PSEUDO_DEFAULT)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetStream(curand_handle_, cudaStreamPerThread)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetPseudoRandomGeneratorSeed( curand_handle_, distribution(generator))'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetGeneratorOffset(curand_handle_, 0)'cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister(audio_processed_.data(), audio_processed_.size() * sizeof(float), 0)' in fileriva/asr/features/extractor.cc line 239'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister( time_steps_processed_.data(), time_steps_processed_.size() * sizeof(int), 0)' in fileriva/asr/features/extractor.cc line 242'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister( feature_offset_start_.data(), feature_offset_start_.size() * sizeof(int), 0)' in fileriva/asr/features/extractor.cc line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister(feature_offset_end_.data(), feature_offset_end_.size() * sizeof(int), 0)' in fileriva/asr/features/extractor.cc line 248'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&d_feature_offset_start_, sizeof(int) * feature_offset_start_.size())' in fileriva/asr/features/extractor.cc line 250'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&d_feature_offset_end_, sizeof(int) * feature_offset_end_.size())' in fileriva/asr/features/extractor.cc line 251'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc( &d_const_left_pad_feature_offset_start_, sizeof(int) * const_left_pad_feature_offset_start.size())' in fileriva/asr/features/extractor.cc line 257'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc( &d_const_left_pad_feature_offset_end_, sizeof(int) * const_left_pad_feature_offset_end.size())' in fileriva/asr/features/extractor.cc line 260'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpy( d_const_left_pad_feature_offset_start_, const_left_pad_feature_offset_start.data(), sizeof(int) * const_left_pad_feature_offset_start.size(), cudaMemcpyHostToDevice)' in fileriva/asr/features/extractor.cc line 263'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpy( d_const_left_pad_feature_offset_end_, const_left_pad_feature_offset_end.data(), sizeof(int) * const_left_pad_feature_offset_end.size(), cudaMemcpyHostToDevice)' in fileriva/asr/features/extractor.cc line 266'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&d_offline_batch_slot_ids_, config_.max_execution_batch_size * sizeof(int))' in fileriva/asr/features/extractor.cc line 273'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpy( d_offline_batch_slot_ids_, offline_batch_slot_ids_.data(), config_.max_execution_batch_size * sizeof(int), cudaMemcpyHostToDevice)' in fileriva/asr/features/extractor.cc line 275'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&d_const_batch_slots_zero_, sizeof(int) * 1)' in fileriva/asr/features/extractor.cc line 281'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset(d_const_batch_slots_zero_, 0, sizeof(int) * 1)' in fileriva/asr/features/extractor.cc line 282'
cublasStatus_t 1 : "CUBLAS_STATUS_NOT_INITIALIZED" returned from 'cublasCreate(&cublas_handle_)'cublasStatus_t 203 : "CUBLAS_STATUS_UNKNOWN_ERROR" returned from 'curandCreateGenerator(&curand_handle_, CURAND_RNG_PSEUDO_DEFAULT)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetGeneratorOrdering(curand_handle_, CURAND_ORDERING_PSEUDO_DEFAULT)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetStream(curand_handle_, cudaStreamPerThread)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetPseudoRandomGeneratorSeed( curand_handle_, config_.is_dither_seed_random ? riva::utils::matrix::RandInt(kMIN_DITHER_SEED, RAND_MAX) : kMIN_DITHER_SEED)'curandStatus_t 101 : "CURAND_STATUS_NOT_INITIALIZED" returned from 'curandSetGeneratorOffset(curand_handle_, 0)'cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, bytes, cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 179'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemsetAsync(data_.get(), 0, dim_ * sizeof(Real), cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 77'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, bytes, cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 179'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemsetAsync(data_.get(), 0, dim_ * sizeof(Real), cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 77'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, bytes, cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 179'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemsetAsync(data_.get(), 0, dim_ * sizeof(Real), cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 77'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_, stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 229'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( RowData(row), mat[row].data(), num_cols * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 245'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpy2DAsync( data_, dst_pitch, M.data_, src_pitch, width, M.num_rows_, cudaMemcpyDeviceToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 271'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaGetLastError()' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 274'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 167'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync( this->data_, &vec[0], vec.size() * sizeof(Real), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-vector.cc line 65'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync(vecs_, &vecs[0], size * sizeof(float *), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 70'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync(offsets_, &offsets[0], size * sizeof(int32_t), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 72'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemcpyAsync(sizes_, &sizes[0], size * sizeof(int32_t), cudaMemcpyHostToDevice, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 74'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaStreamSynchronize(cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudafeat/feature-online-batched-spectral-cuda.cc line 76'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 212'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 212'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_, stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 229'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&lanes_, sizeof(LaneDesc) * max_lanes_)' in fileexternal/cu-feat-extr/src/cudafeat/online-batched-feature-pipeline-cuda.cc line 151'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, bytes, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 105'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_.get(), stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 122'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaHostRegister( cpu_pcm_audio_buffers_.data(), cpu_pcm_audio_buffers_.size() * sizeof(int16_t), 0)' in fileriva/asr/features/extractor.cc line 380'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, bytes)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 212'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc( (void**)&cu_input_audio_signals_, config_.max_execution_batch_size * config_.num_samples * sizeof(float))' in fileriva/asr/features/extractor.cc line 393'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, bytes, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 105'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_.get(), stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 122'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, row_bytes * rows, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 100'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_.get(), stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 122'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, row_bytes * rows, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 100'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_.get(), stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileriva/utils/matrix/cu_matrix.cc line 122'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc( &d_output_buffer_, sizeof(float) * config_.max_execution_batch_size * config_.num_features * output_num_time_steps_)' in fileriva/asr/features/extractor.cc line 413'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMemset2DAsync( data_, stride_ * sizeof(Real), 0, num_cols_ * sizeof(Real), num_rows_, cudaStreamPerThread)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 229'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMallocAsync(&data, bytes, cudaStreamPerThread)' in fileriva/utils/matrix/cu_vector.cc line 179'
cudaError_t 35 : "CUDA driver version is insufficient for CUDA runtime version" returned from 'cudaMalloc(&data, row_bytes * rows)' in fileexternal/cu-feat-extr/src/cudamatrix/cu-matrix.cu line 204'
[1c79d8963c73:144  :0:156] Caught signal 11 (Segmentation fault: address not mapped to object at address 0x73)
==== backtrace (tid:    156) ====
 0  /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2cc) [0xffff48a8278c]
 1  /opt/hpcx/ucx/lib/libucs.so.0(+0x3293c) [0xffff48a8293c]
 2  /opt/hpcx/ucx/lib/libucs.so.0(+0x32c68) [0xffff48a82c68]
 3  linux-vdso.so.1(__kernel_rt_sigreturn+0) [0xffff9ac037c0]
 4  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(_ZN4riva6asrlib4feat32OnlineBatchedFeaturePipelineCuda22ComputeFeaturesBatchedEiRKSt6vectorIiSaIiEES7_RKS3_IbSaIbEESB_fRKNS1_8CuMatrixIfEEPSD_PS5_+0x130) [0xfffe59087600]
 5  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(_ZN4riva3asr9extractor16FeatureExtractor15ComputeFeaturesERKNS_6asrlib4feat8CuMatrixIfEEPS6_iRKSt6vectorIiSaIiEESE_+0x144) [0xfffe59063214]
 6  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(_ZN4riva3asr9extractor16FeatureExtractor19ComputeZeroFeaturesEv+0xb8) [0xfffe590641d8]
 7  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(_ZN4riva3asr9extractor16FeatureExtractor10InitializeEv+0xfd0) [0xfffe590663d0]
 8  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(_ZN4riva3asr8pipeline12asr_ensemble18AsrEnsembleFactory12GetProcessorB5cxx11Ei+0x110) [0xfffe58c19e90]
 9  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(_ZN4riva7backend8pipeline21PipelineInstanceStateINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEE10InitializeEv+0x68) [0xfffe58bbe568]
10  /opt/tritonserver/backends/riva_asr_ensemble_pipeline/libtriton_riva_asr_ensemble_pipeline.so(TRITONBACKEND_ModelInstanceInitialize+0x60c) [0xfffe58bb7bac]
11  /opt/tritonserver/lib/libtritonserver.so(+0x1567a4) [0xffff998267a4]
12  /opt/tritonserver/lib/libtritonserver.so(+0x157124) [0xffff99827124]
13  /opt/tritonserver/lib/libtritonserver.so(+0x13e7c0) [0xffff9980e7c0]
14  /opt/tritonserver/lib/libtritonserver.so(+0x13ebc8) [0xffff9980ebc8]
15  /opt/tritonserver/lib/libtritonserver.so(+0x1480c4) [0xffff998180c4]
16  /lib/aarch64-linux-gnu/libc.so.6(+0x8abdc) [0xffff98c8abdc]
17  /opt/tritonserver/lib/libtritonserver.so(+0x135ce8) [0xffff99805ce8]
18  /opt/tritonserver/lib/libtritonserver.so(+0x142c5c) [0xffff99812c5c]
19  /opt/tritonserver/lib/libtritonserver.so(+0x146830) [0xffff99816830]
20  /opt/tritonserver/lib/libtritonserver.so(+0x22c83c) [0xffff998fc83c]
21  /opt/tritonserver/lib/libtritonserver.so(+0x22f7f0) [0xffff998ff7f0]
22  /opt/tritonserver/lib/libtritonserver.so(+0x37e300) [0xffff99a4e300]
23  /lib/aarch64-linux-gnu/libstdc++.so.6(+0xe1ae0) [0xffff98f91ae0]
24  /lib/aarch64-linux-gnu/libc.so.6(+0x8595c) [0xffff98c8595c]
25  /lib/aarch64-linux-gnu/libc.so.6(+0xeb89c) [0xffff98ceb89c]
=================================
  > Riva waiting for Triton server to load all models...retrying in 1 second
/opt/riva/bin/start-riva: line 11:   144 Segmentation fault      (core dumped) ${CUSTOM_TRITON_ENV} tritonserver --metrics-port ${NIM_TRITON_METRICS_PORT} --grpc-port ${NIM_GRPC_TRITON_PORT} --http-port ${NIM_HTTP_TRITON_PORT} --log-verbose=${TRITON_LOG_VERBOSE} --log-info=${TRITON_LOG_INFO} --disable-auto-complete-config $model_repos --cuda-memory-pool-byte-size=0:1000000000 ${TRITON_EXTRA_ARGS}
  > Triton server died before reaching ready state. Terminating Riva startup.
Check Triton logs with: docker logs
kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
evoc@tegra-ubuntu:~/riva_quickstart_arm64_2.24.0$

It seems that the CUDA version is not compatible with the Riva version.

And the CUDA version is tied to the JetPack version.

My current version:

JetPack : 5.1.4
CUDA : 11.4.19
Riva : 2.24.0-l4t-aarch64

So what is the correct version ?

Thanks!

Which version?

Based on this link, I chose Riva 2.11.0 to pair with Jetpack 5.1.4.

https://forums.developer.nvidia.com/t/riva-quickstart-2-11-fails-on-xavier-nx/256002

OK,i try to download riva 2.11.0.

But when I run “sudo bash riva_init.sh“, a new error occurs.

evoc@tegra-ubuntu:~/riva_quickstart_arm64_2.11.0$ sudo bash riva_init.sh
[sudo] password for evoc:
Please enter API key for ngc.nvidia.com:
Logging into NGC docker registry if necessary...
Pulling required docker images if necessary...
Note: This may take some time, depending on the speed of your Internet connection.
> Pulling Riva Speech Server images.
  > Image nvcr.io/nvidia/riva/riva-speech:2.11.0-l4t-aarch64 exists. Skipping.

Downloading models (RMIRs) from NGC...
Note: this may take some time, depending on the speed of your Internet connection.
To skip this process and use existing RMIRs set the location and corresponding flag in config.sh.

Error: Invalid apikey
Error: Invalid apikey
Error: Invalid apikey
Error: Invalid apikey

+ [[ tegra != \t\e\g\r\a ]]
+ [[ tegra == \t\e\g\r\a ]]
+ '[' -d /home/evoc/riva_quickstart_arm64_2.11.0/model_repository/rmir ']'
+ [[ tegra == \t\e\g\r\a ]]
+ '[' -d /home/evoc/riva_quickstart_arm64_2.11.0/model_repository/prebuilt ']'
+ echo

+ echo 'Riva initialization complete. Run ./riva_start.sh to launch services.'
Riva initialization complete. Run ./riva_start.sh to launch services.
evoc@tegra-ubuntu:~/riva_quickstart_arm64_2.11.0$

I guarantee that my API key is correct.

Then I try to solve it by this way:

https://forums.developer.nvidia.com/t/ngc-api-key-is-invalid-please-check-and-try-again/184211

Then I download NGC CLI by this way.

https://org.ngc.nvidia.com/setup/installers/cli

My command:

evoc@tegra-ubuntu:~$ wget --content-disposition https://api.ngc.nvidia.com/v2/resources/nvidia/ngc-apps/ngc_cli/versions/4.11.1/files/ngccli_arm64.zip -O ngccli_arm64.zip && unzip ngccli_arm64.zip
(Download process omitted...)
evoc@tegra-ubuntu:~$ find ngc-cli/ -type f -exec md5sum {} + | LC_ALL=C sort | md5sum -c ngc-cli.md5
-: OK
evoc@tegra-ubuntu:~$ sha256sum ngccli_arm64.zip
9c285ef8250c30c21e70a836700121945b296a8441005b829179bde6106ce6d1  ngccli_arm64.zip
evoc@tegra-ubuntu:~$ chmod u+x ngc-cli/ngc
evoc@tegra-ubuntu:~$ echo "export PATH=\"\$PATH:$(pwd)/ngc-cli\"" >> ~/.bash_profile && source ~/.bash_profile
evoc@tegra-ubuntu:~$ ngc config set
Enter API key [no-apikey]. Choices: [<VALID_APIKEY>, 'no-apikey']: (my API key)
Enter CLI output format type [ascii]. Choices: ['ascii', 'csv', 'json']:
Enter org [no-org]. Choices: ['0882813415621389']:
Invalid org. Please re-enter.
(I didn't know what to enter in here, so I quit.)

But when I run “sudo bash riva_init.sh“, the same error as above still occurs.

So how to solve it?

Thanks!

continue

evoc@tegra-ubuntu:~/ngc-cli$ ngc config set
Enter API key [no-apikey]. Choices: [<VALID_APIKEY>, 'no-apikey']: (my API key)
Enter CLI output format type [ascii]. Choices: ['ascii', 'csv', 'json']:
Enter org [no-org]. Choices: ['0882813415621389']:
Invalid org. Please re-enter.
Enter org [no-org]. Choices: ['0882813415621389']: 0882813415621389
Enter team [no-team]. Choices: ['no-team']:
Enter ace [no-ace]. Choices: ['no-ace']:
Validating configuration...
Successfully validated configuration.
Saving configuration...
Successfully saved NGC configuration to /home/evoc/.ngc/config

Run “sudo bash riva_init.sh“, still the same error.

Help!

Are you still seeing the “Invalid apikey” error or the “Triton server died before reaching ready state. Terminating Riva startup” error? Riva 2.11 and JetPack5.1.4 are very old versions. Ifpossible, please upgrade to Jetson Thor and use the latest Riva 2.24.0 with JetPack 7.0 and see if you are able to reproduce the issue.

I’m sorry that I have to use JetPack 5.1.4.