I am currently attempting to train using TensorFlow via DGX Spark, but I am encountering the following error.
Can the current Docker setup resolve the following issues?
If there is a Docker version that resolves this issue, please also provide information about it.
The current CUDA version is as follows.
CUDA compilation tools, release 12.8, V12.8.61
Build cuda_12.8.r12.8/compiler.35404655_0
2025-12-04 06:48:34.923630: W tensorflow/core/framework/op_kernel.cc:1840] OP_REQUIRES failed at xla_ops.cc:577 : UNIMPLEMENTED: /usr/local/cuda/bin/ptxas ptxas too old. Falling back to the driver to compile.
Traceback (most recent call last):
File “/workspace/mel_spectrogram_kws_audiomentations.py”, line 340, in
history = model.fit(
^^^^^^^^^^
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/utils/traceback_utils.py”, line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File “/usr/local/lib/python3.12/dist-packages/tensorflow/python/eager/execute.py”, line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tensorflow.python.framework.errors_impl.UnimplementedError: Graph execution error:
The
Detected at node Adam/StatefulPartitionedCall_14 defined at (most recent call last):
File “/workspace/mel_spectrogram_kws_audiomentations.py”, line 340, in
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/utils/traceback_utils.py”, line 65, in error_handler
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/engine/training.py”, line 1804, in fit
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/engine/training.py”, line 1398, in train_function
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/engine/training.py”, line 1381, in step_function
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/engine/training.py”, line 1370, in run_step
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/engine/training.py”, line 1151, in train_step
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/optimizers/optimizer.py”, line 621, in minimize
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/optimizers/optimizer.py”, line 1300, in apply_gradients
The
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/optimizers/optimizer.py”, line 729, in apply_gradients
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/optimizers/optimizer.py”, line 1330, in _internal_apply_gradients
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/optimizers/optimizer.py”, line 1422, in _distributed_apply_gradients_fn
File “/usr/local/lib/python3.12/dist-packages/tf_keras/src/optimizers/optimizer.py”, line 1417, in apply_grad_to_update_var
/usr/local/cuda/bin/ptxas ptxas is too old. Falling back to the driver to compile.
[[{{node Adam/StatefulPartitionedCall_14}}]] [Op:__inference_train_function_2619]