Simple Audio Recognition on Jetson Nano

I’m trying to get the Simple Audio Recognition Tutorial by Tensorflow working on the Jetson Nano. (https://www.tensorflow.org/tutorials/sequences/audio_recognition)
The problem already starts in the first line when running the train.py.
After downloading the dataset, several problems occur:

  • "Loaded runtime CuDNN library: 7.3.1 but source was compiled with: 7.5.0. CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library. If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration."
  • "Use tf.where in 2.0, which has the same broadcast rule as np.where"
  • I also set the $path & $LD_LIBRARY_PATH
    Is there a way to upgrade the cudnn Version for the jetson nano or is it limited to released jetpack versions by Nvidia? I also read that the cudnn Versions for the jetson nano differ to desktop versions.

    If it is not possible, is there another easy way to get started with speech recognition on the jetson nano?

    Full Terminalfeedback:

    $ python3 tensorflow/examples/speech_commands/train.py
    2019-08-20 13:50:53.590602: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    ^[/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint16 = np.dtype([("qint16", np.int16, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint32 = np.dtype([("qint32", np.int32, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      np_resource = np.dtype([("resource", np.ubyte, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint8 = np.dtype([("qint8", np.int8, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint16 = np.dtype([("qint16", np.int16, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_quint16 = np.dtype([("quint16", np.uint16, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      _np_qint32 = np.dtype([("qint32", np.int32, 1)])
    /usr/local/lib/python3.6/dist-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
      np_resource = np.dtype([("resource", np.ubyte, 1)])
    2019-08-20 13:51:10.441541: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
    2019-08-20 13:51:10.442307: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x2f87a900 executing computations on platform Host. Devices:
    2019-08-20 13:51:10.442381: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
    2019-08-20 13:51:10.746295: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
    2019-08-20 13:51:11.020507: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:51:11.020820: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3100ed10 executing computations on platform CUDA. Devices:
    2019-08-20 13:51:11.020879: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
    2019-08-20 13:51:11.021403: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:51:11.021536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
    name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
    pciBusID: 0000:00:00.0
    2019-08-20 13:51:11.021626: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2019-08-20 13:51:11.161753: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
    2019-08-20 13:51:11.308924: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
    2019-08-20 13:51:11.527674: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
    2019-08-20 13:51:11.664733: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
    2019-08-20 13:51:11.763516: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
    2019-08-20 13:51:12.068302: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    2019-08-20 13:51:12.068633: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:51:12.068903: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:51:12.069008: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
    2019-08-20 13:51:12.069135: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2019-08-20 13:51:26.728151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
    2019-08-20 13:51:26.738338: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
    2019-08-20 13:51:26.738644: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
    2019-08-20 13:51:27.015357: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:51:27.064916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:51:27.068805: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 257 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
    >> Downloading speech_commands_v0.02.tar.gz 100.0%
    I0820 13:52:25.632115 548396466192 input_data.py:242] Successfully downloaded speech_commands_v0.02.tar.gz (2428923189 bytes)
    2019-08-20 13:55:11.506140: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.518426: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
    name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
    pciBusID: 0000:00:00.0
    2019-08-20 13:55:11.522768: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2019-08-20 13:55:11.538097: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
    2019-08-20 13:55:11.538202: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
    2019-08-20 13:55:11.538249: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
    2019-08-20 13:55:11.538292: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
    2019-08-20 13:55:11.538333: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
    2019-08-20 13:55:11.578946: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    2019-08-20 13:55:11.579211: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.579463: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.579562: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
    2019-08-20 13:55:11.715736: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.715971: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties: 
    name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
    pciBusID: 0000:00:00.0
    2019-08-20 13:55:11.716065: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
    2019-08-20 13:55:11.716174: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
    2019-08-20 13:55:11.716240: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
    2019-08-20 13:55:11.716293: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
    2019-08-20 13:55:11.716348: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
    2019-08-20 13:55:11.716405: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
    2019-08-20 13:55:11.716464: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    2019-08-20 13:55:11.716709: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.717034: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.717165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
    2019-08-20 13:55:11.749492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
    2019-08-20 13:55:11.749567: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0 
    2019-08-20 13:55:11.749596: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N 
    2019-08-20 13:55:11.749967: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.750858: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
    2019-08-20 13:55:11.751088: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 257 MB memory) -> physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
    W0820 13:55:12.965882 548396466192 deprecation.py:506] From /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:278: calling dropout (from tensorflow.python.ops.nn_ops) with keep_prob is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `rate` instead of `keep_prob`. Rate should be set to `rate = 1 - keep_prob`.
    W0820 13:55:13.656599 548396466192 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/losses/losses_impl.py:121: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where
    I0820 13:55:17.648800 548396466192 train.py:195] Training from step: 1 
    2019-08-20 13:55:20.987891: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
    2019-08-20 13:55:22.898557: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
    2019-08-20 13:55:30.171781: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.3.1 but source was compiled with: 7.5.0.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
    2019-08-20 13:55:30.175960: E tensorflow/stream_executor/cuda/cuda_dnn.cc:319] Loaded runtime CuDNN library: 7.3.1 but source was compiled with: 7.5.0.  CuDNN library major and minor version needs to match or have higher minor version in case of CuDNN 7.0 or later version. If using a binary install, upgrade your CuDNN library.  If building from sources, make sure the library loaded at runtime is compatible with the version specified during compile configuration.
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1356, in _do_call
        return fn(*args)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
        options, feed_dict, fetch_list, target_list, run_metadata)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
        run_metadata)
    tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
      (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
    	 [[{{node Conv2D}}]]
    	 [[cross_entropy/sparse_softmax_cross_entropy_loss/value/_31]]
      (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
    	 [[{{node Conv2D}}]]
    0 successful operations.
    0 derived errors ignored.
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "tensorflow/examples/speech_commands/train.py", line 486, in <module>
        tf.compat.v1.app.run(main=main, argv=[sys.argv[0]] + unparsed)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 40, in run
        _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
      File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 300, in run
        _run_main(main, args)
      File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
        sys.exit(main(argv))
      File "tensorflow/examples/speech_commands/train.py", line 234, in main
        dropout_prob: 0.5
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 950, in run
        run_metadata_ptr)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1173, in _run
        feed_dict_tensor, options, run_metadata)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1350, in _do_run
        run_metadata)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/client/session.py", line 1370, in _do_call
        raise type(e)(node_def, op, message)
    tensorflow.python.framework.errors_impl.UnknownError: 2 root error(s) found.
      (0) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
    	 [[node Conv2D (defined at /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:275) ]]
    	 [[cross_entropy/sparse_softmax_cross_entropy_loss/value/_31]]
      (1) Unknown: Failed to get convolution algorithm. This is probably because cuDNN failed to initialize, so try looking to see if a warning log message was printed above.
    	 [[node Conv2D (defined at /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:275) ]]
    0 successful operations.
    0 derived errors ignored.
    
    Errors may have originated from an input operation.
    Input Source operations connected to node Conv2D:
     first_weights/read (defined at /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:267)	
     Reshape (defined at /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:260)
    
    Input Source operations connected to node Conv2D:
     first_weights/read (defined at /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:267)	
     Reshape (defined at /home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py:260)
    
    Original stack trace for 'Conv2D':
      File "tensorflow/examples/speech_commands/train.py", line 486, in <module>
        tf.compat.v1.app.run(main=main, argv=[sys.argv[0]] + unparsed)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/platform/app.py", line 40, in run
        _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
      File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 300, in run
        _run_main(main, args)
      File "/usr/local/lib/python3.6/dist-packages/absl/app.py", line 251, in _run_main
        sys.exit(main(argv))
      File "tensorflow/examples/speech_commands/train.py", line 139, in main
        is_training=True)
      File "/home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py", line 133, in create_model
        return create_conv_model(fingerprint_input, model_settings, is_training)
      File "/home/dlinano/tensorflow-clone/tensorflow/tensorflow/examples/speech_commands/models.py", line 275, in create_conv_model
        padding='SAME') + first_bias
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_ops.py", line 1954, in conv2d
        name=name)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 1071, in conv2d
        data_format=data_format, dilations=dilations, name=name)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
        op_def=op_def)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
        return func(*args, **kwargs)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
        op_def=op_def)
      File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2005, in __init__
        self._traceback = tf_stack.extract_stack()
    

    Got the same problem. After checking https://developer.nvidia.com/rdp/cudnn-archive, didn’t find arrch64 CuDNN …

    Hi,

    Our latest JetPack 4.2.1 provides cuDNN 7.5.0.
    You can get the library by simply reflashing:
    https://developer.nvidia.com/embedded/jetpack

    Please noticed that all the packages for Jetson are only available from the JetPack.
    The website release is usually for the desktop users.

    Thanks.

    i could run the tutorial on the Nano. But only with a batch_size of 16. When trying to increase it, there is a memory allocation error reasoned by not enough memory. Is there a way to increase this, which i didn’t think of?

    The only thing that comes to my mind is a linux host where i train the networks and just load the trained network on the nano?

    Yes, if you want faster training and larger batch sizes, you should absolutely train on a system with a lot more GPU memory and speed, such as a Linux desktop computer, or a rented GPU instance on Amazon.
    You can also fit a larger batch size if you simplify the model, making it use smaller layers, smaller images, and fewer layers.

    Hi,

    Since Nano is designed as an edge device, the hardware resource is limited and is not recommended to use for training.
    There are some ways to add swap memory but it cannot be used as a GPU memory.

    Thanks.