How can I verify whether a process is actually using my GPU/CUDA?

Robotics & Edge Computing Jetson Systems Jetson AGX Orin
, , ,
1

I’m trying to understand how GPU acceleration works on my system, especially for open-source software and AI/ML tools that I run through VS Code.

I have an NVIDIA GPU installed and believe I have the necessary GPU drivers/software installed, but I’m not sure how to determine whether a particular application or process is actually using the GPU.

What I’m trying to understand

  1. How can I check whether my NVIDIA GPU is working correctly?

    • Is there a simple command or tool to verify that the GPU and driver are detected?

    • How can I check the available VRAM and GPU utilization?

  2. Is CUDA required for every application that uses an NVIDIA GPU?

    • What exactly is CUDA?

    • When do I need the CUDA Toolkit?

    • Is installing the NVIDIA driver enough for some applications?

    • How do I know whether a particular application requires CUDA, CUDA Toolkit, cuDNN, etc.?

  3. How can I tell whether a specific process is using the GPU?

    • For example, if I run an open-source AI model from VS Code, how can I confirm that the model is actually running on the GPU rather than the CPU?

    • How can I see which process is consuming GPU memory?

    • How can I determine whether the application is using the full GPU capability or only a portion of it?

  4. VS Code and GPU usage

    • If I run an application/script from VS Code, does VS Code itself use the GPU, or does the application running inside VS Code determine whether the GPU is used?

    • For example, if I run a Python AI/ML program from the VS Code terminal, what determines whether it runs on CPU or GPU?

  5. How do I verify the software stack?

    I would like to understand the relationship between:

    GPU → NVIDIA Driver → CUDA → CUDA Toolkit → cuDNN → Python/PyTorch/other framework → Application/Model

    Which components are actually required depends on the software, and I would like to understand how to identify the requirements rather than installing everything unnecessarily.

My main goal

I want a reliable way to answer these questions whenever I install or run new software:

Is this application capable of using my GPU?
Is it actually using my GPU right now?
How much GPU/VRAM is it using?
Is it using CUDA or another GPU API?
Are all the required drivers/libraries installed correctly?
Is it running on the CPU because of a configuration or installation problem?

I’m particularly interested in open-source AI/ML software and applications launched through VS Code.

What commands, monitoring tools, or diagnostic procedures would you recommend for checking this systematically?

4

Hi,

You don’t need to maunually install the GPU driver.
The compatible driver is pre-installed in the L4T OS.

  1. Please run the command below to check the GPU utilization (GR3D).
    The command also contains memory utilization info.
$ sudo tegrastats

  1. Yes.

  2. Please run the command below. You will need to use JetPack 7+ to get the information on the processes that use the GPU.

$ sudo nvidia-smi
  1. For example, below is the toy sample that is deployed on the GPU
$ python
Python 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.rand(10).to(torch.device("cuda"))
tensor([0.1885, 0.9269, 0.7221, 0.9849, 0.4552, 0.9575, 0.3322, 0.7051, 0.8399,
        0.4589], device='cuda:0')
  1. The GPU driver and CUDA toolkit are essential. And others depend on your use case.

For an AI use case, you can check the link below that running LLM model in a container.

Thanks.

5

Thank you!!!