Using pinned memory on orin platform，caused increase gpu memory usage

[446456877]

When Try to allocate pinned memory on Jetson Orin Platform，I found a huge gpu memory usage increasing ？ Why ？

code for allocate about 4GB Memory

import torch
from typing import List

def _allocate_kv_cache(
) -> List[torch.Tensor]:
    """Allocates KV cache on the specified device."""
    kv_cache_shape = (4681, 2, 16, 4, 128)
    pin_memory = True
    kv_cache: List[torch.Tensor] = []
    for _ in range(28):
        # null block in CpuGpuBlockAllocator requires at least that
        # block to be zeroed-out.
        # We zero-out everything for simplicity.
        kv_cache.append(
            torch.zeros(kv_cache_shape,
                        dtype=torch.float16,
                        pin_memory=pin_memory,
                        device="cpu"))
    return kv_cache

_allocate_kv_cache()

Memory Usage After allocating：

And Why ？

[carolyuu]

Hi,
Here are some suggestions for the common issues:

1. Performance

Please run the below command before benchmarking deep learning use case:

$ sudo nvpmodel -m 0
$ sudo jetson_clocks

2. Installation

Installation guide of deep learning frameworks on Jetson:

• TensorFlow: Installing TensorFlow for Jetson Platform - NVIDIA Docs
• PyTorch: Installing PyTorch for Jetson Platform - NVIDIA Docs
  We also have containers that have frameworks preinstalled:
  Data Science, Machine Learning, AI, HPC Containers | NVIDIA NGC

3. Tutorial

Startup deep learning tutorial:

• Jetson-inference: Hello AI World guide to deploying deep-learning inference networks and deep vision primitives with TensorRT and NVIDIA Jetson
• TensorRT sample: Jetson/L4T/TRT Customized Example - eLinux.org

4. Report issue

If these suggestions don’t help and you want to report an issue to us, please attach the model, command/step, and the customized app (if any) with us to reproduce locally.

Thanks!

[AastaLLL]

Hi,

Could you compare it to the case with pin_memory = False?
Since Jetson is a shared memory system, all the running processes can occupy the memory.

Thanks