The tool is nvidia-smi calculates VRAM usage. Mangohud also report the same results. Does not include system RAM.
Is not only games cause problem. Is when VRAM fills up. There are reports of users without playing games.
look at another thread here.
There have been reports by users, that not because vulkan using more RAM. AMDGPU users does not have this problem running vkd3d even when VRAM fills up. Windows driver also does not either.