It doesn’t look like memory usage is the problem. nvidia-smi -q says:
FB Memory Usage
Total : 11016 MiB
Used : 9403 MiB
Free : 1613 MiB
...
Process ID : 32036
Type : C+G
Name : DOOMEternalx64vk.exe
Used GPU Memory : 8000 MiB
Although the round value of 8000 MiB VRAM usage is peculiar. System memory is also fine: 6.1 of 16 GiB is used.
This is on RTX 2080 Ti without overclocking.