I made a minimum case to reproduce the bug (on latest NVIDIA drivers and latest Vulkan SDK. Windows 11 with latest updates. I can reproduce the bug on both RTX3090 and RTX3080).
The program call vkResetCommandPool to release command buffers created for previous frames. This should be equivalent to freeing all command buffers / recreate the command pool again. However, calling vkResetCommandPool does not seem to be taking effect.
The memory consumption increases a few hundred KB each frame, and the program significantly slows down after a few thousand frames. Memory consumption can grow over 10G after a few minumtes for some applications.
Switching to recreating the command pool each frame, or free command buffers after use seems fine.

