RTX 8000 Memory limitation

I have an RTX 8000 and I’m using FFMPEG to transcode multiple streams concurrently. Each stream consume around 575MiB, and when the memory usage reaches around 28000MiB, FFMPEG throw the following message when starting a new transcode session:

[h264_nvenc @ 0x55cd1d574800] dl_fn->cuda_dl->cuCtxCreate(&ctx->cu_context_internal, 0, cu_device) failed -> CUDA_ERROR_OUT_OF_MEMORY: out of memory [h264_nvenc @ 0x55cd1d574800] No NVENC capable devices found Error initializing output stream 2:0 – Error while opening encoder for output stream #2:0 - maybe incorrect parameters such as bit_rate, rate, width or height

The machine is running Ubuntu 20.04 FFMPEG is a snap package from Ubuntu repo "version 4.2.2-1ubuntu1" nVidia driver nvidia-driver-435 installed from Ubuntu repo

command used for the transcode: ffmpeg -i SRC -c:v h264_cuvid -vcodec h264_nvenc -preset:v medium -profile:v main -vf "scale=1920:-2" -hls_flags delete_segments -hls_init_time 4 -hls_time 4 -maxrate 6000k -g 100 -bufsize 12000k -b:v 3000k -start_number 1 -async 1 -c:a aac -b:a 128k 1080.m3u8

I also tried the latest drivers on a Fedora 32 machine but still same results.

Is the RTX 8000 have any limitation on the Memory usage? are the parameters I’m using causing this issue?