I don’t use FFmpeg or NVENC, but the error message seems quite clear: It tells you the version of FFmpeg you compiled requires a newer version of NVENC than the one you have installed.
Best I can tell from the FFmpeg GitHub repository, this requirement (NVENC version must be > 5) is quite new, having been a added in March 2016. So if you cannot update NVENC, an alternative approach might be to pull a slightly older version of FFmpeg.
The latest NVIDIA Video Codec SDK version available is 6.0, which requires NVIDIA GPU driver R358 or above for Windows and R358 or above for Linux. You can download the required drivers here Windows,Linux
libavcodec/libavcodec.a(nvenc.o): na função `nvenc_copy_to_inputbuffer':
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1173: referência indefinida para `get_cuda_dl_func'
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1192: referência indefinida para `check_nvinfo'
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1201: referência indefinida para `cuMemCpy2d'
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1202: referência indefinida para `cuMemCpy2d'
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1203: referência indefinida para `cuMemCpy2d'
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1196: referência indefinida para `cuMemCpy2d'
/home/igor/Development/ffmpeg_build/src/libavcodec/nvenc.c:1177: referência indefinida para `check_nvinfo'