NVidia nvidia-440.59-9 driver on ArchLinux 5.5.x kernel using pulse audio resume times are very slow

When resuming video playing on recent linux 5.5.x kernel with recent nvidia driver, audio playback when paused for 5 seconds (the point when pulseaudio suspends the alsa driver inside nvidia), it becomes extremely slow and will hang on a poll() syscall trying to resume alsa. This also requires playing audio over HDMI.

Requirements:

= 5.5 kernel
440.44 nvidia dirver
audio playing over HDMI
using pulseaudio (works fine with direct to ALSA)

Confirmed on:
driver: 440.59-9
kernel: 5.5.5-arch1-1

Documented here with logs: Pulseaudio freezes in pa_mainloop_poll causing mpv video player to freeze (#807) · Issues · PulseAudio / pulseaudio · GitLab