Quadro P4000 audio always play out of one display port (out of four)

Audio only play out of one DisplayPort. This is observed on the P4000 with 440 driver, the sound routing seems to be stuck on whichever monitor that I designated as primary through xorg.conf.

No matter which device I ask to play through aplay –D, sound comes out from the monitor that is plugged into DP-4 (primary in Xorg). e.g.
root@intel-corei7-64:~# aplay -D hdmi:CARD=NVidia,DEV=# piano2.wav
DEV=# 0 to 3 always play on DP-4 and no sound from DP-6

Using ALSA only, no PulseAudio.

Thanks in advance.

aplay -L gives
null
Discard all samples (playback) or generate zero samples (capture)
default:CARD=PCH
HDA Intel PCH, ALC892 Analog
Default Audio Device
sysdefault:CARD=PCH
HDA Intel PCH, ALC892 Analog
Default Audio Device
front:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
Front speakers
surround21:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
2.1 Surround output to Front and Subwoofer speakers
surround40:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
4.0 Surround output to Front and Rear speakers
surround41:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
4.1 Surround output to Front, Rear and Subwoofer speakers
surround50:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
5.0 Surround output to Front, Center and Rear speakers
surround51:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
5.1 Surround output to Front, Center, Rear and Subwoofer speakers
surround71:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Analog
7.1 Surround output to Front, Center, Side, Rear and Woofer speakers
iec958:CARD=PCH,DEV=0
HDA Intel PCH, ALC892 Digital
IEC958 (S/PDIF) Digital Audio Output
hdmi:CARD=NVidia,DEV=0
HDA NVidia, HDMI 0
HDMI Audio Output
hdmi:CARD=NVidia,DEV=1
HDA NVidia, HDMI 1
HDMI Audio Output
hdmi:CARD=NVidia,DEV=2
HDA NVidia, HDMI 2
HDMI Audio Output
hdmi:CARD=NVidia,DEV=3
HDA NVidia, HDMI 3
HDMI Audio Output