NVidia GPU + KVM Switch = X11 Crash

I have a PC running Linux Mint 19.2 with a GTX 1080 connected via DisplayPort to a 4 port KVM Switch. Occasionally, when i switch the KVM input to my other device (a laptop) and then switch back to Linux, the monitor turn off and i can’t get it back on without ssh’ing and rebooting (and sometimes even that doesn’t work and i have to hard reset). When i check the kernel log, i see an X server page allocation failure with a stacktrace involving nvidia_modeset. I’ve tried updating from 430 to the latest drivers (440.26) but i still get the error. Here’s the output from dmesg

[72159.837065] Xorg: page allocation failure: order:4, mode:0x14040c0(GFP_KERNEL|__GFP_COMP), nodemask=(null)
[72159.837067] Xorg cpuset=/ mems_allowed=0
[72159.837073] CPU: 4 PID: 25942 Comm: Xorg Tainted: P           OE    4.15.0-66-generic #75-Ubuntu
[72159.837074] Hardware name: System manufacturer System Product Name/Z170I PRO GAMING, BIOS 3610 03/29/2018
[72159.837075] Call Trace:
[72159.837084]  dump_stack+0x63/0x8e
[72159.837088]  warn_alloc+0xff/0x1a0
[72159.837092]  ? __alloc_pages_direct_compact+0x51/0x100
[72159.837095]  __alloc_pages_slowpath+0xdc5/0xe00
[72159.837100]  __alloc_pages_nodemask+0x29a/0x2c0
[72159.837104]  alloc_pages_current+0x6a/0xe0
[72159.837108]  kmalloc_order+0x18/0x40
[72159.837111]  kmalloc_order_trace+0x24/0xb0
[72159.837130]  ? _nv000489kms+0x50/0x50 [nvidia_modeset]
[72159.837134]  __kmalloc+0x209/0x220
[72159.837148]  ? _nv000489kms+0x50/0x50 [nvidia_modeset]
[72159.837161]  nvkms_alloc+0x25/0x60 [nvidia_modeset]
[72159.837182]  _nv002516kms+0x16/0x30 [nvidia_modeset]
[72159.837185] WARNING: kernel stack frame pointer at 00000000f3fecc0f in Xorg:25942 has bad value 000000008efd1a3f
[72159.837187] unwind stack type:0 next_sp:          (null) mask:0x2 graph_idx:0
[72159.837189] 00000000a8f17517: ffffbadf81fdb870 (0xffffbadf81fdb870)
[72159.837192] 000000001d7c0ce4: ffffffff92631bf3 (show_trace_log_lvl+0x223/0x3d0)
[72159.837193] 000000009913212d: 0000000000000000 ...
[72159.837197] 00000000cdfa4f3d: ffffffff936b7a88 (.LC0+0x4d0/0x670)
[72159.837198] 000000007ce2b488: ffff98add92645c0 (0xffff98add92645c0)
[72159.837217] 000000007eff7b94: ffffffffc0f27656 (_nv002516kms+0x16/0x30 [nvidia_modeset])
[72159.837218] 00000000993b5672: 00000000014040c0 (0x14040c0)
[72159.837219] 00000000c6b00597: 0000000000000002 (0x2)
[72159.837220] 00000000c56565fd: 0000000000000001 (0x1)
[72159.837221] 000000007f09c40f: ffffbadf81fd8000 (0xffffbadf81fd8000)
[72159.837222] 00000000a4d1c884: ffffbadf81fdc000 (0xffffbadf81fdc000)
[72159.837222] 000000007a45e374: 0000000000000000 ...
[72159.837223] 00000000c12f2add: ffffbadf81fd8000 (0xffffbadf81fd8000)
[72159.837224] 00000000311a0a70: ffffbadf81fdc000 (0xffffbadf81fdc000)
[72159.837225] 000000009422918d: 0000000000000000 ...
[72159.837226] 00000000b8201c67: 0000000000000002 (0x2)
[72159.837227] 00000000d2aaa272: ffff98add92645c0 (0xffff98add92645c0)
[72159.837228] 00000000012a7ea3: 0000010100000000 (0x10100000000)
[72159.837229] 0000000069e59149: ffffbadf81fdbba0 (0xffffbadf81fdbba0)
[72159.837230] 0000000070fa505d: ffffbadf81fdb780 (0xffffbadf81fdb780)
[72159.837246] 0000000019af4aa6: ffffffffc0f27656 (_nv002516kms+0x16/0x30 [nvidia_modeset])
[72159.837247] 000000004f8e4e77: 0000000000000000 ...
[72159.837248] 00000000d8da8a7e: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837249] 000000006014b4ea: 0000000000000286 (0x286)
[72159.837249] 0000000044d60e45: 0000000000000000 ...
[72159.837250] 000000009e472fcd: 00000000014040c0 (0x14040c0)
[72159.837251] 000000001389d787: ffffbadf81fdba68 (0xffffbadf81fdba68)
[72159.837252] 000000006e072423: ffffbadf81fdba68 (0xffffbadf81fdba68)
[72159.837253] 00000000f3ea8ac9: ffffbadf81fdb880 (0xffffbadf81fdb880)
[72159.837255] 00000000703a1a49: ffffffff92631dd4 (show_stack+0x34/0x50)
[72159.837256] 0000000005116356: ffffbadf81fdb8a0 (0xffffbadf81fdb8a0)
[72159.837259] 00000000475fe035: ffffffff92f98735 (dump_stack+0x63/0x8e)
[72159.837260] 00000000db74397c: 0000000000000000 ...
[72159.837261] 00000000e3be1c7b: ffff98add92645c0 (0xffff98add92645c0)
[72159.837262] 00000000a5e6e671: ffffbadf81fdb950 (0xffffbadf81fdb950)
[72159.837264] 0000000070f951b6: ffffffff927e228f (warn_alloc+0xff/0x1a0)
[72159.837265] 00000000f4cae554: 0000000000000000 ...
[72159.837266] 00000000990a4f10: ffffbadf81fdb960 (0xffffbadf81fdb960)
[72159.837268] 000000000a24249f: ffffffff936e7b30 (.LC14+0x68/0x970)
[72159.837269] 00000000aec7895d: 014040c09280d0a9 (0x14040c09280d0a9)
[72159.837271] 00000000f6b48413: ffffffff936e7b30 (.LC14+0x68/0x970)
[72159.837272] 00000000eb466048: ffffbadf81fdb8e0 (0xffffbadf81fdb8e0)
[72159.837273] 00000000b819a208: 0000000000000018 (0x18)
[72159.837274] 000000009c23c3f1: ffffbadf81fdb960 (0xffffbadf81fdb960)
[72159.837275] 00000000bd008b4c: ffffbadf81fdb900 (0xffffbadf81fdb900)
[72159.837275] 00000000242a8ad0: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837276] 0000000038e34d8f: 0000000000000040 (0x40)
[72159.837277] 00000000a9b96527: ffffbadf81fdb950 (0xffffbadf81fdb950)
[72159.837280] 00000000bbf39cf8: ffffffff927e2001 (__alloc_pages_direct_compact+0x51/0x100)
[72159.837281] 00000000ae9e1f86: 0000000000000004 (0x4)
[72159.837282] 00000000ae1df03c: 0000000000000080 (0x80)
[72159.837283] 000000006ed2e53e: 0000000000000f7d (0xf7d)
[72159.837284] 000000002a8cd130: 0000000000000001 (0x1)
[72159.837285] 00000000a4234dcf: ffffbadf81fdb8d0 (0xffffbadf81fdb8d0)
[72159.837285] 000000006e517af9: 0000000000000040 (0x40)
[72159.837286] 00000000f46500ee: 00000000014040c0 (0x14040c0)
[72159.837287] 000000000f6f65f2: ffffbadf81fdba58 (0xffffbadf81fdba58)
[72159.837290] 00000000cb573a72: ffffffff927e30f5 (__alloc_pages_slowpath+0xdc5/0xe00)
[72159.837291] 00000000d6b204ee: 00000050d42b9000 (0x50d42b9000)
[72159.837292] 00000000729d7bc6: 0000004000000004 (0x4000000004)
[72159.837293] 00000000e8ec15f7: 0000001000000000 (0x1000000000)
[72159.837293] 0000000085ddf400: 0000000000000000 ...
[72159.837294] 00000000df48305d: 00000000010240c0 (0x10240c0)
[72159.837295] 00000000b148e339: 0040000001000000 (0x40000001000000)
[72159.837296] 00000000af1ecdb9: 0000000000000290 (0x290)
[72159.837297] 000000006c5052e6: 0000000000000260 (0x260)
[72159.837297] 0000000056eb66e4: 0000000000000000 ...
[72159.837298] 0000000071ef356b: 00000000000000bc (0xbc)
[72159.837299] 00000000a08a1c34: 0000000000000001 (0x1)
[72159.837300] 00000000efb81b75: ffff98ad00000040 (0xffff98ad00000040)
[72159.837301] 00000000116be34c: 00000040014040c0 (0x40014040c0)
[72159.837302] 00000000886c9315: ffff98adf6fd4f28 (0xffff98adf6fd4f28)
[72159.837303] 00000000974d1b13: 00000010014040c0 (0x10014040c0)
[72159.837304] 000000002455f00d: ffff98adf6fd4d00 (0xffff98adf6fd4d00)
[72159.837305] 00000000b08ecea6: 0000000000000004 (0x4)
[72159.837306] 000000000c6b3407: 00000002f6fd4ef8 (0x2f6fd4ef8)
[72159.837306] 0000000094348aba: 0000000000000000 ...
[72159.837307] 0000000099709c21: 0000000000000004 (0x4)
[72159.837308] 00000000e5d06225: ffffffff00000000 (0xffffffff00000000)
[72159.837309] 000000008aae1cb2: ffff98adf6fd60b0 (0xffff98adf6fd60b0)
[72159.837310] 0000000000fb2841: ffffffff00000001 (0xffffffff00000001)
[72159.837311] 000000006eee881b: ffff98add9346c10 (0xffff98add9346c10)
[72159.837312] 0000000064b90b77: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837313] 00000000f243847f: 00000000014040c0 (0x14040c0)
[72159.837314] 00000000479e6608: 0000000000000004 (0x4)
[72159.837314] 000000008e364833: 00000000014040c0 (0x14040c0)
[72159.837315] 00000000816ecf9c: 0000000000000000 ...
[72159.837316] 00000000753ba7a1: ffffbadf81fdbac0 (0xffffbadf81fdbac0)
[72159.837318] 00000000357c1f54: ffffffff927e33ca (__alloc_pages_nodemask+0x29a/0x2c0)
[72159.837319] 0000000073b374a9: ffff98adf6fd6080 (0xffff98adf6fd6080)
[72159.837320] 000000001123fc24: 0000000000000000 ...
[72159.837321] 00000000ff2614a7: ffff98adf6fd6080 (0xffff98adf6fd6080)
[72159.837322] 0000000077de0ee5: 0000000200000000 (0x200000000)
[72159.837322] 0000000054df12f2: 0000000000000000 ...
[72159.837323] 000000003d6b64f9: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837326] 00000000db1db7cd: ffffffff940804c0 (policy_zone+0x20/0x20)
[72159.837327] 00000000e40c91fa: 00000000014040c0 (0x14040c0)
[72159.837328] 0000000076e063cb: 0000000000000004 (0x4)
[72159.837328] 00000000c4cf3855: 0000000000000000 ...
[72159.837329] 00000000c16f750f: 0000000000000004 (0x4)
[72159.837330] 00000000c16eb8de: ffffbadf81fdbaf0 (0xffffbadf81fdbaf0)
[72159.837333] 000000007efc5b7b: ffffffff92841f9a (alloc_pages_current+0x6a/0xe0)
[72159.837334] 00000000004e678c: 000000000000ec10 (0xec10)
[72159.837335] 0000000016ac40aa: 000000000000ec10 (0xec10)
[72159.837336] 000000006cd53b13: 00000000014000c0 (0x14000c0)
[72159.837337] 00000000684a6045: 000000000000ec10 (0xec10)
[72159.837338] 000000005404330f: ffffbadf81fdbb00 (0xffffbadf81fdbb00)
[72159.837341] 00000000063077d0: ffffffff928072d8 (kmalloc_order+0x18/0x40)
[72159.837341] 00000000ad87174f: ffffbadf81fdbb40 (0xffffbadf81fdbb40)
[72159.837344] 000000007be769ee: ffffffff92807324 (kmalloc_order_trace+0x24/0xb0)
[72159.837345] 00000000b16d6843: 00000000000042c4 (0x42c4)
[72159.837346] 00000000fca5f304: 000000000000ec10 (0xec10)
[72159.837347] 0000000097c73818: 0000000000000001 (0x1)
[72159.837347] 00000000bae7b272: 00000000014000c0 (0x14000c0)
[72159.837348] 00000000c06f7084: 000000000000ec10 (0xec10)
[72159.837361] 0000000047438816: ffffffffc0eecd40 (_nv000489kms+0x50/0x50 [nvidia_modeset])
[72159.837362] 000000000433541f: ffffbadf81fdbb80 (0xffffbadf81fdbb80)
[72159.837365] 00000000059915a4: ffffffff928519d9 (__kmalloc+0x209/0x220)
[72159.837366] 000000003e787778: ffff98adf6fd4f38 (0xffff98adf6fd4f38)
[72159.837367] 00000000873a3a06: 000000000000ec10 (0xec10)
[72159.837368] 0000000019a0202d: 0000000000000001 (0x1)
[72159.837369] 0000000050ea7fa8: ffff98ab097d8008 (0xffff98ab097d8008)
[72159.837369] 0000000032988644: ffff98ad1d423008 (0xffff98ad1d423008)
[72159.837382] 00000000c3a1b408: ffffffffc0eecd40 (_nv000489kms+0x50/0x50 [nvidia_modeset])
[72159.837383] 0000000054c05658: ffffbadf81fdbba0 (0xffffbadf81fdbba0)
[72159.837396] 0000000098098b14: ffffffffc0eea375 (nvkms_alloc+0x25/0x60 [nvidia_modeset])
[72159.837397] 0000000037bd7de1: 000000000000ec10 (0xec10)
[72159.837397] 00000000bcabdd32: ffff98adddc59508 (0xffff98adddc59508)
[72159.837398] 00000000f3fecc0f: 0000000000000009 (0x9)
[72159.837415] 00000000b1a89344: ffffffffc0f27656 (_nv002516kms+0x16/0x30 [nvidia_modeset])
[72159.837416] 00000000232e9219: ffff98ad1d421008 (0xffff98ad1d421008)
[72159.837434] 000000009a595bd5: ffffffffc0f1d9b8 (_nv002623kms+0x68/0x1f70 [nvidia_modeset])
[72159.837435] 000000005ceecfb7: ffff98aac2758008 (0xffff98aac2758008)
[72159.837436] 00000000c0824c87: ffff98adf6fd4ef8 (0xffff98adf6fd4ef8)
[72159.837437] 0000000067b634eb: ffff98aac275dd60 (0xffff98aac275dd60)
[72159.837437] 000000001e3724e1: 0000000000000000 ...
[72159.837438] 00000000021d87c7: 0000000000000003 (0x3)
[72159.837439] 000000008afc2a4f: ffffffff00000000 (0xffffffff00000000)
[72159.837440] 00000000a006a652: ffff98adf6fd6080 (0xffff98adf6fd6080)
[72159.837441] 0000000052d76add: ffffbadf00000001 (0xffffbadf00000001)
[72159.837443] 00000000dab37fd2: ffffffff92fb746c (schedule+0x2c/0x80)
[72159.837444] 00000000316bd19f: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837445] 0000000024fec65a: 00000000014040c0 (0x14040c0)
[72159.837446] 000000004c7b2aa9: 0000000000000003 (0x3)
[72159.837447] 00000000737bc634: 00000000014040c0 (0x14040c0)
[72159.837448] 00000000e98c6429: ffff98ad1d421008 (0xffff98ad1d421008)
[72159.837448] 00000000132d4d99: 0000000000000001 (0x1)
[72159.837449] 000000001ee88c3f: ffffbadf81fdbca0 (0xffffbadf81fdbca0)
[72159.837452] 00000000593e7b09: ffffffff927e324c (__alloc_pages_nodemask+0x11c/0x2c0)
[72159.837453] 00000000758775d1: ffff98adf6fd6080 (0xffff98adf6fd6080)
[72159.837453] 000000009980d11c: 0000000000000000 ...
[72159.837454] 000000009835fb44: ffff98adf6fd6080 (0xffff98adf6fd6080)
[72159.837455] 00000000871179ce: 0000000200000000 (0x200000000)
[72159.837456] 0000000057cf9210: 0000000000000000 ...
[72159.837457] 000000001a080211: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837459] 000000006c2e35c5: ffffffff940804c0 (policy_zone+0x20/0x20)
[72159.837460] 000000000c28701c: 00000000014040c0 (0x14040c0)
[72159.837461] 00000000af50dabb: 0000000000000003 (0x3)
[72159.837461] 000000000f98d43a: 0000000000000000 ...
[72159.837462] 00000000e0e1535f: 0000000000000003 (0x3)
[72159.837463] 000000004f48c6c1: ffffbadf81fdbcd0 (0xffffbadf81fdbcd0)
[72159.837466] 0000000004a2529b: ffffffff92841f9a (alloc_pages_current+0x6a/0xe0)
[72159.837467] 000000008811fe9f: 0000000000005f88 (0x5f88)
[72159.837468] 0000000090cb6526: 0000000000005f88 (0x5f88)
[72159.837469] 0000000039c9460f: 00000000014000c0 (0x14000c0)
[72159.837469] 0000000022658aa1: 0000000000005f88 (0x5f88)
[72159.837470] 000000002dab2ba3: ffffbadf81fdbce0 (0xffffbadf81fdbce0)
[72159.837473] 000000007747001a: ffffffff928072d8 (kmalloc_order+0x18/0x40)
[72159.837474] 0000000032f0eb0d: ffffbadf81fdbd20 (0xffffbadf81fdbd20)
[72159.837477] 0000000083896987: ffffffff92807324 (kmalloc_order_trace+0x24/0xb0)
[72159.837477] 000000004e8d4212: 0000000000000000 ...
[72159.837478] 000000007d3106ad: 0000000000005f88 (0x5f88)
[72159.837479] 00000000135d2126: 0000000000000001 (0x1)
[72159.837480] 00000000b8beb992: 00000000014000c0 (0x14000c0)
[72159.837480] 00000000727da50f: 0000000000005f88 (0x5f88)
[72159.837493] 000000009a815414: ffffffffc0eecd40 (_nv000489kms+0x50/0x50 [nvidia_modeset])
[72159.837494] 000000000e6504aa: 0000000000000000 ...
[72159.837495] 0000000042ea5c9f: ffff98addd6caa00 (0xffff98addd6caa00)
[72159.837496] 0000000060390052: 0000000000000003 (0x3)
[72159.837497] 0000000053314335: ffff98aac275db90 (0xffff98aac275db90)
[72159.837509] 00000000d2d1a7c4: ffffffffc0eed251 (_nv000618kms+0x31/0xe0 [nvidia_modeset])
[72159.837510] 00000000d75680cd: 0000000892aac45e (0x892aac45e)
[72159.837511] 00000000f3091313: ffff98aac2758008 (0xffff98aac2758008)
[72159.837512] 00000000aa1bd863: 0000000000005d58 (0x5d58)
[72159.837513] 0000000032ba0128: 0000000000000001 (0x1)
[72159.837514] 000000001829ce9b: 0000000000000009 (0x9)
[72159.837515] 00000000aff7503d: ffff98adddc59508 (0xffff98adddc59508)
[72159.837516] 0000000071942e2a: 00007ffdd199dc50 (0x7ffdd199dc50)
[72159.837537] 00000000688009ca: ffffffffc0fa3220 (_nv002112kms+0x80/0xfffffffffffe2e60 [nvidia_modeset])
[72159.837550] 00000000cdc511be: ffffffffc0eecd40 (_nv000489kms+0x50/0x50 [nvidia_modeset])
[72159.837563] 000000003d0c73ac: ffffffffc0eee696 (nvKmsIoctl+0x96/0x1d0 [nvidia_modeset])
[72159.837564] 00000000db0e45e5: ffff98aac2758008 (0xffff98aac2758008)
[72159.837564] 000000006a8007a5: 0000000000005f80 (0x5f80)
[72159.837565] 000000006f41b729: 0000000000000292 (0x292)
[72159.837566] 0000000074a7075a: ffff98ade2ffc540 (0xffff98ade2ffc540)
[72159.837567] 00000000dc56f576: ffffbadf81fdbe10 (0xffffbadf81fdbe10)
[72159.837568] 0000000017510e93: 0000000000000009 (0x9)
[72159.837569] 0000000071cb9d69: 00007ffdd199dc50 (0x7ffdd199dc50)
[72159.837570] 00000000b498e80d: 0000000000005f80 (0x5f80)
[72159.837571] 000000007411a24d: 00007ffdd1997c90 (0x7ffdd1997c90)
[72159.837584] 00000000d6935a55: ffffffffc0eeaf12 (nvkms_ioctl_common+0x42/0x80 [nvidia_modeset])
[72159.837604] 000000000bb35416: ffffffffc0fdcce0 (nvkms_kthread_q+0x40/0xfffffffffffa9360 [nvidia_modeset])
[72159.837605] 00000000b62d8fa8: ffff98ade2ffc540 (0xffff98ade2ffc540)
[72159.837606] 0000000047661e7e: ffff98ad6c05d200 (0xffff98ad6c05d200)
[72159.837607] 00000000152a7ad4: 00000000c0106d00 (0xc0106d00)
[72159.837608] 00000000ba44262e: ffffbadf81fdbe50 (0xffffbadf81fdbe50)
[72159.837621] 00000000a4a5bf30: ffffffffc0eeb017 (nvkms_ioctl+0xc7/0x100 [nvidia_modeset])
[72159.837622] 0000000063f4fbae: ffff98ade27e7d00 (0xffff98ade27e7d00)
[72159.837623] 00000000ea4f0178: 00005f8000000009 (0x5f8000000009)
[72159.837624] 00000000a32cabf9: 00007ffdd199dc50 (0x7ffdd199dc50)
[72159.837625] 000000005b094930: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837626] 00000000f3394f7b: ffff98add70953e0 (0xffff98add70953e0)
[72159.837627] 00000000b106486a: 00007ffdd1997c90 (0x7ffdd1997c90)
[72159.837627] 00000000504af8d4: ffffbadf81fdbe60 (0xffffbadf81fdbe60)
[72159.837890] 00000000fbcc63b2: ffffffffc124a082 (nvidia_frontend_unlocked_ioctl+0x42/0x50 [nvidia])
[72159.837891] 00000000f160382e: ffffbadf81fdbee8 (0xffffbadf81fdbee8)
[72159.837894] 0000000008ca4e7d: ffffffff92896f78 (do_vfs_ioctl+0xa8/0x630)
[72159.837899] 000000009d7c5d88: ffffffff92819a51 (handle_mm_fault+0xb1/0x210)
[72159.837900] 0000000050e0385c: 00007ffdd19a0000 (0x7ffdd19a0000)
[72159.837901] 00000000afa769bc: 0000000000000007 (0x7)
[72159.837902] 000000007302cbe3: 00007ffdd19a0000 (0x7ffdd19a0000)
[72159.837903] 00000000a6ac4286: ffffbadf81fdbf58 (0xffffbadf81fdbf58)
[72159.837905] 00000000e6abe627: 0000000000000055 (0x55)
[72159.837906] 00000000e97d22c1: ffffbadf81fdbf18 (0xffffbadf81fdbf18)
[72159.837910] 00000000664bf5ba: ffffffff92677341 (__do_page_fault+0x2a1/0x4b0)
[72159.837911] 0000000055004a4a: 0000000000000000 ...
[72159.837912] 00000000db3a5c54: 5fa1e98e5be74d00 (0x5fa1e98e5be74d00)
[72159.837913] 00000000b48b8f4b: ffff98ad6c05d201 (0xffff98ad6c05d201)
[72159.837915] 00000000265ca3b8: ffff98ad6c05d200 (0xffff98ad6c05d200)
[72159.837916] 0000000062d71050: 0000000000000012 (0x12)
[72159.837917] 00000000f17841b9: 00000000c0106d00 (0xc0106d00)
[72159.837918] 00000000901ebf2b: 00007ffdd1997c90 (0x7ffdd1997c90)
[72159.837919] 000000005715f6c9: ffffbadf81fdbf28 (0xffffbadf81fdbf28)
[72159.837921] 000000004324cfb6: ffffffff92897579 (SyS_ioctl+0x79/0x90)
[72159.837922] 00000000bf5edd48: 0000000000000007 (0x7)
[72159.837924] 000000004cc23c2b: ffffbadf81fdbf58 (0xffffbadf81fdbf58)
[72159.837924] 000000001b2f518e: 0000000000000000 ...
[72159.837926] 00000000bc05a0c6: ffffbadf81fdbf48 (0xffffbadf81fdbf48)
[72159.837930] 0000000065a98b18: ffffffff92603bb3 (do_syscall_64+0x73/0x130)
[72159.837930] 00000000c2bca447: 0000000000000000 ...
[72159.837934] 00000000755d5bf3: ffffffff93000081 (entry_SYSCALL_64_after_hwframe+0x3d/0xa2)
[72159.837935] 000000008edfbe18: 0000000000000000 ...
[72159.837936] 000000005d541ab8: 000055e57f001cb0 (0x55e57f001cb0)
[72159.837938] 00000000ddfdc009: 0000000000000012 (0x12)
[72159.837939] 000000005955e15f: 00007ffdd199dc50 (0x7ffdd199dc50)
[72159.837940] 00000000676e17cf: 0000000000003246 (0x3246)
[72159.837941] 000000006fe1ca25: 0000000000000000 ...
[72159.837942] 000000001116061e: 00007ffdd199dc50 (0x7ffdd199dc50)
[72159.837943] 00000000c86a5e15: 0000000000000001 (0x1)
[72159.837944] 00000000b503b364: ffffffffffffffda (0xffffffffffffffda)
[72159.837945] 0000000083f3cd47: 00007f25158f35d7 (0x7f25158f35d7)
[72159.837946] 00000000697612a3: 00007ffdd1997c90 (0x7ffdd1997c90)
[72159.837948] 000000000d396aae: 00000000c0106d00 (0xc0106d00)
[72159.837949] 000000003ca171fb: 0000000000000012 (0x12)
[72159.837950] 00000000d2dc5406: 0000000000000010 (0x10)
[72159.837951] 000000001fb83c84: 00007f25158f35d7 (0x7f25158f35d7)
[72159.837952] 00000000846bf60d: 0000000000000033 (0x33)
[72159.837953] 0000000024441cf4: 0000000000003246 (0x3246)
[72159.837954] 0000000020346520: 00007ffdd1997c88 (0x7ffdd1997c88)
[72159.837955] 000000004c391274: 000000000000002b (0x2b)
[72159.837981]  ? _nv002623kms+0x68/0x1f70 [nvidia_modeset]
[72159.837989]  ? schedule+0x2c/0x80
[72159.837993]  ? __alloc_pages_nodemask+0x11c/0x2c0
[72159.837998]  ? alloc_pages_current+0x6a/0xe0
[72159.838002]  ? kmalloc_order+0x18/0x40
[72159.838006]  ? kmalloc_order_trace+0x24/0xb0
[72159.838023]  ? _nv000489kms+0x50/0x50 [nvidia_modeset]
[72159.838039]  ? _nv000618kms+0x31/0xe0 [nvidia_modeset]
[72159.838056]  ? _nv000489kms+0x50/0x50 [nvidia_modeset]
[72159.838073]  ? nvKmsIoctl+0x96/0x1d0 [nvidia_modeset]
[72159.838090]  ? nvkms_ioctl_common+0x42/0x80 [nvidia_modeset]
[72159.838106]  ? nvkms_ioctl+0xc7/0x100 [nvidia_modeset]
[72159.838335]  ? nvidia_frontend_unlocked_ioctl+0x42/0x50 [nvidia]
[72159.838339]  ? do_vfs_ioctl+0xa8/0x630
[72159.838343]  ? handle_mm_fault+0xb1/0x210
[72159.838347]  ? __do_page_fault+0x2a1/0x4b0
[72159.838350]  ? SyS_ioctl+0x79/0x90
[72159.838354]  ? do_syscall_64+0x73/0x130
[72159.838358]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[72159.838361] Mem-Info:
[72159.838367] active_anon:897688 inactive_anon:169930 isolated_anon:0
                active_file:1062236 inactive_file:836789 isolated_file:0
                unevictable:12 dirty:340 writeback:304 unstable:0
                slab_reclaimable:976178 slab_unreclaimable:36116
                mapped:88594 shmem:28673 pagetables:14067 bounce:0
                free:49370 free_pcp:0 free_cma:0
[72159.838372] Node 0 active_anon:3590752kB inactive_anon:679720kB active_file:4248944kB inactive_file:3347156kB unevictable:48kB isolated(anon):0kB isolated(file):0kB mapped:354376kB dirty:1360kB writeback:1216kB shmem:114692kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no
[72159.838374] Node 0 DMA free:15860kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15988kB managed:15884kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[72159.838379] lowmem_reserve[]: 0 3019 15902 15902 15902
[72159.838385] Node 0 DMA32 free:91800kB min:12816kB low:16020kB high:19224kB active_anon:579728kB inactive_anon:127408kB active_file:580372kB inactive_file:445716kB unevictable:0kB writepending:200kB present:3215944kB managed:3150092kB mlocked:0kB kernel_stack:544kB pagetables:2232kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[72159.838390] lowmem_reserve[]: 0 0 12882 12882 12882
[72159.838395] Node 0 Normal free:89820kB min:54696kB low:68368kB high:82040kB active_anon:3011024kB inactive_anon:552312kB active_file:3668484kB inactive_file:2901440kB unevictable:48kB writepending:2376kB present:13484032kB managed:13198208kB mlocked:48kB kernel_stack:13168kB pagetables:54036kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[72159.838401] lowmem_reserve[]: 0 0 0 0 0
[72159.838405] Node 0 DMA: 1*4kB (U) 2*8kB (U) 2*16kB (U) 0*32kB 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15860kB
[72159.838424] Node 0 DMA32: 4231*4kB (UMEH) 1194*8kB (UMEH) 3656*16kB (UMEH) 196*32kB (UMEH) 7*64kB (H) 4*128kB (H) 1*256kB (H) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 92460kB
[72159.838441] Node 0 Normal: 9562*4kB (UME) 5305*8kB (UME) 499*16kB (UMEH) 60*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 90592kB
[72159.838458] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[72159.838460] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[72159.838461] 1935996 total pagecache pages
[72159.838463] 8228 pages in swap cache
[72159.838465] Swap cache stats: add 170297, delete 162070, find 391700/409129
[72159.838466] Free swap  = 1606652kB
[72159.838467] Total swap = 2097148kB
[72159.838468] 4178991 pages RAM
[72159.838469] 0 pages HighMem/MovableOnly
[72159.838470] 87945 pages reserved
[72159.838471] 0 pages cma reserved
[72159.838472] 0 pages hwpoisoned
[72159.838479] BUG: unable to handle kernel paging request at 0000000000006f80
[72159.838517] IP: _nv002472kms+0x60/0x100 [nvidia_modeset]
[72159.838520] PGD 0 P4D 0 
[72159.838525] Oops: 0000 [#1] SMP PTI
[72159.838528] Modules linked in: rfcomm rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache arc4 cmac bnep snd_hda_codec_hdmi nvidia_uvm(OE) nvidia_drm(POE) eeepc_wmi nvidia_modeset(POE) asus_wmi sparse_keymap mxm_wmi wmi_bmof intel_rapl nvidia(POE) x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel kvm irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_hda_codec_realtek pcbc snd_hda_codec_generic nls_iso8859_1 aesni_intel snd_hda_intel aes_x86_64 crypto_simd snd_seq_midi snd_hda_codec glue_helper snd_seq_midi_event cryptd intel_cstate snd_hda_core joydev intel_rapl_perf snd_hwdep snd_rawmidi ath10k_pci ath10k_core ath snd_seq mac80211 snd_pcm drm_kms_helper snd_seq_device snd_timer cfg80211 snd drm btusb btrtl btbcm btintel ipmi_devintf bluetooth ipmi_msghandler fb_sys_fops
[72159.838593]  syscopyarea sysfillrect soundcore sysimgblt mei_me input_leds ecdh_generic shpchp mei mac_hid wmi acpi_pad sch_fq_codel sunrpc parport_pc ppdev lp parport ip_tables x_tables autofs4 btrfs xor zstd_compress raid6_pq dm_mirror dm_region_hash dm_log uas usb_storage hid_logitech_hidpp hid_logitech_dj hid_generic usbhid hid e1000e(OE) ptp pps_core nvme ahci nvme_core libahci video
[72159.838634] CPU: 4 PID: 25942 Comm: Xorg Tainted: P           OE    4.15.0-66-generic #75-Ubuntu
[72159.838636] Hardware name: System manufacturer System Product Name/Z170I PRO GAMING, BIOS 3610 03/29/2018
[72159.838666] RIP: 0010:_nv002472kms+0x60/0x100 [nvidia_modeset]
[72159.838669] RSP: 0018:ffffbadf81fdbb80 EFLAGS: 00010202
[72159.838672] RAX: 0000000000000004 RBX: 0000000000006f80 RCX: 0000000000000004
[72159.838675] RDX: ffff98ad1d425318 RSI: 0000000000006f80 RDI: ffff98ad1d423008
[72159.838677] RBP: 0000000000000000 R08: 0000000000001000 R09: 0000000000000000
[72159.838679] R10: 0000000000000004 R11: 0000000000000001 R12: 0000000000006f80
[72159.838681] R13: 0000000000006f80 R14: ffff98ad1d423008 R15: 0000000000000001
[72159.838684] FS:  00007f25184f3600(0000) GS:ffff98adf6d00000(0000) knlGS:0000000000000000
[72159.838687] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[72159.838690] CR2: 0000000000006f80 CR3: 000000041c768005 CR4: 00000000003606e0
[72159.838692] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[72159.838695] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[72159.838696] Call Trace:
[72159.838723]  ? _nv002623kms+0x3aa/0x1f70 [nvidia_modeset]
[72159.838727]  ? __schedule+0x840/0x880
[72159.838732]  ? __alloc_pages_nodemask+0x11c/0x2c0
[72159.838738]  ? alloc_pages_current+0x6a/0xe0
[72159.838743]  ? kmalloc_order+0x18/0x40
[72159.838747]  ? kmalloc_order_trace+0x24/0xb0
[72159.838765]  ? _nv000489kms+0x50/0x50 [nvidia_modeset]
[72159.838783]  ? _nv000618kms+0x31/0xe0 [nvidia_modeset]
[72159.838800]  ? _nv000489kms+0x50/0x50 [nvidia_modeset]
[72159.838818]  ? nvKmsIoctl+0x96/0x1d0 [nvidia_modeset]
[72159.838836]  ? nvkms_ioctl_common+0x42/0x80 [nvidia_modeset]
[72159.838853]  ? nvkms_ioctl+0xc7/0x100 [nvidia_modeset]
[72159.839064]  ? nvidia_frontend_unlocked_ioctl+0x42/0x50 [nvidia]
[72159.839068]  ? do_vfs_ioctl+0xa8/0x630
[72159.839073]  ? handle_mm_fault+0xb1/0x210
[72159.839078]  ? __do_page_fault+0x2a1/0x4b0
[72159.839081]  ? SyS_ioctl+0x79/0x90
[72159.839086]  ? do_syscall_64+0x73/0x130
[72159.839091]  ? entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[72159.839094] Code: 2a eb 40 0f 1f 84 00 00 00 00 00 48 c7 03 00 00 00 00 c6 43 08 00 41 8b 86 d8 00 00 00 83 c5 01 48 81 c3 d0 03 00 00 39 e8 76 18 <48> 8b 3b 48 85 ff 74 ea 80 7b 08 00 75 d2 e8 fd d3 ff ff eb cb 
[72159.839174] RIP: _nv002472kms+0x60/0x100 [nvidia_modeset] RSP: ffffbadf81fdbb80
[72159.839176] CR2: 0000000000006f80
[72159.839180] ---[ end trace 510621728cd82cca ]---

and the output from nvidia-bug-report: https://www.dropbox.com/s/emnie4qumzbfhte/nvidia-bug-report.log.gz?dl=0

Looks like the same bug some other users reported in conjunction with vm/hugepages. Not enough continuous system memory available and the driver fails to handle this gracefully and crashes. I think/hope nvidia already investigates this.
Meanwhile, you could work around it by extracting the edid, and use the ConnectedMonitor and CustomEDID options to make the driver believe the monitor is available all the time so it doesn’t have to allocate mem on switching back.

Thanks @generix i’ll give that a shot and report back. I also ordered and HDMI switch to see if that works better than using DP. Some people report HDMI KVM’s don’t exhibit this behavior because they emulate the display and EDID so the computer thinks it’s always connected to the monitor.