Using 8x GTX 1070 on Ubuntu 16.04 results in no display

Hi, all.

My customer is using 8x GTX 1070 on Ubuntu 16.04, and cannot enable display if all 8x GPU are connected. System will output display if 7x GPU are installed, but not 8x GPU…!

Same system with Ubuntu 14.04.x or CentOS 7 will run happily, but just not on Ubuntu 16.04.

From running nvidia-bug-report.sh, I see segmentation fault in Xorg.0.log as shown below:

*** /var/log/Xorg.0.log
*** ls: -rw-r--r-- 1 root root 9204 2016-08-30 16:20:14.663791289 +0900 /var/log/Xorg.0.log
[     4.528] 
X.Org X Server 1.18.3
Release Date: 2016-04-04
[     4.528] X Protocol Version 11, Revision 0
[     4.528] Build Operating System: Linux 3.13.0-85-generic x86_64 Ubuntu
[     4.528] Current Operating System: Linux super 4.4.0-21-generic #37-Ubuntu SMP Mon Apr 18 18:33:37 UTC 2016 x86_64
[     4.528] Kernel command line: BOOT_IMAGE=/vmlinuz-4.4.0-21-generic root=UUID=baa01eb8-7ec8-42f2-8cd1-1a77042bf2c5 ro quiet splash vt.handoff=7
[     4.528] Build Date: 07 April 2016  09:18:50AM
[     4.528] xorg-server 2:1.18.3-1ubuntu2 (For technical support please see http://www.ubuntu.com/support) 
[     4.528] Current version of pixman: 0.33.6
[     4.528] 	Before reporting problems, check http://wiki.x.org
	to make sure that you have the latest version.
[     4.528] Markers: (--) probed, (**) from config file, (==) default setting,
	(++) from command line, (!!) notice, (II) informational,
	(WW) warning, (EE) error, (NI) not implemented, (??) unknown.
[     4.528] (==) Log file: "/var/log/Xorg.0.log", Time: Tue Aug 30 16:20:14 2016
[     4.528] (==) Using system config directory "/usr/share/X11/xorg.conf.d"
[     4.528] (==) No Layout section.  Using the first Screen section.
[     4.528] (==) No screen section available. Using defaults.
[     4.528] (**) |-->Screen "Default Screen Section" (0)
[     4.528] (**) |   |-->Monitor "<default monitor>"
[     4.528] (==) No monitor specified for screen "Default Screen Section".
	Using a default monitor configuration.
[     4.528] (==) Automatically adding devices
[     4.528] (==) Automatically enabling devices
[     4.528] (==) Automatically adding GPU devices
[     4.529] (==) Max clients allowed: 256, resource mask: 0x1fffff
[     4.529] (WW) The directory "/usr/share/fonts/X11/cyrillic" does not exist.
[     4.529] 	Entry deleted from font path.
[     4.529] (WW) The directory "/usr/share/fonts/X11/100dpi/" does not exist.
[     4.529] 	Entry deleted from font path.
[     4.529] (WW) The directory "/usr/share/fonts/X11/75dpi/" does not exist.
[     4.529] 	Entry deleted from font path.
[     4.529] (WW) The directory "/usr/share/fonts/X11/100dpi" does not exist.
[     4.529] 	Entry deleted from font path.
[     4.529] (WW) The directory "/usr/share/fonts/X11/75dpi" does not exist.
[     4.529] 	Entry deleted from font path.
[     4.529] (==) FontPath set to:
	/usr/share/fonts/X11/misc,
	/usr/share/fonts/X11/Type1,
	built-ins
[     4.529] (==) ModulePath set to "/usr/lib/x86_64-linux-gnu/xorg/extra-modules,/usr/lib/xorg/extra-modules,/usr/lib/xorg/modules"
[     4.529] (II) The server relies on udev to provide the list of input devices.
	If no devices become available, reconfigure udev or disable AutoAddDevices.
[     4.529] (II) Loader magic: 0x562d4b9dada0
[     4.529] (II) Module ABI versions:
[     4.529] 	X.Org ANSI C Emulation: 0.4
[     4.529] 	X.Org Video Driver: 20.0
[     4.529] 	X.Org XInput driver : 22.1
[     4.529] 	X.Org Server Extension : 9.0
[     4.529] (++) using VT number 7

[     4.529] (II) systemd-logind: logind integration requires -keeptty and -keeptty was not provided, disabling logind integration
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card1)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card2)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card3)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card4)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card0)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card5)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card6)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card7)
[     4.530] (II) xfree86: Adding drm device (/dev/dri/card8)
[     4.537] (--) PCI:*(0:4:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xc4000000/16777216, 0x23fe0000000/268435456, 0x23ff0000000/33554432, I/O @ 0x00005000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:5:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xc2000000/16777216, 0x23fc0000000/268435456, 0x23fd0000000/33554432, I/O @ 0x00004000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:8:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xc0000000/16777216, 0x23fa0000000/268435456, 0x23fb0000000/33554432, I/O @ 0x00003000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:9:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xbe000000/16777216, 0x23f80000000/268435456, 0x23f90000000/33554432, I/O @ 0x00002000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:12:0:0) 1a03:2000:15d9:0892 rev 48, Mem @ 0xc6000000/16777216, 0xc7000000/131072, I/O @ 0x00006000/128
[     4.537] (--) PCI: (0:132:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xfa000000/16777216, 0x27fe0000000/268435456, 0x27ff0000000/33554432, I/O @ 0x0000e000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:133:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xf8000000/16777216, 0x27fc0000000/268435456, 0x27fd0000000/33554432, I/O @ 0x0000d000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:136:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xf6000000/16777216, 0x27fa0000000/268435456, 0x27fb0000000/33554432, I/O @ 0x0000c000/128, BIOS @ 0x????????/524288
[     4.537] (--) PCI: (0:137:0:0) 10de:17c2:10de:1132 rev 161, Mem @ 0xf4000000/16777216, 0x27f80000000/268435456, 0x27f90000000/33554432, I/O @ 0x0000b000/128, BIOS @ 0x????????/524288
[     4.537] (II) LoadModule: "glx"
[     4.538] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[     4.570] (II) Module glx: vendor="NVIDIA Corporation"
[     4.570] 	compiled for 4.0.2, module version = 1.0.0
[     4.570] 	Module class: X.Org Server Extension
[     4.570] (II) NVIDIA GLX Module  367.44  Wed Aug 17 21:50:26 PDT 2016
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card1
[     4.570] 	loading driver: nvidia
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card2
[     4.570] 	loading driver: nvidia
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card3
[     4.570] 	loading driver: nvidia
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card4
[     4.570] 	loading driver: nvidia
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card5
[     4.570] 	loading driver: nvidia
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card6
[     4.570] 	loading driver: nvidia
[     4.570] (II) Applying OutputClass "nvidia" to /dev/dri/card7
[     4.570] 	loading driver: nvidia
[     4.570] (==) Matched nvidia as autoconfigured driver 0
[     4.570] (==) Matched nvidia as autoconfigured driver 1
[     4.570] (==) Matched nouveau as autoconfigured driver 2
[     4.570] (==) Matched nvidia as autoconfigured driver 3
[     4.570] (==) Matched nvidia as autoconfigured driver 4
[     4.570] (==) Matched nouveau as autoconfigured driver 5
[     4.570] (==) Matched nvidia as autoconfigured driver 6
[     4.570] (==) Matched nvidia as autoconfigured driver 7
[     4.570] (==) Matched nouveau as autoconfigured driver 8
[     4.570] (==) Matched nvidia as autoconfigured driver 9
[     4.570] (==) Matched nvidia as autoconfigured driver 10
[     4.570] (==) Matched nouveau as autoconfigured driver 11
[     4.570] (==) Matched ast as autoconfigured driver 12
[     4.570] (==) Matched nvidia as autoconfigured driver 13
[     4.570] (==) Matched nvidia as autoconfigured driver 14
[     4.570] (==) Matched nouveau as autoconfigured driver 15
[     4.570] (==) Matched nvidia as autoconfigured driver 16
[     4.570] (==) Matched nvidia as autoconfigured driver 17
[     4.570] (==) Matched nouveau as autoconfigured driver 18
[     4.570] (==) Matched nvidia as autoconfigured driver 19
[     4.570] (==) Matched modesetting as autoconfigured driver 20
[     4.570] (EE) 
[     4.570] (EE) Backtrace:
[     4.571] (EE) 0: /usr/lib/xorg/Xorg (xorg_backtrace+0x4e) [0x562d4b74e50e]
[     4.571] (EE) 1: /usr/lib/xorg/Xorg (0x562d4b59c000+0x1b6899) [0x562d4b752899]
[     4.571] (EE) 2: /lib/x86_64-linux-gnu/libc.so.6 (0x7fcdc0477000+0x354a0) [0x7fcdc04ac4a0]
[     4.571] (EE) 3: /lib/x86_64-linux-gnu/libc.so.6 (_IO_vfprintf+0x1b50) [0x7fcdc04c5c80]
[     4.571] (EE) 4: /lib/x86_64-linux-gnu/libc.so.6 (__vsnprintf_chk+0x96) [0x7fcdc058cfd6]
[     4.571] (EE) 5: /usr/lib/xorg/Xorg (Xvscnprintf+0x2e) [0x562d4b75a74e]
[     4.571] (EE) 6: /usr/lib/xorg/Xorg (LogVMessageVerb+0x63) [0x562d4b75c303]
[     4.571] (EE) 7: /usr/lib/xorg/Xorg (xf86Msg+0x9c) [0x562d4b63c4bc]
[     4.571] (EE) 8: /usr/lib/xorg/Xorg (0x562d4b59c000+0xb5ba4) [0x562d4b651ba4]
[     4.571] (EE) 9: /usr/lib/xorg/Xorg (InitOutput+0xb40) [0x562d4b634c10]
[     4.571] (EE) 10: /usr/lib/xorg/Xorg (0x562d4b59c000+0x57a54) [0x562d4b5f3a54]
[     4.571] (EE) 11: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0xf0) [0x7fcdc0497830]
[     4.571] (EE) 12: /usr/lib/xorg/Xorg (_start+0x29) [0x562d4b5ddf59]
[     4.571] (EE) 
[     4.571] (EE) Segmentation fault at address 0x0
[     4.571] (EE) 
Fatal server error:
[     4.571] (EE) Caught signal 11 (Segmentation fault). Server aborting
[     4.571] (EE) 
[     4.571] (EE) 
Please consult the The X.Org Foundation support 
	 at http://wiki.x.org
 for help. 
[     4.571] (EE) Please also check the log file at "/var/log/Xorg.0.log" for additional information.
[     4.571] (EE)

Any advice on how to get this system working on Ubuntu 16.04 would be much appreciated!
nvidia-bug-report.log (1.95 MB)

Are you trying to use one of the GTX 1070 cards for display, or is there a separate display output?

I have no hands-on experience with such systems myself, but from previous forum postings this could well be a BIOS problem. If you post BIOS and mainboard information, someone else might be able to help. Also try googling for your specific mainboard.

Another reason could be insufficient power to drive all 8 cards.

Customer is using all 8x GPU for GPGPU, and use just one of them for display. Refer to below for reference.

nvidia-smi output on CentOS 7 and Ubuntu 14.04.x:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.35                 Driver Version: 367.35                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|        Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+=|
|   0  GeForce GTX 1070    Off  | 0000:04:00.0      On |                  N/A |
| 27%   35C    P8     6W / 151W |    226MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1070    Off  | 0000:05:00.0     Off |                  N/A |
| 27%   36C    P8     6W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1070    Off  | 0000:08:00.0     Off |                  N/A |
| 27%   34C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 1070    Off  | 0000:09:00.0     Off |                  N/A |
| 27%   33C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 1070    Off  | 0000:84:00.0     Off |                  N/A |
| 27%   32C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 1070    Off  | 0000:85:00.0     Off |                  N/A |
| 27%   33C    P8     6W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX 1070    Off  | 0000:88:00.0     Off |                  N/A |
| 27%   39C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX 1070    Off  | 0000:89:00.0     Off |                  N/A |
| 27%   34C    P8     6W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|========================================================|
|    0      1243    G   /usr/lib/xorg/Xorg                             104MiB |
|    0      3190    G   compiz                                         125MiB |
+-----------------------------------------------------------------------------+

nvidia-smi output on Ubuntu 16.04:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 367.35                 Driver Version: 367.35                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+====|
|   0  GeForce GTX 1070    Off  | 0000:04:00.0      On |                  N/A |
| 27%   35C    P8     6W / 151W |    226MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce GTX 1070    Off  | 0000:05:00.0     Off |                  N/A |
| 27%   36C    P8     6W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce GTX 1070    Off  | 0000:08:00.0     Off |                  N/A |
| 27%   34C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce GTX 1070    Off  | 0000:09:00.0     Off |                  N/A |
| 27%   33C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   4  GeForce GTX 1070    Off  | 0000:84:00.0     Off |                  N/A |
| 27%   32C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   5  GeForce GTX 1070    Off  | 0000:85:00.0     Off |                  N/A |
| 27%   33C    P8     6W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   6  GeForce GTX 1070    Off  | 0000:88:00.0     Off |                  N/A |
| 27%   39C    P8     5W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   7  GeForce GTX 1070    Off  | 0000:89:00.0     Off |                  N/A |
| 27%   34C    P8     6W / 151W |      1MiB /  8113MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|========================================================|
|  No running processes found                                                        |
+-----------------------------------------------------------------------------+

Let me check on BIOS… Power supply should be okay, since I’m told that no complaints are seen in IPMI log or OS log.

Wish there was a nvidia-bug-report.sh output from Ubuntu 14.04 for comparison, but the system is tied up, busy running tests…