So per thread-starting post this system uses dual Intel® Xeon® Silver 4316 Processors, each with 8 DDR4-3200 channels and 64 PCIe 4 lanes. Are all DDR4 channels populated?
Yes, correct. I think all channels are populated. There are 64 RAM slots and 8 of them are filled according to the manual.
If the system presents more x16 links than that, the PLX chip would appear to multiplex them onto the physical lanes available.
This is indeed the case.
What is the actual link configuration ues by the RTX A4000?
Timestamp : Wed Jan 18 09:35:21 2023
Driver Version : 525.60.11
CUDA Version : 12.0
Attached GPUs : 10
GPU 00000000:4F:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:52:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:53:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:56:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:57:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:CE:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:D1:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:D2:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:D5:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
GPU 00000000:D6:00.0
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Device Current : 4
Device Max : 4
Host Max : 4
Link Width
Max : 16x
Current : 16x
By the way, are you sure the system uses DDR4-3200 memory? Intel’s specifications for the Xeon Silver 4316 processor list DDR4-2667 (Intel Xeon Silver 4316 Processor 30M Cache 2.30 GHz Product Specifications), which would provide about 166 GB/sec of usable system memory bandwidth per socket.
You’re right! I was looking at the reported “Speed” in dmidecode
but apparently there’s a section “Configured Speed” and the reported speed there is 2666.