When I have kernel which is able to occupy entire k80, is it going to work on two gpu?
a single kernel will not occupy both GK210 devices. The K80 GPU appears to the system much like 2 K40 devices would. To make full utilization of the K80, you must have a multi-GPU aware application.
Who is managing these two memory area? (It looks like kind of numa architecture?
Each GK210 device is independent and independently manages it’s own 12GB memory.
If hardware or cuda manages everything, what if some part of my kernel wants to access data from
both memory area ?
Memory from one GK210 is visible to the other if a mechanism like P2P is used, just as it would be if two K40 devices were used.
If I launch dynamic parallelism, is hardware or cuda going to take account data locality?(maybe this is no :) )
CUDA dynamic parallelism runs on a single GPU device, so in the case of K80 that would be a single GK210 device. It does not automatically use the resources of the other GK210 device.
Shared memory is bigger. Does it reside distributed on two gpu?
Each GK210 device (compute capability 3.7) has an expanded shared memory system. The shared memory of one GK210 is not usable by the other.
What’s difference between using 2xK40 and 1 K80 ?
Conceptually, the usage is quite similar from a programmer’s perspective.