If I have two GPU cards, how can I use the GPUs' constant memory properly?

Accelerated Computing CUDA CUDA Programming and Performance
1

Do I need to define two arrays both with constant qualifiers in global scope for each of the GPUs?

2

No, a constant declaration is instantiated on all GPUs that are currently visible to the process.

However, you will need to populate each one individually:

cudaSetDevice(0);
cudaMemcpyToSymbol(…);
cudaSetDevice(1);
cudaMemcpyToSymbol(…);
…