Hi, guys!
I am learning CUDA today, and I want to know whether it has been explicitly described in CUDA documentation that 2D-array is stored in column major?
Your answer and guide will be appreciated!
There is no 2D-array or multidimensional array in CUDA C++ that is any different than C++. The storage behavior of C-style arrays in CUDA follows the C++ specification. And, for what it is worth, that is not a column-major storage format/specification.
If you are asking about CUDA Fortran, then my answer would change, but it follows the language behavior, and in that case it is column-major.
CUBLAS might be considered an unusual case, as by default it expects column-major storage, and you can find options for storage patterns in the CUBLAS docs.
1 Like
Hi, @Robert_Crovella !
Thanks sincerely for your response.
I got this impression from this presentation, How CUDA Programming Works, whose PDF showed it as,
From this, I guessed that, if a 2D-array has been transferred into a GPU, the it would be stored in column major by default with nvcc.
Now I think that this behavior might depend on the specifical compiler, like gcc and so on.
It says row major layout right at the bottom of the diagram.
The diagram is intending to communicate the benefits of coalesced access across a warp. The blue line represents a coalesced access pattern. The green line represents a non-coalesced pattern, and it is indicated as being slower.
1 Like
Appreciate your detailed explanation!
I think I confused matrix dimensions with array dimensions.
In our knowledge idioms,
Matrix: A[M][N], M is row, and N is column;
But in the slides, array[A][B], first A is column, and second B is row.
So, I thought 2D-array is column-major, which in fact is first array index is column.
No that’s not the case. The diagram is using x to represent column index (i.e. the 2nd subscript), and y to represent row index (the first subscript), which is consistent with syntax for both C++ and fortran.
row-major vs. column-major is not something that can be inspected purely by looking at matrix or array notation or order of subscripts. In Fortran, C++, and those slides, the first subscript gives the row index, and the second subscript gives the column index.
1 Like
Ok, I now can figure it out.
Thanks a lot for correcting my understanding!
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.