# Programming Tensor Cores in CUDA 9 **URL:** \ **Category:** Technical Blog\ **Created:** [October 17, 2017, 9:29am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685 "2017-10-17T09:29:32Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![jwitsoe](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/jwitsoe/32/16591_2.png) [@jwitsoe](https://forums.developer.nvidia.com/u/jwitsoe)\ **Post date:** [October 17, 2017, 9:29am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/1 "2017-10-17T09:29:32Z")
Originally published at: [https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/](https://developer.nvidia.com/blog/programming-tensor-cores-cuda-9/) Tensor cores provide a huge boost to convolutions and matrix operations. Tensor cores are programmable using NVIDIA libraries and directly in CUDA C++ code. A defining feature of the new Volta GPU Architecture is its Tensor Cores, which give the Tesla V100 accelerator a peak throughput 12 times the 32-bit floating point throughput of the previous-generation… ---
**Author:** ![anon7279269](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon7279269](https://forums.developer.nvidia.com/u/anon7279269)\ **Post date:** [October 17, 2017, 1:40pm UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/2 "2017-10-17T13:40:30Z")
Tensor-Cores - will it also be available within the next Gamer-GPU (Geforce) ? ---
**Author:** ![anon61728009](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon61728009](https://forums.developer.nvidia.com/u/anon61728009)\ **Post date:** [October 29, 2017, 9:26am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/3 "2017-10-29T09:26:40Z")
That's a good question ! ---
**Author:** ![anon68409244](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon68409244](https://forums.developer.nvidia.com/u/anon68409244)\ **Post date:** [January 1, 2018, 1:51am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/4 "2018-01-01T01:51:04Z")
interesting possibilities for crypto currencies using tensor cores... ---
**Author:** ![anon19689551](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon19689551](https://forums.developer.nvidia.com/u/anon19689551)\ **Post date:** [January 5, 2018, 11:53pm UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/5 "2018-01-05T23:53:05Z")
Which NN framework did you use for Figure4 inference? ---
**Author:** ![anon90255818](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon90255818](https://forums.developer.nvidia.com/u/anon90255818)\ **Post date:** [January 8, 2018, 7:46pm UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/6 "2018-01-08T19:46:07Z")
This will really help the 3D porn industry, oh and cancer research, possibly... When is "Titan W" coming out? You know, Two Titan-V's in one card? Are you waiting for "Windows 4K" to be made? ---
**Author:** ![anon19750978](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon19750978](https://forums.developer.nvidia.com/u/anon19750978)\ **Post date:** [March 4, 2018, 4:17am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/7 "2018-03-04T04:17:07Z")
I concur, I think I even know the theoretical solution: For each instance of CUDA algorithm line calculation, it gets stored in Matrix A until 16 instances are filled, then Stored into B, where their computation is multiplied which ought to lower the the net computation time by 4096 per tensor core utilized.... I've already seen this applied with the cryptonight algorithm, in terms of Hash Calculation however it was insufficient to generate a nonce. Create the nonce with that version of cryptonight and you revolutionize cryptomining! ---
**Author:** ![anon56595808](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon56595808](https://forums.developer.nvidia.com/u/anon56595808)\ **Post date:** [August 24, 2018, 9:14am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/8 "2018-08-24T09:14:18Z")
10 months later, the answer is yes. ---
**Author:** ![anon70852881](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon70852881](https://forums.developer.nvidia.com/u/anon70852881)\ **Post date:** [April 26, 2019, 3:57am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/9 "2019-04-26T03:57:01Z")
Extremely helpful. Thanks as always :) ---
**Author:** ![anon5260794](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon5260794](https://forums.developer.nvidia.com/u/anon5260794)\ **Post date:** [May 31, 2019, 7:07am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/10 "2019-05-31T07:07:00Z")
Question: Do the Tensor cores run concurrently with the CUDA cores? If I were to have my deep learning model cranking away on TCs, could I simultaneously be rendering high quality graphics? ---
**Author:** ![anon76359922](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon76359922](https://forums.developer.nvidia.com/u/anon76359922)\ **Post date:** [September 6, 2019, 8:10am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/11 "2019-09-06T08:10:17Z")
Nice blog but it misses the most important statement - "Tensor Cores require that the Tensors be in NHWC data layout." So if NCHW is given, it transposes it to NHWC. ---
**Author:** ![anon78119193](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@anon78119193](https://forums.developer.nvidia.com/u/anon78119193)\ **Post date:** [October 30, 2019, 12:46am UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/12 "2019-10-30T00:46:14Z")
Can you provide information about relative performance V100 / RTX 2080 TI or V100 / RTX 2080 ? Thank you ---
**Author:** ![khlorghaal](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/khlorghaal/32/42980_2.png) [@khlorghaal](https://forums.developer.nvidia.com/u/khlorghaal)\ **Post date:** [October 20, 2020, 3:11pm UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/13 "2020-10-20T15:11:41Z")
> GEMMs that do not satisfy the above rules will fall back to a non-Tensor Core implementation this sounds like a silent failure, and a really bad thing i ~~assume~~ hope theres some function to assert or check that TPUs are being used? ---
**Author:** ![mitra.subhrajit007](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mitra.subhrajit007](https://forums.developer.nvidia.com/u/mitra.subhrajit007)\ **Post date:** [November 27, 2021, 9:03pm UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/14 "2021-11-27T21:03:36Z")
I think there is a technical error in this image [https://developer-blogs.nvidia.com/wp-content/uploads/2017/12/tensor\_cube\_white-624x934.png](https://developer-blogs.nvidia.com/wp-content/uploads/2017/12/tensor_cube_white-624x934.png). There should be 16 green layers instead of 12. As tensor core only performing multiplication of a 4x4 data, as a result 16 -4x4 array will be generated. ---
**Author:** ![santamonic](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@santamonic](https://forums.developer.nvidia.com/u/santamonic)\ **Post date:** [November 28, 2022, 12:47pm UTC](https://forums.developer.nvidia.com/t/programming-tensor-cores-in-cuda-9/148685/15 "2022-11-28T12:47:57Z")
I have a little question for tensor core code example in blog. Since the threadblock `dim3` is (128, 4), namely 16 warps, I think the loop over k is should be 4xWMMA\_K which would optimize the performance. Would this change the loop stride affect the correctness of result ?