# High Compute in Flight, low DRAM Bandwidth usage **URL:** \ **Category:** CUDA Programming and Performance\ **Created:** [January 1, 2025, 7:44pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531 "2025-01-01T19:44:28Z")\ **Posts on this page:** 16\ **Page:** 2
**Author:** ![SBenner](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@SBenner](https://forums.developer.nvidia.com/u/SBenner)\ **Post date:** [January 3, 2025, 7:46pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/21 "2025-01-03T19:46:13Z")
The page you linked tells me that the T1000, with its compute 7.5 should have a performance ratio of 1:2, but all the data I can find about the T1000, including from this page: > **[NVIDIA T1000 Specs](https://www.techpowerup.com/gpu-specs/t1000.c3797)** > > NVIDIA TU117, 1395 MHz, 896 Cores, 56 TMUs, 32 ROPs, 4096 MB GDDR6, 1250 MHz, 128 bit as suggested by @njuffa, shows a ratio of 1:32, which aligns well with my performance data. The compute version is taken from NVIDIA’s own GPU Compute Capability webpage. Given my experimental values line up with the 1:32 ratio, can you explain to me what is going on? Am I misinterpreting the table? Is the T1000 not actually using compute 7.5 for some reason? ---
**Author:** ![rs277](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@rs277](https://forums.developer.nvidia.com/u/rs277)\ **Post date:** [January 3, 2025, 8:06pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/22 "2025-01-03T20:06:53Z")
> [@SBenner](#): > > Given my experimental values line up with the 1:32 ratio, can you explain to me what is going on? Am I misinterpreting the table? If you look at the [Throughput](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#arithmetic-instructions) table, under column 7.X, the DP64 figure has a “5” subscript, which if clicked , regretably informs you, “2 for compute capability 7.5 GPUs”. ---
**Author:** ![SBenner](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@SBenner](https://forums.developer.nvidia.com/u/SBenner)\ **Post date:** [January 3, 2025, 8:09pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/23 "2025-01-03T20:09:18Z")
Thank you, I missed that. You wouldn’t happen to have an explanation for the T1000 outperforming the P2000 despite having lower FP64 core counts and lower clocks speeds? The problem is not memory-bandwidth limited. ---
**Author:** ![rs277](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@rs277](https://forums.developer.nvidia.com/u/rs277)\ **Post date:** [January 3, 2025, 8:21pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/24 "2025-01-03T20:21:15Z")
Am unsure. Although the T1000 does have more SM’s, all I can do is regurgitate the Turing white paper: “Turing introduces a new processor architecture, the Turing SM, that delivers a dramatic boost in shading efficiency, achieving 50% improvement in delivered performance per CUDA Core compared to the Pascal generation. These improvements are enabled by two key architectural changes. First, the Turing SM adds a new independent integer datapath that can execute instructions concurrently with the floating-point math datapath. In previous generations, executing these instructions would have blocked floating-point instructions from issuing. Second, the SM memory path has been redesigned to unify shared memory, texture caching, and memory load caching into one unit. This translates to 2x more bandwidth and more than 2x more capacity available for L1 cache for common workloads.” Careful examination of both with Nsight Compute may be helpful. ---
**Author:** ![Curefab](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Curefab](https://forums.developer.nvidia.com/u/Curefab)\ **Post date:** [January 3, 2025, 8:55pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/25 "2025-01-03T20:55:40Z")
The P2000 is cc. 6.1, so has double as many FP64 units, and 43% less SMs (edited, corrected thanks to @rs277) compared to the T1000. The clock speeds both go from around 1000 to 1400 MHz. The only other remaining general reason can be that the FP64 units are more efficiently occupied on the T1000. ---
**Author:** ![rs277](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@rs277](https://forums.developer.nvidia.com/u/rs277)\ **Post date:** [January 3, 2025, 8:59pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/26 "2025-01-03T20:59:06Z")
> [@Curefab](#): > > 20% more SMs compared to the T1000 According to [this](https://www.techpowerup.com/gpu-specs/t1000.c3797), the T1000 has 14 SM’s, the [P2000](https://www.techpowerup.com/gpu-specs/quadro-p2000.c2931) 8. ---
**Author:** ![SBenner](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@SBenner](https://forums.developer.nvidia.com/u/SBenner)\ **Post date:** [January 3, 2025, 10:44pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/27 "2025-01-03T22:44:11Z")
The P2000’s compute 6.1 means that the T1000 and P2000 both have a double-to-single ratio of 1:32. Why would the number of SM’s matter? I know the T1000 only has half as many cores per SM, but can the warp scheduling be that heavily influenced by the amount of cores it has to deal with? ---
**Author:** ![Curefab](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Curefab](https://forums.developer.nvidia.com/u/Curefab)\ **Post date:** [January 3, 2025, 11:23pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/28 "2025-01-03T23:23:37Z")
You were talking about the T1000 outperforming the P2000. Were you talking about overall performance? Then SM count matters a lot of course, especially if the kernel performance is not memory-bound, but compute-bound. The table in the programming guide lists instructions per cycle per SM. ---
**Author:** ![SBenner](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@SBenner](https://forums.developer.nvidia.com/u/SBenner)\ **Post date:** [January 3, 2025, 11:53pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/29 "2025-01-03T23:53:12Z")
I see, I was using it as a direct ratio of FP64 to FP32 cores and multiplying the total cuda core count by that. You’re saying that I also need to take into account the number of cuda cores per SM when using the ratio. So if I am understanding correctly, if I were to compare the T1000 to a GPU with the standard 128 cores per SM, the T1000’s ratio is equivalent to 1:16 instead of 1:32 due to the fact that it has only 64 cores per SM? ---
**Author:** ![njuffa](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@njuffa](https://forums.developer.nvidia.com/u/njuffa)\ **Post date:** [January 4, 2025, 1:00am UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/30 "2025-01-04T01:00:49Z")
How big is the performance difference between P2000 and T1000? By raw specs, the computational throughput (78 FP64 GFLOPS vs 94.7 FP64 GFLOPS) and memory throughput (160 GB/sec vs 140 GB/sec) of these two GPUs is rather similar. These two GPUs use different architectures, and a potentially significant difference, namely cache hierarchy design, has already been mentioned. You would want to use the profiler to investigate. The performance of this code may not be purely limited by FP64 throughput. If I recall correctly, there were various changes to the instruction set in the transition from Pascal to Turing; this may have an impact on dynamic instruction count overall. We can speculate in this forum to our heart’s content without coming to a resolution. You have access to code and hardware and a profiler, that is, at least an order of magnitude more in terms of information. ---
**Author:** ![SBenner](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@SBenner](https://forums.developer.nvidia.com/u/SBenner)\ **Post date:** [January 4, 2025, 2:46am UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/31 "2025-01-04T02:46:50Z")
The T1000 consistently outperforms the P2000 by 8-10%. I am only as curious as I am about this GPU, because it is the one outlier in all of my performance charts, all other GPUs lie exactly on the predicted performance curves. It is also the only GPU in my lineup which seems to have deterministic CUDA behavior, something else I have discussed in this forum in the past couple months, but never found an answer to. I haven’t had the opportunity to use the profiler, as the academic lab in which I am working has been closed since well before christmas. I will be looking at those metrics next week, when the lab opens again. I appreciate all your help nonetheless, you have clarified some important subjects for me about which I was misinformed, and certainly helped me further. ---
**Author:** ![Curefab](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Curefab](https://forums.developer.nvidia.com/u/Curefab)\ **Post date:** [January 4, 2025, 5:07am UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/32 "2025-01-04T05:07:44Z")
If you look at table 4 in [CUDA C++ Programming Guide](https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#arithmetic-instructions), you get a “Number of Results per Clock Cycle per Multiprocessor”. So this calculation is without the total Cuda core count, but the SM count instead. It normally should lead to the same results. For 6.1 you have 128 32-bit floating-point operations per cycle per SM and 4 64-bit operations. For 7.5 (7.x with footnote 5) you have 64 32-bit floating-point operations per cycle per SM and 2 64-bit operations. For comparison, the marketing “core count” should be number of SMs times number of FP32 operations per cycle per SM. So the ratio between “core counts” should be the same as the ratio between number of SMs times FP64 (double) units per SM, as the FP32:FP64 ratio is the same. As for the 8-10% faster T1000 (measurement) vs. the 20% faster P2000 (FP64 GFLOPs): - Make sure, it is not the P2000 Mobile with 6 instead of 8 SMs - Are you also using FP32 operations? As the Turing has separate units for INT32 and FP32. - To get the same throughput, the Pascal has to dual-issue instructions. With FP64 alone, it would make no difference, but with an instruction mix it could perhaps explain the performance difference. - Are you sure about the frequencies? The base to boost frequency intervals are very similar, but the GPUs could be configured differently or have different thermal characteristics. ---
**Author:** ![SBenner](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@SBenner](https://forums.developer.nvidia.com/u/SBenner)\ **Post date:** [January 4, 2025, 11:52pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/33 "2025-01-04T23:52:51Z")
It is not the mobile variant. I am only using FP64 operations. Single precision results in unstable numerical solvers due to loss of precision. I’ve never heard of dual-issue instructions. How does one use that, and what affect does it have on c++ code? ---
**Author:** ![rs277](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@rs277](https://forums.developer.nvidia.com/u/rs277)\ **Post date:** [January 5, 2025, 12:15am UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/34 "2025-01-05T00:15:51Z")
> [@SBenner](#): > > I’ve never heard of dual-issue instructions. How does one use that, and what affect does it have on c++ code? It’s not a directly accessible feature - something the instruction scheduler does when the conditions are right. You can read a little about it [here.](https://www.anandtech.com/show/10325/the-nvidia-geforce-gtx-1080-and-1070-founders-edition-review/4) If you compile for sm6.1 and use the cuobjdump utility to inspect the SASS code generated, you can locate instructions that are dual issued, anywhere the listing has curly braces around two instructions, most commonly with memory operations: ```auto /*0408*/ LDS.U.U16 R45, [R3] ; /* 0xef4a10000007032d */ /*0410*/ { SHR.U32 R32, R6, 0x8 ; /* 0x3828000000870620 */ /*0418*/ LDS.U.128 R4, [0x220] } /* 0xef4e10002207ff04 */ /* 0x001d4400fe0007e4 */ /*0428*/ BFE.U32 R34, R29, 0x810 ; /* 0x3800000081071d22 */ /*0430*/ { LOP3.LUT R39, R42, R40, R15, 0x96 ; /* 0x5be7078962872a27 */ /*0438*/ LDC.U8 R11, c[0x3][R34+0x900] } ``` ---
**Author:** ![Curefab](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@Curefab](https://forums.developer.nvidia.com/u/Curefab)\ **Post date:** [January 5, 2025, 12:28pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/35 "2025-01-05T12:28:00Z")
# Dual-Issue See @rs277’s answer. It exists for cc 6.1, (since 2.1 - additionally 1.0-1.3 could dual-issue in a different form: MAD + FPU), but not for later architectures. Dual-Issue only works for combining instructions for different execution units. E.g. memory and load/store. Later architectures typically only have single-issue, but half as many computation units per SM (or per SM Partition) and double as many SMs for the same number of execution units (“CUDA cores”). So double as many scheduler relative to the execution units. Dedicated schedulers (double as many single-issue vs. the double-issue) can issue instructions for different warps instead of just from the same warp. As your kernel perhaps groups several FP64 instructions in a row, the cc 6.1 would not profit so much from dual-issue and your kernel could have advantages with the higher relative count of actual schedulers of later architectures. # Pipeline of FP64 Another difference is that for later Cuda consumer GPUs (which have 2 FP64 instructions per cycle per SM), they cannot evenly be divided on 4 SM Partitions. So they use the SM shared pipeline, which is also used for shared memory accesses. I am not sure, whether some earlier consumer architectures also used the SM shared pipeline, even if there were 4 FP64 execution units. The earliest architectures definitely reused the data ports of the FP32 units. And before Maxwell (i.e. Tesla, Fermi, early and late Kepler) there were no SM Parititions whatsoever, so probably also no shared pipeline. So we are talking about Maxwell and Pascal. Whether the shared pipeline could even have advantages in your case, e.g. more FP64 instructions can be in flight, is difficult to say, it can be totally possible. ---
**Author:** ![system](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/system/32/68080_2.png) [@system](https://forums.developer.nvidia.com/u/system)\ **Post date:** [January 19, 2025, 12:28pm UTC](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531/36 "2025-01-19T12:28:46Z")
Closed on January 19, 2025, 12:28pm UTC This topic was automatically closed 14 days after the last reply. New replies are no longer allowed. [Previous page](https://forums.developer.nvidia.com/t/high-compute-in-flight-low-dram-bandwidth-usage/318531.md?page=1)