# memCpy : Device to Device VERY SLOW **URL:** \ **Category:** CUDA Programming and Performance\ **Created:** [September 12, 2009, 8:52pm UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123 "2009-09-12T20:52:16Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Buanderie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/buanderie/32/12085_2.png) [@Buanderie](https://forums.developer.nvidia.com/u/Buanderie)\ **Post date:** [September 12, 2009, 8:52pm UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/1 "2009-09-12T20:52:16Z")
Hello, I needed to use cudaMemcpy from a device array to another… and it was incredibly slow, as figured on the visual profiler… I checked this using the “bandwith” SDK project : Running on… device 0:GeForce GTX 260 Quick Mode Host to Device Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 1885.7 Quick Mode Device to Host Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 1949.6 Quick Mode Device to Device Bandwidth . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 0.4 &&&& Test PASSED Press ENTER to exit… If I understand I better should copy Device to Host and then Host to Device, instead of Device to Device ? Hu… ---
**Author:** ![seibert](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/seibert/32/520688_2.png) [@seibert](https://forums.developer.nvidia.com/u/seibert)\ **Post date:** [September 13, 2009, 3:10am UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/2 "2009-09-13T03:10:09Z")
That is extremely abnormal behavior. I don’t have any idea what is wrong, but a GTX 260 should have much higher device to device bandwidth. Something more like 80 to 100 GB/sec is normal. ---
**Author:** ![LSChien](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@LSChien](https://forums.developer.nvidia.com/u/LSChien)\ **Post date:** [September 13, 2009, 5:58am UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/3 "2009-09-13T05:58:06Z")
> [@](#): > > Hello, > > I needed to use cudaMemcpy from a device array to another… and it was incredibly slow, as figured on the visual profiler… > > I checked this using the “bandwith” SDK project : > > Running on… > > ``` > device 0:GeForce GTX 260 > > ``` > > Quick Mode > > Host to Device Bandwidth for Pageable memory > > . > > Transfer Size (Bytes) Bandwidth(MB/s) > > 33554432 1885.7 > > Quick Mode > > Device to Host Bandwidth for Pageable memory > > . > > Transfer Size (Bytes) Bandwidth(MB/s) > > 33554432 1949.6 > > Quick Mode > > Device to Device Bandwidth > > . > > Transfer Size (Bytes) Bandwidth(MB/s) > > 33554432 0.4 > > &&&& Test PASSED > > Press ENTER to exit… > > If I understand I better should copy Device to Host and then Host to Device, instead of Device to Device ? > > Hu… I have a GTX260, and it works fine as described in the SPEC, your d2d bandwidth is very strange. what’s your driver and what’s your memory frequency in NVIDIA control panel ---
**Author:** ![Buanderie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/buanderie/32/12085_2.png) [@Buanderie](https://forums.developer.nvidia.com/u/Buanderie)\ **Post date:** [September 13, 2009, 9:29am UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/4 "2009-09-13T09:29:34Z")
> [@](#): > > I have a GTX260, and it works fine as described in the SPEC, > > your d2d bandwidth is very strange. > > what’s your driver and what’s your memory frequency in NVIDIA control panel Here it is… Screenshot of the nVidia control panel. [External Image](http://www.monadiclabs.com/cours/nvidia.jpg "Image hosted on another site. Click to open in a new tab.") It’s very strange, indeed… [External Media](http://hqnveipbwb20/public/style_emoticons/<#EMO_DIR#>/blarg.gif "Media hosted on another site. Click to open in a new tab.") I hope it’s not hardware related. Since I put a lot of money in this card (I’m a student…). Btw, my CUDA Toolkit and SDK version is 2.3 (downloaded it last week). Any advice from nVidia staff would be welcomed :) ---
**Author:** ![Buanderie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/buanderie/32/12085_2.png) [@Buanderie](https://forums.developer.nvidia.com/u/Buanderie)\ **Post date:** [September 13, 2009, 2:19pm UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/5 "2009-09-13T14:19:21Z")
Anyone ? I know this forum is not a helpdesk… but I’d like to continue writing my cellular automata based RNG… This copy operation is biasing all my optimization results… :( ---
**Author:** ![Buanderie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/buanderie/32/12085_2.png) [@Buanderie](https://forums.developer.nvidia.com/u/Buanderie)\ **Post date:** [September 13, 2009, 2:24pm UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/6 "2009-09-13T14:24:11Z")
Okay… The bandwidthTest I ran was in Debug mode (but not EMU mode !)… I tried in Release mode… Here is the result : Running on… device 0:GeForce GTX 260 Quick Mode Host to Device Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 2550.2 Quick Mode Device to Host Bandwidth for Pageable memory . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 2303.3 Quick Mode Device to Device Bandwidth . Transfer Size (Bytes) Bandwidth(MB/s) 33554432 92093.3 &&&& Test PASSED Press ENTER to exit… Question is… Is it a normal behaviour ? Device to Device bandwidth should be the same in Debug and Release mode… As long at it’s not Emulation mode. But I don’t know… Maybe it’s normal. Since I can’t test on other hardware, I can’t say. ---
**Author:** ![LSChien](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@LSChien](https://forums.developer.nvidia.com/u/LSChien)\ **Post date:** [September 13, 2009, 2:46pm UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/7 "2009-09-13T14:46:09Z")
> [@](#): > > Okay… > > The bandwidthTest I ran was in Debug mode (but not EMU mode !)… I tried in Release mode… Here is the result : > > Running on… > > ``` > device 0:GeForce GTX 260 > > ``` > > Quick Mode > > Host to Device Bandwidth for Pageable memory > > . > > Transfer Size (Bytes) Bandwidth(MB/s) > > 33554432 2550.2 > > Quick Mode > > Device to Host Bandwidth for Pageable memory > > . > > Transfer Size (Bytes) Bandwidth(MB/s) > > 33554432 2303.3 > > Quick Mode > > Device to Device Bandwidth > > . > > Transfer Size (Bytes) Bandwidth(MB/s) > > 33554432 92093.3 > > &&&& Test PASSED > > Press ENTER to exit… > > Question is… Is it a normal behaviour ? Device to Device bandwidth should be the same in Debug and Release mode… As long at it’s not Emulation mode. But I don’t know… Maybe it’s normal. Since I can’t test on other hardware, I can’t say. I compile source code of bandwidthTest in SDK by debug mode, then d2d bandwidth is the same as that by release mode in GTX295 ---
**Author:** ![Buanderie](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/buanderie/32/12085_2.png) [@Buanderie](https://forums.developer.nvidia.com/u/Buanderie)\ **Post date:** [September 13, 2009, 2:59pm UTC](https://forums.developer.nvidia.com/t/memcpy-device-to-device-very-slow/12123/8 "2009-09-13T14:59:04Z")
> [@](#): > > I compile source code of bandwidthTest in SDK by debug mode, then d2d bandwidth is the same as that by release mode > > in GTX295 In fact I checked the CUDA configuration of the .cu file in my VS2008 solution… The bandwidthTest.cu was compiled in Emulation mode… Even though the project was set to device mode. I feel kind of stupid [External Media](http://hqnveipbwb20/public/style_emoticons/<#EMO_DIR#>/blarg.gif "Media hosted on another site. Click to open in a new tab.") Anyway thank you everyone for your support :)