I using cuda-gdb to debug a program started from python. The program calls tensorflow and has CUDA kernels inside it. I got the following output. The point is, running the program in cuda-gdb is too slow to be usable, and I saw many threads created (which shouldn’t be the reason why the program ran slow in cuda-gdb). The same program ran much faster in gdb. I am very confused, as in my understanding, cuda-gdb is approximately equal to gdb without breaking into kernels. Can anyone tell me how to make cuda-gdb work more efficiently?
Starting program: /usr/bin/python test_cr_bbp_tf2.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/x86_64-linux-gnu/libthread_db.so.1".
2022-05-04 19:27:19.085367: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcudart.so.11.0
[Detaching after fork from child process 47666]
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
warning: Cannot parse .gnu_debugdata section; LZMA support was disabled at compile time
testmatching
2022-05-04 19:27:20.874867: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcudart.so.11.0
2022-05-04 19:27:21.440945: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcuda.so.1
[Detaching after fork from child process 47699]
[New Thread 0x7fff83ec3700 (LWP 47801)]
2022-05-04 19:27:25.868608: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:25.869528: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 1 with properties:
pciBusID: 0000:41:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:25.870421: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 2 with properties:
pciBusID: 0000:61:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:25.871317: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 3 with properties:
pciBusID: 0000:81:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:25.872626: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 4 with properties:
pciBusID: 0000:c1:00.0 name: NVIDIA A40 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 84 deviceMemorySize: 44.56GiB deviceMemoryBandwidth: 648.29GiB/s
2022-05-04 19:27:25.872649: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcudart.so.11.0
2022-05-04 19:27:25.918062: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcublas.so.11
2022-05-04 19:27:25.918138: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcublasLt.so.11
2022-05-04 19:27:25.949031: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcufft.so.10
2022-05-04 19:27:25.975378: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcurand.so.10
2022-05-04 19:27:26.000168: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcutensor.so.1
2022-05-04 19:27:26.025727: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcusolver.so.11
2022-05-04 19:27:26.051433: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcusparse.so.11
2022-05-04 19:27:26.074068: I tensorflow/stream_executor/platform/default/dso_loader.cc:54] Successfully opened dynamic library libcudnn.so.8
2022-05-04 19:27:26.089536: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0, 1, 2, 3, 4
[New Thread 0x7fff8349a700 (LWP 47809)]
[New Thread 0x7fff82c99700 (LWP 47810)]
[New Thread 0x7fff82498700 (LWP 47811)]
[New Thread 0x7fff81c97700 (LWP 47812)]
[New Thread 0x7fff81496700 (LWP 47813)]
[New Thread 0x7fff80c95700 (LWP 47814)]
[New Thread 0x7fff0fa23700 (LWP 47815)]
[New Thread 0x7fff0f222700 (LWP 47816)]
[New Thread 0x7fff0ea21700 (LWP 47817)]
[New Thread 0x7fff0e220700 (LWP 47818)]
[New Thread 0x7fff0da1f700 (LWP 47819)]
[New Thread 0x7fff0d21e700 (LWP 47820)]
[New Thread 0x7fff0ca1d700 (LWP 47821)]
[New Thread 0x7ffed7fff700 (LWP 47822)]
[New Thread 0x7ffed77fe700 (LWP 47823)]
[New Thread 0x7ffecffff700 (LWP 47824)]
[New Thread 0x7ffed6ffd700 (LWP 47825)]
[New Thread 0x7ffed67fc700 (LWP 47826)]
[New Thread 0x7ffed5ffb700 (LWP 47827)]
[New Thread 0x7ffed57fa700 (LWP 47828)]
[New Thread 0x7ffed4ff9700 (LWP 47829)]
[New Thread 0x7ffecf7fe700 (LWP 47830)]
[New Thread 0x7ffeceffd700 (LWP 47831)]
[New Thread 0x7ffece7fc700 (LWP 47832)]
[New Thread 0x7ffecdffb700 (LWP 47833)]
[New Thread 0x7ffecd7fa700 (LWP 47834)]
[New Thread 0x7ffeccff9700 (LWP 47835)]
[New Thread 0x7ffe97fff700 (LWP 47836)]
[New Thread 0x7ffe9ffff700 (LWP 47837)]
[New Thread 0x7ffe9f7fe700 (LWP 47838)]
[New Thread 0x7ffe9effd700 (LWP 47839)]
[New Thread 0x7ffe9e7fc700 (LWP 47840)]
[New Thread 0x7ffe9dffb700 (LWP 47841)]
[New Thread 0x7ffe9d7fa700 (LWP 47842)]
[New Thread 0x7ffe9cff9700 (LWP 47843)]
[New Thread 0x7ffe977fe700 (LWP 47844)]
[New Thread 0x7ffe96ffd700 (LWP 47845)]
[New Thread 0x7ffe967fc700 (LWP 47846)]
[New Thread 0x7ffe95ffb700 (LWP 47847)]
[New Thread 0x7ffe957fa700 (LWP 47848)]
[New Thread 0x7ffe94ff9700 (LWP 47849)]
[New Thread 0x7ffe5ffff700 (LWP 47850)]
[New Thread 0x7ffe5f7fe700 (LWP 47851)]
[New Thread 0x7ffe5effd700 (LWP 47852)]
[New Thread 0x7ffe5e7fc700 (LWP 47853)]
[New Thread 0x7ffe5dffb700 (LWP 47854)]
[New Thread 0x7ffe5d7fa700 (LWP 47855)]
[New Thread 0x7ffe5cff9700 (LWP 47856)]
[New Thread 0x7ffe3bfff700 (LWP 47857)]
[New Thread 0x7ffe33fff700 (LWP 47858)]
[New Thread 0x7ffe3b7fe700 (LWP 47859)]
[New Thread 0x7ffe3affd700 (LWP 47860)]
[New Thread 0x7ffe3a7fc700 (LWP 47861)]
[New Thread 0x7ffe39ffb700 (LWP 47862)]
[New Thread 0x7ffe397fa700 (LWP 47863)]
[New Thread 0x7ffe38ff9700 (LWP 47864)]
[New Thread 0x7ffe337fe700 (LWP 47865)]
[New Thread 0x7ffe32ffd700 (LWP 47866)]
[New Thread 0x7ffe327fc700 (LWP 47867)]
[New Thread 0x7ffe31ffb700 (LWP 47868)]
[New Thread 0x7ffe317fa700 (LWP 47869)]
[New Thread 0x7ffe30ff9700 (LWP 47870)]
[New Thread 0x7ffdfbfff700 (LWP 47871)]
[New Thread 0x7ffdf3fff700 (LWP 47872)]
[New Thread 0x7ffdfb7fe700 (LWP 47873)]
[New Thread 0x7ffdfaffd700 (LWP 47874)]
[New Thread 0x7ffdfa7fc700 (LWP 47875)]
[New Thread 0x7ffdf9ffb700 (LWP 47876)]
[New Thread 0x7ffdf97fa700 (LWP 47877)]
[New Thread 0x7ffdf8ff9700 (LWP 47878)]
[New Thread 0x7ffdf37fe700 (LWP 47879)]
[New Thread 0x7ffdf2ffd700 (LWP 47880)]
[New Thread 0x7ffdf27fc700 (LWP 47881)]
[New Thread 0x7ffdf1ffb700 (LWP 47882)]
[New Thread 0x7ffdf17fa700 (LWP 47883)]
[New Thread 0x7ffdf0ff9700 (LWP 47884)]
[New Thread 0x7ffdbbfff700 (LWP 47885)]
[New Thread 0x7ffdb3fff700 (LWP 47886)]
[New Thread 0x7ffdbb7fe700 (LWP 47887)]
[New Thread 0x7ffdbaffd700 (LWP 47888)]
[New Thread 0x7ffdba7fc700 (LWP 47889)]
[New Thread 0x7ffdb9ffb700 (LWP 47890)]
[New Thread 0x7ffdb97fa700 (LWP 47891)]
[New Thread 0x7ffdb8ff9700 (LWP 47892)]
[New Thread 0x7ffdb37fe700 (LWP 47893)]
[New Thread 0x7ffdb2ffd700 (LWP 47894)]
[New Thread 0x7ffdb27fc700 (LWP 47895)]
[New Thread 0x7ffdb1ffb700 (LWP 47896)]
[New Thread 0x7ffdb17fa700 (LWP 47897)]
[New Thread 0x7ffdb0ff9700 (LWP 47898)]
[New Thread 0x7ffd7bfff700 (LWP 47899)]
[New Thread 0x7ffd737fe700 (LWP 47900)]
[New Thread 0x7ffd7b7fe700 (LWP 47901)]
[New Thread 0x7ffd7affd700 (LWP 47902)]
[New Thread 0x7ffd7a7fc700 (LWP 47903)]
[New Thread 0x7ffd79ffb700 (LWP 47904)]
[New Thread 0x7ffd797fa700 (LWP 47905)]
[New Thread 0x7ffd78ff9700 (LWP 47906)]
[New Thread 0x7ffd73fff700 (LWP 47915)]
[New Thread 0x7ffd72ffd700 (LWP 47916)]
[New Thread 0x7ffd727fc700 (LWP 47926)]
[New Thread 0x7ffd71ffb700 (LWP 47927)]
[New Thread 0x7ffd717fa700 (LWP 47971)]
[New Thread 0x7ffd70ff9700 (LWP 47972)]
[New Thread 0x7ffd41fff700 (LWP 47997)]
[New Thread 0x7ffd417fe700 (LWP 47998)]
2022-05-04 19:27:35.141033: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 0 with properties:
pciBusID: 0000:01:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:35.142880: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 1 with properties:
pciBusID: 0000:41:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:35.144492: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 2 with properties:
pciBusID: 0000:61:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:35.145595: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 3 with properties:
pciBusID: 0000:81:00.0 name: NVIDIA A10 computeCapability: 8.6
coreClock: 1.695GHz coreCount: 72 deviceMemorySize: 22.20GiB deviceMemoryBandwidth: 558.88GiB/s
2022-05-04 19:27:35.147159: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1734] Found device 4 with properties:
pciBusID: 0000:c1:00.0 name: NVIDIA A40 computeCapability: 8.6
coreClock: 1.74GHz coreCount: 84 deviceMemorySize: 44.56GiB deviceMemoryBandwidth: 648.29GiB/s
2022-05-04 19:27:35.156881: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1872] Adding visible gpu devices: 0, 1, 2, 3, 4
2022-05-04 19:29:36.389511: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1258] Device interconnect StreamExecutor with strength 1 edge matrix:
2022-05-04 19:29:36.389578: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1264] 0 1 2 3 4
2022-05-04 19:29:36.389588: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 0: N Y Y Y Y
2022-05-04 19:29:36.389593: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 1: Y N Y Y Y
2022-05-04 19:29:36.389598: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 2: Y Y N Y Y
2022-05-04 19:29:36.389602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 3: Y Y Y N Y
2022-05-04 19:29:36.389606: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1277] 4: Y Y Y Y N
2022-05-04 19:29:37.270308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 20566 MB memory) -> physical GPU (device: 0, name: NVIDIA A10, pci bus id: 0000:01:00.0, compute capability: 8.6)
[New Thread 0x7ffd40ffd700 (LWP 49726)]
[New Thread 0x7ffcf3fff700 (LWP 49727)]
2022-05-04 19:29:38.663441: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:1 with 20566 MB memory) -> physical GPU (device: 1, name: NVIDIA A10, pci bus id: 0000:41:00.0, compute capability: 8.6)
[New Thread 0x7ffcf37fe700 (LWP 49745)]
[New Thread 0x7ffcf2f14700 (LWP 49746)]
2022-05-04 19:29:40.049576: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:2 with 20566 MB memory) -> physical GPU (device: 2, name: NVIDIA A10, pci bus id: 0000:61:00.0, compute capability: 8.6)
[New Thread 0x7ffcf2713700 (LWP 49764)]
[New Thread 0x7ffcf1f12700 (LWP 49765)]
2022-05-04 19:29:41.402189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:3 with 20566 MB memory) -> physical GPU (device: 3, name: NVIDIA A10, pci bus id: 0000:81:00.0, compute capability: 8.6)
[New Thread 0x7ffcf1711700 (LWP 49823)]
[New Thread 0x7ffcf0f10700 (LWP 49824)]
2022-05-04 19:29:42.773617: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1418] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:4 with 43434 MB memory) -> physical GPU (device: 4, name: NVIDIA A40, pci bus id: 0000:c1:00.0, compute capability: 8.6)
[New Thread 0x7ffc9bfff700 (LWP 49849)]
[New Thread 0x7ffc9b7fe700 (LWP 49850)]
2022-05-04 19:29:44.166107: I tensorflow/core/common_runtime/process_util.cc:146] Creating new thread pool with default inter op setting: 96. Tune using inter_op_parallelism_threads for best performance.
[New Thread 0x7ffc9affd700 (LWP 49851)]
[New Thread 0x7ffc9a7fc700 (LWP 49852)]
[New Thread 0x7ffc99ffb700 (LWP 49853)]
[New Thread 0x7ffc997fa700 (LWP 49854)]
[New Thread 0x7ffc98ff9700 (LWP 49855)]
[New Thread 0x7ffc77fff700 (LWP 49856)]
[New Thread 0x7ffc6ffff700 (LWP 49857)]
[New Thread 0x7ffc777fe700 (LWP 49858)]
[New Thread 0x7ffc76ffd700 (LWP 49859)]
[New Thread 0x7ffc767fc700 (LWP 49860)]
[New Thread 0x7ffc75ffb700 (LWP 49861)]
[New Thread 0x7ffc757fa700 (LWP 49862)]
[New Thread 0x7ffc74ff9700 (LWP 49863)]
[New Thread 0x7ffc6f7fe700 (LWP 49864)]
[New Thread 0x7ffc6effd700 (LWP 49865)]
[New Thread 0x7ffc6e7fc700 (LWP 49866)]
[New Thread 0x7ffc6dffb700 (LWP 49867)]
[New Thread 0x7ffc6d7fa700 (LWP 49868)]
[New Thread 0x7ffc6cff9700 (LWP 49869)]
[New Thread 0x7ffc3ffff700 (LWP 49870)]
[New Thread 0x7ffc3f7fe700 (LWP 49871)]
[New Thread 0x7ffc3effd700 (LWP 49872)]
[New Thread 0x7ffc3e7fc700 (LWP 49873)]
[New Thread 0x7ffc3dffb700 (LWP 49874)]
[New Thread 0x7ffc3d7fa700 (LWP 49875)]
[New Thread 0x7ffc3cff9700 (LWP 49876)]
[New Thread 0x7ffc13fff700 (LWP 49877)]
[New Thread 0x7ffc137fe700 (LWP 49878)]
[New Thread 0x7ffc12ffd700 (LWP 49879)]
[New Thread 0x7ffc127fc700 (LWP 49880)]
[New Thread 0x7ffc11ffb700 (LWP 49881)]
[New Thread 0x7ffc117fa700 (LWP 49882)]
[New Thread 0x7ffc10ff9700 (LWP 49883)]
[New Thread 0x7ffbfbfff700 (LWP 49884)]
[New Thread 0x7ffbfb7fe700 (LWP 49885)]
[New Thread 0x7ffbfaffd700 (LWP 49886)]
[New Thread 0x7ffbfa7fc700 (LWP 49887)]
[New Thread 0x7ffbf9ffb700 (LWP 49888)]
[New Thread 0x7ffbf97fa700 (LWP 49889)]
[New Thread 0x7ffbf8ff9700 (LWP 49890)]
[New Thread 0x7ffbd7fff700 (LWP 49891)]
[New Thread 0x7ffbd77fe700 (LWP 49892)]
[New Thread 0x7ffbd6ffd700 (LWP 49893)]
[New Thread 0x7ffbd67fc700 (LWP 49894)]
[New Thread 0x7ffbd5ffb700 (LWP 49895)]
[New Thread 0x7ffbd57fa700 (LWP 49896)]
[New Thread 0x7ffbd4ff9700 (LWP 49897)]
[New Thread 0x7ffbb7fff700 (LWP 49898)]
[New Thread 0x7ffbb77fe700 (LWP 49899)]
[New Thread 0x7ffbb6ffd700 (LWP 49900)]
[New Thread 0x7ffbb67fc700 (LWP 49901)]
[New Thread 0x7ffbb5ffb700 (LWP 49902)]
[New Thread 0x7ffbb57fa700 (LWP 49903)]
[New Thread 0x7ffbb4ff9700 (LWP 49904)]
[New Thread 0x7ffb97fff700 (LWP 49905)]
[New Thread 0x7ffb8ffff700 (LWP 49906)]
[New Thread 0x7ffb977fe700 (LWP 49907)]
[New Thread 0x7ffb96ffd700 (LWP 49908)]
[New Thread 0x7ffb967fc700 (LWP 49909)]
[New Thread 0x7ffb95ffb700 (LWP 49910)]
[New Thread 0x7ffb957fa700 (LWP 49911)]
[New Thread 0x7ffb94ff9700 (LWP 49912)]
[New Thread 0x7ffb8f7fe700 (LWP 49913)]
[New Thread 0x7ffb8effd700 (LWP 49914)]
[New Thread 0x7ffb8e7fc700 (LWP 49915)]
[New Thread 0x7ffb8dffb700 (LWP 49916)]
[New Thread 0x7ffb8d7fa700 (LWP 49917)]
[New Thread 0x7ffb8cff9700 (LWP 49918)]
[New Thread 0x7ffb57fff700 (LWP 49919)]
[New Thread 0x7ffb577fe700 (LWP 49920)]
[New Thread 0x7ffb56ffd700 (LWP 49921)]
[New Thread 0x7ffb567fc700 (LWP 49922)]
[New Thread 0x7ffb55ffb700 (LWP 49923)]
[New Thread 0x7ffb557fa700 (LWP 49924)]
[New Thread 0x7ffb54ff9700 (LWP 49925)]
[New Thread 0x7ffb37fff700 (LWP 49926)]
[New Thread 0x7ffb2f7fe700 (LWP 49927)]
[New Thread 0x7ffb377fe700 (LWP 49928)]
[New Thread 0x7ffb36ffd700 (LWP 49929)]
[New Thread 0x7ffb367fc700 (LWP 49930)]
[New Thread 0x7ffb35ffb700 (LWP 49931)]
[New Thread 0x7ffb357fa700 (LWP 49932)]
[New Thread 0x7ffb34ff9700 (LWP 49933)]
[New Thread 0x7ffb2ffff700 (LWP 49934)]
[New Thread 0x7ffb2effd700 (LWP 49935)]
[New Thread 0x7ffb2e7fc700 (LWP 49936)]
[New Thread 0x7ffb2dffb700 (LWP 49937)]
[New Thread 0x7ffb2d7fa700 (LWP 49938)]
[New Thread 0x7ffb2cff9700 (LWP 49939)]
[New Thread 0x7ffaf7fff700 (LWP 49940)]
[New Thread 0x7ffaeffff700 (LWP 49941)]
[New Thread 0x7ffaf77fe700 (LWP 49942)]
[New Thread 0x7ffaf6ffd700 (LWP 49943)]
[New Thread 0x7ffaf67fc700 (LWP 49944)]
[New Thread 0x7ffaf5ffb700 (LWP 49945)]
[New Thread 0x7ffaf57fa700 (LWP 49946)]
inBox: [ 8 1527 56 839]
outBox: [ 9 1526 57 838]
msdiffGPUBBP 9.787128717049706 8.836871411244479 8.792443641292834 1.0240484074218186
msdiffGPUBBP 8.284130262488215 8.504984482311862 8.468490694873381 0.9001938761471568
msdiffGPUBBP 7.607680601497159 8.326243672859956 8.302741359766436 0.7882571994561002
msdiffGPUBBP 9.01851150047646 8.826698188773078 8.800456119590438 0.852269245604869
msdiffGPUBBP 10.002452782253538 9.02661007092474 9.003068931959348 0.7974960688804729
msdiffGPUBBP 9.362822708408418 8.672302889066888 8.648869787310186 0.8399175541051185
msdiffGPUBBP 13.275880608755909 13.290950955327459 13.297561482034173 0.5154759081244825
mean (GPU-BBP) 0.8168083228200026
mean diff Median (GPU-BBP) 0.0
[Thread 0x7ffaf67fc700 (LWP 49944) exited]
[Thread 0x7ffaf57fa700 (LWP 49946) exited]
[Thread 0x7ffaf5ffb700 (LWP 49945) exited]
[Thread 0x7ffaf6ffd700 (LWP 49943) exited]
[Thread 0x7ffaf77fe700 (LWP 49942) exited]
[Thread 0x7ffaeffff700 (LWP 49941) exited]
[Thread 0x7ffaf7fff700 (LWP 49940) exited]
[Thread 0x7ffb2cff9700 (LWP 49939) exited]
[Thread 0x7ffb2d7fa700 (LWP 49938) exited]
[Thread 0x7ffb2dffb700 (LWP 49937) exited]
[Thread 0x7ffb2e7fc700 (LWP 49936) exited]
[Thread 0x7ffb2effd700 (LWP 49935) exited]
[Thread 0x7ffb2ffff700 (LWP 49934) exited]
[Thread 0x7ffb34ff9700 (LWP 49933) exited]
[Thread 0x7ffb357fa700 (LWP 49932) exited]
[Thread 0x7ffb35ffb700 (LWP 49931) exited]
[Thread 0x7ffb367fc700 (LWP 49930) exited]
[Thread 0x7ffb36ffd700 (LWP 49929) exited]
[Thread 0x7ffb377fe700 (LWP 49928) exited]
[Thread 0x7ffb2f7fe700 (LWP 49927) exited]
[Thread 0x7ffb37fff700 (LWP 49926) exited]
[Thread 0x7ffb54ff9700 (LWP 49925) exited]
[Thread 0x7ffb557fa700 (LWP 49924) exited]
[Thread 0x7ffb55ffb700 (LWP 49923) exited]
[Thread 0x7ffb567fc700 (LWP 49922) exited]
[Thread 0x7ffb56ffd700 (LWP 49921) exited]
[Thread 0x7ffb577fe700 (LWP 49920) exited]
[Thread 0x7ffb57fff700 (LWP 49919) exited]
[Thread 0x7ffb8cff9700 (LWP 49918) exited]
[Thread 0x7ffb8d7fa700 (LWP 49917) exited]
[Thread 0x7ffb8dffb700 (LWP 49916) exited]
[Thread 0x7ffb8e7fc700 (LWP 49915) exited]
[Thread 0x7ffb8effd700 (LWP 49914) exited]
[Thread 0x7ffb8f7fe700 (LWP 49913) exited]
[Thread 0x7ffb94ff9700 (LWP 49912) exited]
[Thread 0x7ffb957fa700 (LWP 49911) exited]
[Thread 0x7ffb95ffb700 (LWP 49910) exited]
[Thread 0x7ffb967fc700 (LWP 49909) exited]
[Thread 0x7ffb96ffd700 (LWP 49908) exited]
[Thread 0x7ffb977fe700 (LWP 49907) exited]
[Thread 0x7ffb8ffff700 (LWP 49906) exited]
[Thread 0x7ffb97fff700 (LWP 49905) exited]
[Thread 0x7ffbb4ff9700 (LWP 49904) exited]
[Thread 0x7ffbb57fa700 (LWP 49903) exited]
[Thread 0x7ffbb5ffb700 (LWP 49902) exited]
[Thread 0x7ffbb67fc700 (LWP 49901) exited]
[Thread 0x7ffbb6ffd700 (LWP 49900) exited]
[Thread 0x7ffbb77fe700 (LWP 49899) exited]
[Thread 0x7ffbb7fff700 (LWP 49898) exited]
[Thread 0x7ffbd4ff9700 (LWP 49897) exited]
[Thread 0x7ffbd57fa700 (LWP 49896) exited]
[Thread 0x7ffbd5ffb700 (LWP 49895) exited]
[Thread 0x7ffbd67fc700 (LWP 49894) exited]
[Thread 0x7ffbd6ffd700 (LWP 49893) exited]
[Thread 0x7ffbd77fe700 (LWP 49892) exited]
[Thread 0x7ffbd7fff700 (LWP 49891) exited]
[Thread 0x7ffbf8ff9700 (LWP 49890) exited]
[Thread 0x7ffbf97fa700 (LWP 49889) exited]
[Thread 0x7ffbf9ffb700 (LWP 49888) exited]
[Thread 0x7ffbfa7fc700 (LWP 49887) exited]
[Thread 0x7ffbfaffd700 (LWP 49886) exited]
[Thread 0x7ffbfb7fe700 (LWP 49885) exited]
[Thread 0x7ffbfbfff700 (LWP 49884) exited]
[Thread 0x7ffc10ff9700 (LWP 49883) exited]
[Thread 0x7ffc117fa700 (LWP 49882) exited]
[Thread 0x7ffc11ffb700 (LWP 49881) exited]
[Thread 0x7ffc127fc700 (LWP 49880) exited]
[Thread 0x7ffc12ffd700 (LWP 49879) exited]
[Thread 0x7ffc137fe700 (LWP 49878) exited]
[Thread 0x7ffc13fff700 (LWP 49877) exited]
[Thread 0x7ffc3cff9700 (LWP 49876) exited]
[Thread 0x7ffc3d7fa700 (LWP 49875) exited]
[Thread 0x7ffc3dffb700 (LWP 49874) exited]
[Thread 0x7ffc3e7fc700 (LWP 49873) exited]
[Thread 0x7ffc3effd700 (LWP 49872) exited]
[Thread 0x7ffc3f7fe700 (LWP 49871) exited]
[Thread 0x7ffc3ffff700 (LWP 49870) exited]
[Thread 0x7ffc6cff9700 (LWP 49869) exited]
[Thread 0x7ffc6d7fa700 (LWP 49868) exited]
[Thread 0x7ffc6dffb700 (LWP 49867) exited]
[Thread 0x7ffc6e7fc700 (LWP 49866) exited]
[Thread 0x7ffc6effd700 (LWP 49865) exited]
[Thread 0x7ffc6f7fe700 (LWP 49864) exited]
[Thread 0x7ffc74ff9700 (LWP 49863) exited]
[Thread 0x7ffc757fa700 (LWP 49862) exited]
[Thread 0x7ffc75ffb700 (LWP 49861) exited]
[Thread 0x7ffc767fc700 (LWP 49860) exited]
[Thread 0x7ffc76ffd700 (LWP 49859) exited]
[Thread 0x7ffc777fe700 (LWP 49858) exited]
[Thread 0x7ffc6ffff700 (LWP 49857) exited]
[Thread 0x7ffc77fff700 (LWP 49856) exited]
[Thread 0x7ffc98ff9700 (LWP 49855) exited]
[Thread 0x7ffc997fa700 (LWP 49854) exited]
[Thread 0x7ffc99ffb700 (LWP 49853) exited]
[Thread 0x7ffc9a7fc700 (LWP 49852) exited]
[Thread 0x7ffc9affd700 (LWP 49851) exited]
[Thread 0x7ffc9b7fe700 (LWP 49850) exited]
[Thread 0x7ffc9bfff700 (LWP 49849) exited]
[Thread 0x7ffcf0f10700 (LWP 49824) exited]
[Thread 0x7ffcf1711700 (LWP 49823) exited]
[Thread 0x7ffcf1f12700 (LWP 49765) exited]
[Thread 0x7ffcf2713700 (LWP 49764) exited]
[Thread 0x7ffcf2f14700 (LWP 49746) exited]
[Thread 0x7ffcf37fe700 (LWP 49745) exited]
[Thread 0x7ffcf3fff700 (LWP 49727) exited]
[Thread 0x7ffd40ffd700 (LWP 49726) exited]
[Thread 0x7ffd417fe700 (LWP 47998) exited]
[Thread 0x7ffd41fff700 (LWP 47997) exited]
[Thread 0x7ffd70ff9700 (LWP 47972) exited]
[Thread 0x7ffd717fa700 (LWP 47971) exited]
[Thread 0x7ffd71ffb700 (LWP 47927) exited]
[Thread 0x7ffd727fc700 (LWP 47926) exited]
[Thread 0x7ffd72ffd700 (LWP 47916) exited]
[Thread 0x7ffd73fff700 (LWP 47915) exited]
[Thread 0x7ffd78ff9700 (LWP 47906) exited]
[Thread 0x7ffd797fa700 (LWP 47905) exited]
[Thread 0x7ffd79ffb700 (LWP 47904) exited]
[Thread 0x7ffd7a7fc700 (LWP 47903) exited]
[Thread 0x7ffd7affd700 (LWP 47902) exited]
[Thread 0x7ffd7b7fe700 (LWP 47901) exited]
[Thread 0x7ffd737fe700 (LWP 47900) exited]
[Thread 0x7ffd7bfff700 (LWP 47899) exited]
[Thread 0x7ffdb0ff9700 (LWP 47898) exited]
[Thread 0x7ffdb17fa700 (LWP 47897) exited]
[Thread 0x7ffdb1ffb700 (LWP 47896) exited]
[Thread 0x7ffdb27fc700 (LWP 47895) exited]
[Thread 0x7ffdb2ffd700 (LWP 47894) exited]
[Thread 0x7ffdb37fe700 (LWP 47893) exited]
[Thread 0x7ffdb8ff9700 (LWP 47892) exited]
[Thread 0x7ffdb97fa700 (LWP 47891) exited]
[Thread 0x7ffdb9ffb700 (LWP 47890) exited]
[Thread 0x7ffdba7fc700 (LWP 47889) exited]
[Thread 0x7ffdbaffd700 (LWP 47888) exited]
[Thread 0x7ffdbb7fe700 (LWP 47887) exited]
[Thread 0x7ffdb3fff700 (LWP 47886) exited]
[Thread 0x7ffdbbfff700 (LWP 47885) exited]
[Thread 0x7ffdf0ff9700 (LWP 47884) exited]
[Thread 0x7ffdf17fa700 (LWP 47883) exited]
[Thread 0x7ffdf1ffb700 (LWP 47882) exited]
[Thread 0x7ffdf27fc700 (LWP 47881) exited]
[Thread 0x7ffdf2ffd700 (LWP 47880) exited]
[Thread 0x7ffdf37fe700 (LWP 47879) exited]
[Thread 0x7ffdf8ff9700 (LWP 47878) exited]
[Thread 0x7ffdf97fa700 (LWP 47877) exited]
[Thread 0x7ffdf9ffb700 (LWP 47876) exited]
[Thread 0x7ffdfa7fc700 (LWP 47875) exited]
[Thread 0x7ffdfaffd700 (LWP 47874) exited]
[Thread 0x7ffdfb7fe700 (LWP 47873) exited]
[Thread 0x7ffdf3fff700 (LWP 47872) exited]
[Thread 0x7ffdfbfff700 (LWP 47871) exited]
[Thread 0x7ffe30ff9700 (LWP 47870) exited]
[Thread 0x7ffe317fa700 (LWP 47869) exited]
[Thread 0x7ffe31ffb700 (LWP 47868) exited]
[Thread 0x7ffe327fc700 (LWP 47867) exited]
[Thread 0x7ffe32ffd700 (LWP 47866) exited]
[Thread 0x7ffe337fe700 (LWP 47865) exited]
[Thread 0x7ffe38ff9700 (LWP 47864) exited]
[Thread 0x7ffe397fa700 (LWP 47863) exited]
[Thread 0x7ffe39ffb700 (LWP 47862) exited]
[Thread 0x7ffe3a7fc700 (LWP 47861) exited]
[Thread 0x7ffe3affd700 (LWP 47860) exited]
[Thread 0x7ffe3b7fe700 (LWP 47859) exited]
[Thread 0x7ffe33fff700 (LWP 47858) exited]
[Thread 0x7ffe3bfff700 (LWP 47857) exited]
[Thread 0x7ffe5cff9700 (LWP 47856) exited]
[Thread 0x7ffe5d7fa700 (LWP 47855) exited]
[Thread 0x7ffe5dffb700 (LWP 47854) exited]
[Thread 0x7ffe5effd700 (LWP 47852) exited]
[Thread 0x7ffe5f7fe700 (LWP 47851) exited]
[Thread 0x7ffe5ffff700 (LWP 47850) exited]
[Thread 0x7ffe94ff9700 (LWP 47849) exited]
[Thread 0x7ffe957fa700 (LWP 47848) exited]
[Thread 0x7ffe95ffb700 (LWP 47847) exited]
[Thread 0x7ffe967fc700 (LWP 47846) exited]
[Thread 0x7ffe96ffd700 (LWP 47845) exited]
[Thread 0x7ffe977fe700 (LWP 47844) exited]
[Thread 0x7ffe9cff9700 (LWP 47843) exited]
[Thread 0x7ffe9d7fa700 (LWP 47842) exited]
[Thread 0x7ffe9dffb700 (LWP 47841) exited]
[Thread 0x7ffe9e7fc700 (LWP 47840) exited]
[Thread 0x7ffe9effd700 (LWP 47839) exited]
[Thread 0x7ffe9f7fe700 (LWP 47838) exited]
[Thread 0x7ffe9ffff700 (LWP 47837) exited]
[Thread 0x7ffe97fff700 (LWP 47836) exited]
[Thread 0x7ffeccff9700 (LWP 47835) exited]
[Thread 0x7ffecd7fa700 (LWP 47834) exited]
[Thread 0x7ffecdffb700 (LWP 47833) exited]
[Thread 0x7ffece7fc700 (LWP 47832) exited]
[Thread 0x7ffeceffd700 (LWP 47831) exited]
[Thread 0x7ffecf7fe700 (LWP 47830) exited]
[Thread 0x7ffed4ff9700 (LWP 47829) exited]
[Thread 0x7ffed57fa700 (LWP 47828) exited]
[Thread 0x7ffed5ffb700 (LWP 47827) exited]
[Thread 0x7ffed67fc700 (LWP 47826) exited]
[Thread 0x7ffed6ffd700 (LWP 47825) exited]
[Thread 0x7ffecffff700 (LWP 47824) exited]
[Thread 0x7ffed77fe700 (LWP 47823) exited]
[Thread 0x7ffed7fff700 (LWP 47822) exited]
[Thread 0x7fff0ca1d700 (LWP 47821) exited]
[Thread 0x7fff0d21e700 (LWP 47820) exited]
[Thread 0x7fff0da1f700 (LWP 47819) exited]
[Thread 0x7fff0e220700 (LWP 47818) exited]
[Thread 0x7fff0ea21700 (LWP 47817) exited]
[Thread 0x7fff0f222700 (LWP 47816) exited]
[Thread 0x7fff0fa23700 (LWP 47815) exited]
[Thread 0x7fff80c95700 (LWP 47814) exited]
[Thread 0x7fff81496700 (LWP 47813) exited]
[Thread 0x7fff81c97700 (LWP 47812) exited]
[Thread 0x7fff82498700 (LWP 47811) exited]
[Thread 0x7fff82c99700 (LWP 47810) exited]
[Thread 0x7fff8349a700 (LWP 47809) exited]
[Thread 0x7fff83ec3700 (LWP 47801) exited]
[Thread 0x7ffff7c09740 (LWP 47591) exited]