Trouble in using cudaMemCpy in MPI

Accelerated Computing CUDA CUDA Programming and Performance
1

I write two codes,for simplicity, they are named “MPI.c” and “CUDA.cu” here, and a header file, “cufunction.h” including the function name in Cuda.cu. The layouts are below.
////////////////////////
MPI.c
#include"cufunction.h"
main()
{
cudaSetDevice(rank);
cufunction1();
cufunction2();
}
////////////////////////

////////////////////////
CUDA.cu
extern “C” void cufunction1()
{
cudaMalloc(device); cudaMemcpy(device,host,cudaMemCpyHosttoDevice);
Malloc(host_temp1); cudaMemcpy(host_temp1,device,cudaMemCpyDevicetoHost);
printf(host_temp1);
}
extern “C” void cufunction2()
{
Malloc(host_temp2); cudaMemcpy(host_temp2,device,cudaMemCpyDevicetoHost);
printf(host_temp2);
}
///////////////////////

When I call cufunction1 in MPI main function, the host_temp1 comes right, it is equal the value of host.
But when I call cufunction 2 in MPI main function, the host_temp2 comes wrong and it is the initialized value of 0. I think the cudaMemCpy does not work in cufunction2 but there is not anything when running the program.

I hope someone could give me some tips on this one and I will be really grateful!!!

2

If you allocate for device using cudaMalloc in cufunction1, there is a reasonable chance you are passing that device pointer value incorrectly to cufunction2. Its a fairly common type of error. But its impossible to say based on the “pseudocode” you have shown.

It’s always good practice to use proper cuda error checking. statements of the form “there is not anything when running the program.” when the code has no indication of proper cuda error checking are immediately suspect.

3

Actually, all the pointers and arguments are defined in the header files and they are included in the MPi.c.
I have used nvprof to watch the running progress, the cudamemcpy in cufunction 2 is called but the elapse time is shorter than the one in cufunction1, I will take your advice and try the cuda error check.

Thank you!

4

There is a button in the top bar to indent your code as code, please use it!
Why are you using extern “C”?

Without declarations of your variables it is not possible to reproduce the error. I recommend you to use a macro which checks returning results of cuda API functions. Something like this:

#define CUDA_CHECK(ans) { __cudaCheckError((ans), __FILE__, __LINE__); }
inline void __cudaCheckError(cudaError_t code, const char *file, int line, bool abort = true)
{
	cudaError sync = cudaDeviceSynchronize();
	// cudaSuccess async = cudaDeviceASynchronize();
	if(sync != cudaSuccess)
	{
		if (code != cudaSuccess)
		{
			printf("Synchronous Cuda Error: %s %s %d\n",
				cudaGetErrorString(code),
				file,
				line);
		}
	}
	if (code != cudaSuccess)
	{
		printf("Cuda Error: %s %s %d\n",
			cudaGetErrorString(code),
			file,
			line);
	}
}
...
// If you call a function from the CUDA API use the macro like this
CUDA_CHECK(cudaMemcpy(dst, src, size, cudaMemcpyKind));

Furthermore the cudaMemcpy() function requires the size in bytes you want to copy as third parameter. Read the documentation: http://horacio9573.no-ip.org/cuda/group__CUDART__MEMORY_g48efa06b81cc031b2aa6fdc2e9930741.html Normally, your compiler should tell you that function parameters are missing.

5

Thank you for your help, I figure out the problem “invalid argument” now. But yet I can not fix it…