clGetKernelWorkGroupInfo does not return correct local mem

I am trying to query the local memory usage of a kernel.
The kernel takes a local memory argument.
I am setting the size of the local memory with clSetKernelArg.
Then simple try to use the function clGetKernelWorkGroupInfo with CL_KERNEL_LOCAL_MEM_SIZE to get the local mem usage.
It gives me incorrect number.
I am using cuda OpenCL.
Any ideas?