As far as I understand, OpenCL’s concept of memory objects does not approve to store device memory pointers between kernel calls, which also allows that memory objects can be shared between different devices (e.g. CPUs and a GPU) in a single context. So in principle a memory object could be moved around in memory space if the OpenCL runtime decides to do so, e.g. for memory compaction or to swap that object out and in. So for OpenCL 1.0 and 1.1, you will have to put every memory object a kernel relies on into its argument list.