One important thing to keep in mind is also that size_t and ptrdiff_t are likely not to be the same size on the host and on the GPU, which can cause misalignment

//Host code
struct my_struct{
int* my_ptr;
int a;
};

//Kernel code
struct my_kernel_struct{
int* my_ptr;
int a;
};

Here, a raw copy between a my_struct structure and a my_kernel_stuct structure might not give the expected result if your host is on a 64bit OS and your device works with 32bit arithmetic, which is often the case.
If portability is not an issue, you can querry CL_DEVICE_ADDRESS_BITS , and check that this is the same as your host.