Aaaah I finally figured out what I was doing wrong thanks to this post. I was creating a cuda context per model, whereas I should be having one global cuda context for all models in the same process. I have now made the call to
cfx = cuda.Device(0).make_context()
global, and now the overhead is only once instead of per model. I can now run my model with just 12 MB of added memory, so very similar to the CPU memory usage.
I hope this thread helps someone else who is just as obtuse as me. :D
Edit: or even better: use cfx = cuda.Device(0).retain_primary_context().