Do we need to defrag the card memory?

The easiest way to solve this is to allocate large chunks of memory at the beginning of your program, then manage this yourself in pools.