Some details on parts of this might help to start…
Swap is “virtual” memory, meaning it is stored to a disk partition or file rather than in RAM. This is useful for user space programs, although it slows things down. Additionally, some devices cannot use virtual memory. The particular device most important on that list is the GPU, which can use only physical RAM. It is true though that if a user space program uses less RAM and instead swaps to virtual memory, then there is more available for the GPU. More precisely though, swap is block device memory.
zram is odd because it is RAM pretending to be a partition. Just calling it a “ramdisk” would imply no compression, whereas zram has compression on the RAM to make it more efficient. Anywhere a swap file or swap partition can be used, so too can zram. zram though is much faster. Sadly, it uses RAM! :) So when zram is used for swap it is an attempt to make swap much faster, and yet make the best use possible of the RAM. RAM pretending to be a disk used by a swap process which is trying to avoid using RAM. Confusing?
zram can be used as an ordinary partition, but you seem to be asking about the case of using it for swap.
If swap is off, then any zram used for swap will still be reserved, but there is no time required to unload it at swap off. If zram is used for swap and has content in it, then turning off swap requires first loading the content from zram back into ordinary RAM (when its identity changes from a block device back to physical memory). This is rather fast compared to turning off swap from a swap file or physical disk partition since RAM is much faster than eMMC or SD cards.
There is no need to ever clear swap files. If you’ve run into a case where you lack enough physical RAM for something requiring this, then turning off eMMC or SD card swap won’t help (e.g., needing more GPU memory). In fact, this would probably hurt. If turning off swap using zram, then initially it will hurt because the zram is still reserved even though user space processes have moved from that to ordinary RAM. If, following that, the zram partitions are themselves released, then all of that RAM becomes available to the GPU (or whatever wanted physical RAM). Keep in mind though that the processes which had previously swapped out to zram are now using regular RAM, and that RAM is not compressed. Depending on the content of the zram at the time of turning off swap and removing zram it might help or hurt what is available to the GPU.
I have doubts about the 4.x or 5.x kernel being a significant part of the difference in timing. I suspect what really matters is (A), what is in the zram, (B) if any swap is going to eMMC or SD, and (C) what overall RAM use changes occur while disabling.
What is it you want to speed up? If it is turning off swap, including zram, then I am skeptical that this will accomplish anything since it depends more on what is using the zram than the mechanism which turns it off.