Most of the extra packages for CUDA and related go to “/usr/local”. You may find far simpler and effect results simply mounting the SD card on “/usr/local” after moving content from the existing "/usr/local/* to the SD card partition.
You could do something similar with “/home”, but this can be more complicated than it sounds. Any subdirectory of “/home/the_user_name/somewhere/” is an easy choice though.
I couldn’t tell you what all of the reasons are why one card may work and not another. There are definitely variations in each SD card, even when the same model is used. Typically, if you wanted to make sure something would work over a wide quality range, then the clock speed would be cut back. On the other hand, the industry is both speed hungry and cost-aware. The goals are always in a constant war with each other.
Signal quality in general is a huge topic, and even the slightest difference in layout can change things. This includes the layout both inside and outside of the SD card. Slightly higher or lower power delivery voltage to the SD card (or most any digital component) can also change stability and reliability (we’re talking small amounts, e.g., millivolts or centivolts). Overclockers of PCs know this well, along with fighting the extra heat from even a small increase in voltage/clock. You’d have to look at an individual case with expensive equipment to know about that specific case…and it may or may not apply to an entire brand or line of hardware.
FYI, I think timing requirements are tighter during boot versus after boot.