I see what you’re interested in, but this is probably not practical, except perhaps for some corner cases with specifically modified programs. Basically what you have are two full computers, one lighter weight than the other, but having some niche performance capabilities that make it act like a faster computer for those capabilities. The real benefit of the TK1 is in how fast it can be with GPU applications versus the power it draws.
Consider the common case of two full sized computers being connected to each other…typically one can’t behave as a co-processor for the other. You can come up with distributed computing libraries which let you modify individual programs and network them together, e.g., anything suitable for a Beowulf cluster or distributed message protocols.
A common video card could be added and anything on the gaming computer using GPU would take advantage of it, and the separate video card would be far faster than the TK1 (but power hungry). The number of CUDA cores and the dedicated video memory of a typical gaming video card is extreme in comparison…just don’t ask to run it off of a battery for long periods of time the way a TK1 could.
There are some cases where a dedicated TK1 could be a nice appliance. Video conferencing or media streaming would be a nice fit. Lots of people already use a TK1 for vision processing, not many have taken advantage of the rest of the multi-media capabilities for tasks like home theater.
If you have a particular application you want to work on, you could write software which uses the Linux USB gadget interface to have the device appear to be virtually any kind of USB device you want (the TK1 could appear to be a camera, a hard drive, a network hot spot, so on). What it comes down to though is really how much work you’re willing to go through to do it.