The graphics driver is tied to the X server ABI, and the X server is normally bound to a particular kernel (or at least a kernel series). The X and graphics driver mix changed quite a bit when going from the 3.x kernels to the 4.x kernels. I would guess that if you have a 4.x kernel that the R27.1 X server would work (it might still need porting if it isn’t the 4.x kernel from the R27.1 sample rootfs). I would expect the X server from R24.2.1 or earlier to be difficult to get working with a 4.x kernel (R24.2.1 and earlier depend on a 3.x kernel series).
I don’t know how compatible the R27.1 X server and graphics driver are with the TX1 since R27.1 was intended for a TX2…no doubt the TX2 graphics hardware has a lot of similarities, but there are also probably differences. Unless there is some urgent need I would think it might be better to wait for the next L4T release…then the TX1 will have a 4.x kernel with support from NVIDIA and full hardware acceleration (including CUDA).