In my experience you can extend the pixel formats, and you will likely also need to extend the colorfmt table in sensor common. This leads to code that compiles for me, but I am debugging other issues so am not 100% sure if extending these tables will be enough. I do know other manufacturers have done this with cameras, such as LI with the AR0144CS which adds both a PIX_FMT and colorfmt. However, I do not know if they received any special support from NVIDIA when they extended those tables, on the ISP side of things or in any of the binary blobs.
BTW L4T 32.1 was released recently and includes fairly major changes to camera common, so you may want to forget 31.1 and immediately start porting to 32.1 using the new tegracam structs and function calls. 31.1 is, as I understand, just a developer preview.