Gadget was designed only for standard classes. You won’t be able to use it as a framework for bulk transmission. You can study the code and learn from it, but you will be writing your driver from scratch. That implies the driver at both sides will be custom.
I do not know about the error, but likely that if you are trying to use gadget, then this error is because it is missing a configuration step…the one turning it into mass storage.
A question arises as to whether the nature of your data requires custom drivers, or if you could treat this like a partition (where the data partition might be VFAT or ext4 or even some custom file system type…maybe a binary unformatted partition) and simply allow the other end to look at a partition.
Partitions can be used with and without formatting. Partitions can be read or written at any offset for any span of bytes. Partitions can be marked read only or even write only. Knowing more about your use of the data, and how the data might change would help. Otherwise the only possibility is to write everything from scratch for both sides of the USB connection.