Ill comment here since our old issue is closed and we forgot about it (the linked one above):
Note that after a more thorough investigation, the delays in kernel did not fix the problem at all. We could make this work for our specific use-case by controlling the underlying USB-hardware in a way that made it work. This was only possible due to the fact that we could create our own firmware for that USB-dongle.