Reducing time between SPI reads

I am using SPI to read video information via a kernel module thread. The data is time sensitive and must all be read out before the next interrupt trigger. currently the irq wakes the thread and the thread reads data 164 bytes at a time until 60 packets are successfully read out, then scheduled until the next interrupt.

I am required to read the data in blocks of 164 so this cannot change, is it possible to reduce the time between transfer’s as marked by a and b?
This delay takes almost the same amount of time as the data transfer time effectively doubling the time it takes to receive a full segment of data.

Did you try and hit problem?

I don’t understand what you mean, could you please clarify?