I don’t know what would be the cause, but I can only speculate that either it was being starved from CPU time by another process, or by a lack of data throughput. I would look closely to see what other processes may have gone up in CPU use as yours drops in CPU use…it wouldn’t necessarily have to be something taking a lot of CPU, but any one or more processes which might push your process away would be suspicious.
If this latter is not the case, then you might be interested memory use by other processes. It could be a case of data transfer slowing if there is a lack of ram suitable for your process. There are different types of memory (and different requirements due to those types), so it wouldn’t necessarily be a lot of use in general (meaning the extra use may not seem like a large increase, but the nature of the ram used by that small case might have a bigger impact than it would appear at first).
If it does look like perhaps there is a memory issue, you might try adding an SD card and formatting it as swap. Enable that as swap, and do the same htop monitoring again; swap should start out unused, and if there is no swap being used at the time when the problem occurs, then it isn’t likely a ram usage issue. If it does start swapping out it may or may not run faster, but you would notice swap usage building up.