Hi,
For using qtmux or mp4mux, you would need to send end-of-stream signal in termination. Please check if you have this. Or may try matroskamux.
Since encoder does not touch/modify timestamps, it is very likely the timestamps are not set correctly.