About optimize cuda program and get more throughput on T4

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I am optimizing the cuda program to get more throughput. Then I did the following experiment:

  1. Single thread serial executes 6 algorithm models
    The result: up to 24 channels on the T4, and memory and GPU usage are still surplus.
  2. Multithreading executes 6 algorithms in parallel, and uses multiple streams(create a stream by using cudaStreamCreateWithFlags and cudaStreamNonBlocking flags).
    The result: up to 26 runs on the T4, and memory and GPU usage are still surplus.

I looked at nvidia’s official website. It suggests using a multi-threaded default stream to get more throughput. Is this still true in the latest cuda10? And how can i do to get more throughput?

If anyone has any idea, please let me know. Thanks!!!