# Execution time difference of execute() and enqueue()

**URL:** <https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636>\
**Category:** TensorRT\
**Created:** [February 7, 2020, 2:38pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636 "2020-02-07T14:38:23Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 7, 2020, 2:38pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/1 "2020-02-07T14:38:23Z")

</div>

So I changed sampleOnnxMNIST sample so it will read videofile and process frames in loop. And verything was OK.  
But when I changed code from sync:

```auto
auto istart = chrono::high_resolution_clock::now();
buffers.copyInputToDevice();
bool status = context-><b><u>execute</u></b>(mParams.batchSize, buffers.getDeviceBindings().data());
buffers.copyOutputToHost();
chrono::duration<double> i_time_span = chrono::duration_cast<chrono::duration<double>>(chrono::high_resolution_clock::now() - istart);
// count is frame index:
cout << "Inference " << count << ": " << i_time_span.count() << " secs\n";

```

to async:

```auto
auto istart = chrono::high_resolution_clock::now();
// cudaStream_t cstream;
buffers.copyInputToDeviceAsync(cstream);
bool status = context-><b><u>enqueue</u></b>(mParams.batchSize, buffers.getDeviceBindings().data(), cstream, nullptr);
buffers.copyOutputToHostAsync(cstream);
chrono::duration<double> i_time_span = chrono::duration_cast<chrono::duration<double>>(chrono::high_resolution_clock::now() - istart);
cout << "Inference " << count << ": " << i_time_span.count() << " secs\n";

```

I got literally same time of execution. CUDA events give same results as chrono does.  
I checked on PC with GTX 1063 and Jetson Nano — same time of execute() and enqueue() on both.  
I was thinking it should be different.  
Am I doing something wrong?  
Or is it normal when only one inference is happening?

---

<div class="post-metadata">

**Author:** ![SunilJB](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/suniljb/32/14043_2.png) [@SunilJB](https://forums.developer.nvidia.com/u/SunilJB)\
**Post date:** [February 10, 2020, 8:47am UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/2 "2020-02-10T08:47:48Z")

</div>

Hi,

Async copy will impact the response time if there is significant amount of data movement between host and device.  
Could you please share the nvidia profiler output as well so we can help better?

Meanwhile, please refer to below link:  
[https://devblogs.nvidia.com/how-overlap-data-transfers-cuda-cc/](https://devblogs.nvidia.com/how-overlap-data-transfers-cuda-cc/)  
[https://github.com/NVIDIA/TensorRT/blob/07ed9b57b1ff7c24664388e5564b17f7ce2873e5/samples/opensource/sampleMovieLens/sampleMovieLens.cpp#L294](https://github.com/NVIDIA/TensorRT/blob/07ed9b57b1ff7c24664388e5564b17f7ce2873e5/samples/opensource/sampleMovieLens/sampleMovieLens.cpp#L294)

Thanks

---

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 10, 2020, 9:45am UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/3 "2020-02-10T09:45:50Z")

</div>

> [@SunilJB](#):
>
> Could you please share the nvidia profiler output as well so we can help better?

How can I view/create it?

---

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 10, 2020, 12:11pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/4 "2020-02-10T12:11:45Z")

</div>

I swapped copyInputToDeviceAsync() and copyOutputToHostAsync() back to sync-versions keeping asynchronous enqueue().  
Total time per frame didn’t change.  
Also I put CUDA events right before and after execute() and enqueue() so copying data won’t affect measuring:

```auto
cudaEventRecord(cstart, cstream);
bool status = context->enqueue(mParams.batchSize, buffers.getDeviceBindings().data(), cstream, nullptr);
cudaEventRecord(cend, cstream);
cudaEventSynchronize(cend);
float totalTime;
cudaEventElapsedTime(&totalTime, cstart, cend);

```

And I’m stil getting same ~42 ms for both execute() and enqueue().

---

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 11, 2020, 10:06am UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/5 "2020-02-11T10:06:55Z")

</div>

So I checked materials you gave and found that there’s examples for 1-task-multiple-streams only for CUDA w/o TensorRT. TensorRT examples with multiple CUDA streams are used only for multiple inferences (with multiple frames) at once.  
I assume that inference on 1 image can’t be split into multiple streams, am I right?  
Aslo enqueue() as I understand waits till inference completed: _“Asynchronously execute inference on a batch.”_ Just like execute() does.  
So is there any way to:

- just send signal to start inference (only start, not complete)
- do other things (prepare next frame for example) at the same time
- after that check if inference is completed
? Just like OpenVINO does: [i]void InferenceEngine::InferRequest::StartAsync() Start inference of specified input(s) in asynchronous mode. Note: It returns immediately. Inference starts also immediately.[/i] and [i]StatusCode InferenceEngine::InferRequest::Wait() Waits for the result to become available. Blocks until specified millis\_timeout has elapsed or the result becomes available, whichever comes first.[/i]

---

<div class="post-metadata">

**Author:** ![SunilJB](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/suniljb/32/14043_2.png) [@SunilJB](https://forums.developer.nvidia.com/u/SunilJB)\
**Post date:** [February 12, 2020, 6:06am UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/6 "2020-02-12T06:06:31Z")

</div>

> [@](#):
>
> > [@SunilJB](#):
> >
> > Could you please share the nvidia profiler output as well so we can help better?
> 
> How can I view/create it?

Hi,

You can use Nvidia visual profiler. Please refer below link for more details:  
[https://developer.nvidia.com/nvidia-visual-profiler](https://developer.nvidia.com/nvidia-visual-profiler)

Thanks

---

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 12, 2020, 11:08am UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/7 "2020-02-12T11:08:27Z")

</div>

> [@SunilJB](#):
>
> You can use Nvidia visual profiler. Please refer below link for more details:  
> [https://developer.nvidia.com/nvidia-visual-profiler](https://developer.nvidia.com/nvidia-visual-profiler)

Here it is: [asyncOnnxWithLoop.nvvp - Google Drive](https://drive.google.com/open?id=1d3rLmhRlN7zOlUI1AI2tgm9EVxyCaxJ6)  
If I understood correctly what I had to do.

---

<div class="post-metadata">

**Author:** ![christian.m](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@christian.m](https://forums.developer.nvidia.com/u/christian.m)\
**Post date:** [February 12, 2020, 3:48pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/8 "2020-02-12T15:48:04Z")

</div>

This almost tripped me up yesterday but make sure you’re also using pinned memory.

You can only overlap computation with memory copying by using the page-locked memory via

```auto
::cudaMallocHost()

```

> [@](#):
>
> So is there any way to:
> 
> - just send signal to start inference (only start, not complete)
> - do other things (prepare next frame for example) at the same time
> - after that check if inference is completed
> 
> ?

Of course!

You would call `enqueue()` on the stream (this is non-blocking) and then schedule the data transfer back via `::cudaMemcpyAsync`.

To check if the inference is completed on that stream, simply invoke `::cudaStreamQuery`.

You can read about this function here:  
[https://docs.nvidia.com/cuda/cuda-runtime-api/group\_\_CUDART\_\_STREAM.html#group\_\_CUDART\_\_STREAM\_1g2021adeb17905c7ec2a3c1bf125c5435](https://docs.nvidia.com/cuda/cuda-runtime-api/group __CUDART__ STREAM.html#group __CUDART__ STREAM_1g2021adeb17905c7ec2a3c1bf125c5435)

---

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 12, 2020, 5:28pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/9 "2020-02-12T17:28:22Z")

</div>

> [@](#):
>
> You can only overlap computation with memory copying by using the page-locked memory via
> 
> ```auto
> ::cudaMallocHost()
> 
> ```

So I need to allocate memory in RAM using cudaMallocHost() instead of something like this:

```auto
float* hostDataBuffer = static_cast<float*>(buffers.getHostBuffer(mParams.inputTensorNames[0]));

```

Do I understand this correct?

> [@](#):
>
> You would call `enqueue()` on the stream (this is non-blocking) and then schedule the data transfer back via `::cudaMemcpyAsync`.  
> To check if the inference is completed on that stream, simply invoke `::cudaStreamQuery`.  
> You can read about this function here:  
> [CUDA Runtime API :: CUDA Toolkit Documentation](https://docs.nvidia.com/cuda/cuda-runtime-api/group __CUDART__ STREAM.html#group __CUDART__ STREAM_1g2021adeb17905c7ec2a3c1bf125c5435)

You’re saying enqueue() should return immidiately? Is this correct?  
I checked my code again to be sure.  
And understood that I never checked time of sole enqueue() (w/o copying data) with chrono::. I only tried to use CUDA events for that (and I suppose they gave time of full execution, not only starting call enqueue() itself).  
Measured time again (now with chrono) — and got 0.6 ms for enqueue() call on GTX 1063!  
Thanks for pointing at this.  
Now I removed bufferManager method and did copying output manually:

```auto
context->enqueue(mParams.batchSize, buffers.getDeviceBindings().data(), cstream, nullptr);
auto start = chrono::high_resolution_clock::now();
cudaMemcpyAsync(buffers.mManagedBuffers[1]->hostBuffer.data(),
	buffers.mManagedBuffers[1]->deviceBuffer.data(),
	buffers.mManagedBuffers[1]->hostBuffer.nbBytes(),
	cudaMemcpyDeviceToHost, cstream);
auto end1 = chrono::high_resolution_clock::now();
cudaMemcpyAsync(buffers.mManagedBuffers[2]->hostBuffer.data(),
	buffers.mManagedBuffers[2]->deviceBuffer.data(),
	buffers.mManagedBuffers[2]->hostBuffer.nbBytes(),
	cudaMemcpyDeviceToHost, cstream);
auto end2 = chrono::high_resolution_clock::now();

```

But somehow copying data from output takes a lot of time!  
Second cudaMemcpyAsync() (end1-end2) takes only 0.065 msecs.  
But first cudaMemcpyAsync() (start-end1) takes \>3 ms for some reason.  
Feels like cudaMemcpyAsync() waits untill execution is completed. Is it so?

---

<div class="post-metadata">

**Author:** ![christian.m](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@christian.m](https://forums.developer.nvidia.com/u/christian.m)\
**Post date:** [February 13, 2020, 4:25pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/10 "2020-02-13T16:25:48Z")

</div>

> [@](#):
>
> So I need to allocate memory in RAM using cudaMallocHost() instead of something like this:
> 
> ```auto
> float* hostDataBuffer = static_cast<float*>(buffers.getHostBuffer(mParams.inputTensorNames[0]));
> 
> ```
> 
> Do I understand this correct?

You understand correctly but keep in mind, I’m not sure of the implementation of the buffer class you’re using there. It very well could be using cudaMallocHost or cudaHostAlloc.

The thing to take away is that overlapping copying and kernel execution requires page-locked host mem which you can only get via cudaMallocHost.

> [@](#):
>
> You’re saying enqueue() should return immidiately? Is this correct?

It _should_, in theory. Most things are non-blocking. I just know that `execute` synchronizes with the default stream before it returns.

> [@](#):
>
> I checked my code again to be sure.  
> And understood that I never checked time of sole enqueue() (w/o copying data) with chrono::. I only tried to use CUDA events for that (and I suppose they gave time of full execution, not only starting call enqueue() itself).  
> Measured time again (now with chrono) — and got 0.6 ms for enqueue() call on GTX 1063!  
> Thanks for pointing at this.  
> Now I removed bufferManager method and did copying output manually:
> 
> ```auto
> context->enqueue(mParams.batchSize, buffers.getDeviceBindings().data(), cstream, nullptr);
> auto start = chrono::high_resolution_clock::now();
> cudaMemcpyAsync(buffers.mManagedBuffers[1]->hostBuffer.data(),
> buffers.mManagedBuffers[1]->deviceBuffer.data(),
> buffers.mManagedBuffers[1]->hostBuffer.nbBytes(),
> cudaMemcpyDeviceToHost, cstream);
> auto end1 = chrono::high_resolution_clock::now();
> cudaMemcpyAsync(buffers.mManagedBuffers[2]->hostBuffer.data(),
> buffers.mManagedBuffers[2]->deviceBuffer.data(),
> buffers.mManagedBuffers[2]->hostBuffer.nbBytes(),
> cudaMemcpyDeviceToHost, cstream);
> auto end2 = chrono::high_resolution_clock::now();
> 
> ```
> 
> But somehow copying data from output takes a lot of time!  
> Second cudaMemcpyAsync() (end1-end2) takes only 0.065 msecs.  
> But first cudaMemcpyAsync() (start-end1) takes \>3 ms for some reason.  
> Feels like cudaMemcpyAsync() waits untill execution is completed. Is it so?

Typically, you only see speedups when you’re doing things in parallel. If you’re only doing a single inference batch, you likely won’t see any speedup.

Instead, copy-and-execute is useful when you’re performing multiple inference batches at once. For example, you can copy over the second batch to run your inferences over while you’re still calculating the inferences for your first batch.

This kind of stuff can help you fully saturate your GPU as most of these inference routines seem to have a theoretical utilization of like 50%.

---

<div class="post-metadata">

**Author:** ![redoutracer](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@redoutracer](https://forums.developer.nvidia.com/u/redoutracer)\
**Post date:** [February 13, 2020, 6:41pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/11 "2020-02-13T18:41:35Z")

</div>

> [@](#):
>
> You understand correctly but keep in mind, I’m not sure of the implementation of the buffer class you’re using there. It very well could be using cudaMallocHost or cudaHostAlloc.

Thanks.  
Also I checked implementation and didn’t find any of these two. Only **malloc** for host and **cudaMalloc** for device.

> [@](#):
>
> For example, you can copy over the second batch to run your inferences over while you’re still calculating the inferences for your first batch.

I was curious and did almost same today. But only prepared new batch while inference is running, not copied to input. And it almost doubled FPS on Jetson Nano: from 82-83 ms/frame to 45 ms/frame.  
So as I undestand now I can also copy to input buffer while inference is still running? That’s nice.  
Didn’t try cudaMallocHost/cudaHostAlloc yet, I’ll try this tomorrow.

---

<div class="post-metadata">

**Author:** ![TomNVIDIA](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/tomnvidia/32/14181_2.png) [@TomNVIDIA](https://forums.developer.nvidia.com/u/TomNVIDIA)\
**Post date:** [October 12, 2021, 7:47pm UTC](https://forums.developer.nvidia.com/t/execution-time-difference-of-execute-and-enqueue/111636/12 "2021-10-12T19:47:50Z")

</div>

Closed on October 12, 2021, 7:47pm UTC


