# Problem tensorrt5 python api with flask service 

**URL:** <https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603>\
**Category:** TensorRT\
**Created:** [December 17, 2018, 11:45am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603 "2018-12-17T11:45:37Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![mengzhangjian](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mengzhangjian](https://forums.developer.nvidia.com/u/mengzhangjian)\
**Post date:** [December 17, 2018, 11:45am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/1 "2018-12-17T11:45:37Z")

</div>

Hi, we want to use python flask to start web service to use tensorrt,when program looks like below ,everythining is ok  
@app()  
def inference():  
cuda.init()  
ctx = device.make\_context()  
ctx.push()

```
   engine = build_engie()

```

as you konw,build\_engile will take too much time, this cause tensorrt speed has no advantages.I want to take build\_engine outside of inference,looks like  
def init()  
build\_engine()  
def inference():  
cuda.init()  
ctx = device …  
…  
it will not work when cuda.memcpy\_dtoh\_async() and print :cuStreamSynchronize failed an illegal memory access was encountered .  
please help me find how to solve this

---

<div class="post-metadata">

**Author:** ![NVES](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nves/32/14043_2.png) [@NVES](https://forums.developer.nvidia.com/u/NVES)\
**Post date:** [December 17, 2018, 4:07pm UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/2 "2018-12-17T16:07:04Z")

</div>

Hello,

I’m not sure where or how cuda.memcpy\_dtoh\_async() and print :cuStreamSynchronize are called in your workflow, but I’d recommend serializing your TRT engine/model to a file offline, ready to deploy.

Take a look at this presentation on designing Flask application to expose TRT models via REST endpoints.

> **[s8495-deploying-deep-neural-networks-as-a-service-using-tensorrt-and-nvidia-docker.pdf](https://on-demand.gputechconf.com/gtc/2018/presentation/s8495-deploying-deep-neural-networks-as-a-service-using-tensorrt-and-nvidia-docker.pdf)**
>
> 1166.59 KB

regards,  
NVIDIA Enterprise Support

---

<div class="post-metadata">

**Author:** ![mengzhangjianwi7oq](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mengzhangjianwi7oq](https://forums.developer.nvidia.com/u/mengzhangjianwi7oq)\
**Post date:** [December 18, 2018, 6:34am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/3 "2018-12-18T06:34:45Z")

</div>

@NVES，thanks for your replay.here is the code.

import tensorrt as trt  
import pycuda.driver as cuda  
import pycuda.autoinit

TRT\_LOGGER = trt.Logger()

def get\_engine(engineFilePath):  
with open(engineFilePath,“rb” ) as f,trt.Runtime(TRT\_LOGGRR) as runtime:  
return(runtime.deserialize\_cuda\_engine(f.read()))

@app.route(‘./yoloimg’,methods=[‘’])  
def img():  
cuda.init()  
device = cuda.Device(0)  
ctx = device.make\_context()  
ctx.push()  
\*\*\*\*  
engine = get\_engine()  
inference code  
\*\*\*\*\*\*  
ctx.pop()

above code is ok ,while get\_engine take much time,so i want to put the piece code outside of img,looks like

import tensorrt as trt  
import pycuda.driver as cuda  
import pycuda.autoinit

TRT\_LOGGER = trt.Logger()

def get\_engine(engineFilePath):  
with open(engineFilePath,“rb” ) as f,trt.Runtime(TRT\_LOGGRR) as runtime:  
return(runtime.deserialize\_cuda\_engine(f.read()))  
++++++engine = get\_engine()  
@app.route(‘./yoloimg’,methods=[‘’])  
def img():  
cuda.init()  
device = cuda.Device(0)  
ctx = device.make\_context()  
ctx.push()  
\*\*\*\*  
----- engine = get\_engine()  
inference code  
\*\*\*\*\*\*  
ctx.pop()  
it will make wrong. i find that if

import tensorrt as trt  
import pycuda.driver as cuda  
import pycuda.autoinit  
+++++cuda.init()  
+++++ device = cuda.Device(0)  
+++++ctx = device.make\_context()  
TRT\_LOGGER = trt.Logger()

def get\_engine(engineFilePath):  
with open(engineFilePath,“rb” ) as f,trt.Runtime(TRT\_LOGGRR) as runtime:  
return(runtime.deserialize\_cuda\_engine(f.read()))  
++++++engine = get\_engine()  
++++++ctx.push()  
@app.route(‘./yoloimg’,methods=[‘’])  
def img():  
— cuda.init()  
— device = cuda.Device(0)  
— ctx = device.make\_context()  
ctx.push()  
\*\*\*\*  
----- engine = get\_engine()  
inference code  
\*\*\*\*\*\*  
ctx.pop()  
it will only success once.since cuda.init() only initialize once. can you help me find the solution? if neeed,i will provide you the code.

---

<div class="post-metadata">

**Author:** ![mengzhangjianwi7oq](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@mengzhangjianwi7oq](https://forums.developer.nvidia.com/u/mengzhangjianwi7oq)\
**Post date:** [December 18, 2018, 7:03am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/4 "2018-12-18T07:03:13Z")

</div>

@NVES,close this issue, i have solved

---

<div class="post-metadata">

**Author:** ![chienchih.lin](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@chienchih.lin](https://forums.developer.nvidia.com/u/chienchih.lin)\
**Post date:** [October 27, 2019, 4:19am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/5 "2019-10-27T04:19:28Z")

</div>

@mengzhangjian @mengzhangjianwi7oq  
Could you please share how do you solve this issue? I encounter the same issue as well.

---

<div class="post-metadata">

**Author:** ![462181979](https://developer.download.nvidia.com/images/forums/profile-default-devtalk-84.png) [@462181979](https://forums.developer.nvidia.com/u/462181979)\
**Post date:** [June 18, 2020, 1:04am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/6 "2020-06-18T01:04:53Z")

</div>

I have the same problem,could you share your way to solve this issue?

---

<div class="post-metadata">

**Author:** ![krzysztof.osinski](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/krzysztof.osinski/32/53992_2.png) [@krzysztof.osinski](https://forums.developer.nvidia.com/u/krzysztof.osinski)\
**Post date:** [October 15, 2021, 10:40am UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/7 "2021-10-15T10:40:19Z")

</div>

can you let us know how to solve this, I’ve experience same problem?

---

<div class="post-metadata">

**Author:** ![NVES](https://sea2.discourse-cdn.com/nvidia/user_avatar/forums.developer.nvidia.com/nves/32/14043_2.png) [@NVES](https://forums.developer.nvidia.com/u/NVES)\
**Post date:** [October 15, 2021, 5:02pm UTC](https://forums.developer.nvidia.com/t/problem-tensorrt5-python-api-with-flask-service/68603/8 "2021-10-15T17:02:12Z")

</div>

Hi,  
Please check the below link, as they might answer your concerns

> **[API Reference :: NVIDIA Deep Learning TensorRT Documentation](https://docs.nvidia.com/deeplearning/tensorrt/api/index.html)**
>
> This is the API Reference documentation for the NVIDIA TensorRT library. The following set of APIs allows developers to import pre-trained models, calibrate networks for INT8, and build and deploy optimized networks with TensorRT. Networks can be...

Thanks!