pycuda._driver.LogicError: cuMemcpyDtoHAsync failed: an illegal memory access was encountered[SOLVED]

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

I am trying to save and then load a tensorrt engine in in the python API with tensorrt 4 but i get the following error: "pycuda._driver.LogicError:
“pycuda._driver.LogicError: cuMemcpyDtoHAsync failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuStreamDestroy failed: an illegal memory access was encountered”

If I don’t load the engine it works:

plugin_factory = tensorrtplugins.PRELUPluginFactory()

    # Convert caffe model to TensorRT engine
    runtime = trt.infer.create_infer_runtime(G_LOGGER)
    engine = trt.utils.caffe_to_trt_engine(G_LOGGER,
        MODEL_PROTOTXT,
        CAFFE_MODEL,
        1,
        1 << 20,
        OUTPUT_LAYERS,
        trt.infer.DataType.FLOAT,
        plugin_factory)

data = np.load(os.path.join(DATA_DIR, "im_data.npy")).astype(np.float32)

    context = engine.create_execution_context()

    out = infer(context, data, OUTPUT_SIZE, 1)

# Destroy engine
    runtime.destroy()
    engine.destroy()
    context.destroy()

    # Destroy plugin created by factory
    plugin_factory.destroyPlugin()

But if I try to save it and load it then it gives me the error:

plugin_factory = tensorrtplugins.PRELUPluginFactory()

    # Convert caffe model to TensorRT engine
    runtime = trt.infer.create_infer_runtime(G_LOGGER)
    engine = trt.utils.caffe_to_trt_engine(G_LOGGER,
        MODEL_PROTOTXT,
        CAFFE_MODEL,
        1,
        1 << 20,
        OUTPUT_LAYERS,
        trt.infer.DataType.FLOAT,
        plugin_factory)

    trt.utils.write_engine_to_file("PReLU1.engine",engine.serialize())
    engine.destroy()

    engine = trt.utils.load_engine(G_LOGGER, "PReLU1.engine", plugin_factory)

    data = np.load(os.path.join(DATA_DIR, "im_data.npy")).astype(np.float32)

    context = engine.create_execution_context()

    out = infer(context, data, OUTPUT_SIZE, 1)

    # Destroy engine
    runtime.destroy()
    engine.destroy()
    context.destroy()

    # Destroy plugin created by factory
    plugin_factory.destroyPlugin()
2

SOLVED: there was an issue in serialize (PluginFactory)

3 
pycuda._driver.LogicError: cuMemcpyDtoHAsync failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuMemFree failed: an illegal memory access was encountered
PyCUDA WARNING: a clean-up operation failed (dead context maybe?)
cuStreamDestroy failed: an illegal memory access was encountered

@stiven.kullastyz5
I have same problem, Could u give me some advise or solution?
Thank u

4

I want to use load engine file’s method.
so maybe u have some solution in serialize well.

5

I resolved it ,please try trt version 5.0.2.6 and replace the API tensorrt.legency.lite.engine()

6

Can U tell me what did U do to the serialize?

7

Solved? A favor is in need.