nvinfer serialization path?

Is there a way to get nvinfer to spit out a serialized model somewhere specific, or will it always use the same path as the model? I mentioned it in another thread, but it would be nice if a user cache could be used by default (~/.deepstream/cache or something). My deepstream app installs models to /usr/local/share/… and I don’t want to make that writable.

Hi,
There are two properties in nvinfer plugin:

> config-file-path    : Path to the configuration file for this instance of nvinfer
                        flags: readable, writable, changeable only in NULL or READY state
                        String. Default: ""

> model-engine-file   : Absolute path to the pre-generated serialized engine file for the model
                        flags: readable, writable, changeable only in NULL or READY state
                        String. Default: ""

It should work by configuring these properly. Please share more information about what you would like to do and why the current implementation cannot achieve it. Thanks.

DaneLLL,

Thanks for the response but I tried that and nvinfer doesn’t seem to care about what I set model-engine-file to at runtime. This is the behavior I am experiencing. Here is where I set properties on the nvinfer element:

# inference bin constuctor (this is Genie, not Python, but you get the idea):
	...
	self.pie.set_property("config-file-path", pie_config != null ? pie_config : DEFAULT_PIE_CONFIG)
	self.pie.set_property("model-engine-file", self.model_engine_file)

I’ve triple-checked that the properties are set on the nvinfer element (in my case to “/home/username/.app/redaction.engine”), and this path exists and is writable, but when the pipeline starts, nvinfer still wants to write to /usr/local/share where the model is loaded from in the config file.

0:00:00.818962868 20892   0x5588934200 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:00.819060760 20892   0x5588934200 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:15.592730144 20892   0x5588934200 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /usr/local/share/app/models/fd_lpd.caffemodel_b1_fp32.engine

Am I doing something wrong?

Hi,
Please match the file name with batch-size, network-mode and give it a try.
For example,

model-engine-file=model_b1_fp32.engine

means batch-size=1 and network-mode=0(FP32)

That didn’t fix it. Still trying to store to the original filename at the original path.

[username@hostname] -- [~/.nvalhalla] 
 $ nvalhalla --uri $(youtube-dl -f best -g https://www.youtub3.com/watch?v=awdX61DPWf4)
Creating LL OSD context new
0:00:00.801387122  5992   0x558cf24070 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:00.801466806  5992   0x558cf24070 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:12.902342913  5992   0x558cf24070 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /usr/local/share/nvalhalla/models/fd_lpd.caffemodel_b1_fp32.engine
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Creating LL OSD context new
^CProcess 5992 has received SIGINT, ending...
[username@hostname] -- [~/.nvalhalla] 
 $ nvalhalla --uri $(youtube-dl -f best -g https ://www.youtub3.com/watch?v=awdX61DPWf4) --uri $(youtube-dl -f best -g https://www.youtub3.com/watch?v=FPs_lU01KoI)
Creating LL OSD context new
0:00:00.793193468  6034   0x55ba351630 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:00.793282656  6034   0x55ba351630 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:13.532042957  6034   0x55ba351630 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<pie> NvDsInferContext[UID 1]:generateTRTModel(): Storing the serialized cuda engine to file at /usr/local/share/nvalhalla/models/fd_lpd.caffemodel_b1_fp32.engine
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261 
Creating LL OSD context new
Opening in BLOCKING MODE 
NvMMLiteOpen : Block : BlockType = 261 
NVMEDIA: Reading vendor.tegra.display-size : status: 6 
NvMMLiteBlockCreate : Block : BlockType = 261

nvinfer “model-engine-file” property is set to “/home/username/.nvalhalla/redaction_b#_fp32.engine” but it doesn’t want to generate it there. I will attach a pdf of the pipeline (just after generation, before the sources are connected from the uridecodebin).
engine-filename-problem.pdf (33.5 KB)

Hi,
deepstream-test3 is based on uridecodebin. Could you please share a patch on it so that we can reproduce it?

Example is not deepstream-test3. Code is here in dev branch:
https://github.com/mdegans/nvalhalla

it dumps .dot files to ~/.nvlhalla/ and --gst-debug-level works

… but if you want I can probably patch deepstream-test3 so it breaks similarly. You prefer Python or C? or is the Genie above ok?

same issue with deepstream_test_3.py

WARNING: Overriding infer-config batch-size 1  with number of sources  2  

<b>setting model-engine-file property on pgie to /home/username/.pyds/resnet10_b2_int8.engine</b>
Adding elements to Pipeline 

Linking elements in the Pipeline 

Now playing...
1 :  https://r2---sn-nx5s7n7d.googlevideo.com/videoplayback?expire=1583300484&ei=JOteXpbvF9iMkwaJraP4Dw&ip=24.143.115.166&id=o-AJm41vM3QxQA_RlS1LHtHFe8zQFJb7YjclaLxQfFVbTb&itag=22&source=youtube&requiressl=yes&mm=31%2C29&mn=sn-nx5s7n7d%2Csn-nx5e6nez&ms=au%2Crdu&mv=m&mvi=1&pl=18&initcwndbps=2107500&vprv=1&mime=video%2Fmp4&ratebypass=yes&dur=6028.527&lmt=1582233218947338&mt=1583278772&fvip=2&fexp=23842630&c=WEB&txp=7316222&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cratebypass%2Cdur%2Clmt&sig=ADKhkGMwRQIhANmxsc2t0MHPwKffU6S2MB-AwD-naFUieVfq9eLb0q4gAiBMcvgZzT3mSma4pf32s_t_3qIe02_vr2osVM2OzP5OuQ%3D%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABSNjpQwRAIgHbPEuh5OCByxSRDTz3NcK_2tHQXni1nFhoIuWu-cqVQCIHsk-EoOs_TG9ud-SL6fa4oeVT71g9R6egGgS2PVdPI4
2 :  https://r6---sn-nx5e6nez.googlevideo.com/videoplayback?expire=1583300487&ei=J-teXqWmHIHLkgbyqr2IBg&ip=24.143.115.166&id=o-AAnABEhbC4xu1eHygPQu4Zjh4a9t0VPXZBF8UHd0-KRT&itag=22&source=youtube&requiressl=yes&mm=31%2C29&mn=sn-nx5e6nez%2Csn-nx5s7n7s&ms=au%2Crdu&mv=m&mvi=5&pl=18&initcwndbps=2107500&vprv=1&mime=video%2Fmp4&ratebypass=yes&dur=11984.584&lmt=1579852275658714&mt=1583278772&fvip=2&fexp=23842630&c=WEB&txp=5531432&sparams=expire%2Cei%2Cip%2Cid%2Citag%2Csource%2Crequiressl%2Cvprv%2Cmime%2Cratebypass%2Cdur%2Clmt&sig=ADKhkGMwRgIhAPXNQI_VbLKVQboFxPOhj4bLWZfBTTvLtGZ092HLSW5vAiEAst55NMUwre59w-zg3z6-5fBDKVy3yC8mFARFE_SQ0CQ%3D&lsparams=mm%2Cmn%2Cms%2Cmv%2Cmvi%2Cpl%2Cinitcwndbps&lsig=ABSNjpQwRQIhAJm3u05Rqvt7YK9h1igQm3tXxhdK5X6h-3m9TABMMU-5AiB2hj7RBVnj4pfJc51RL5AVgqjkI6N9VpOkVS3RR4QpMw%3D%3D
Starting pipeline 


Using winsys: x11 
Creating LL OSD context new
0:00:00.790471152  9634     0x2d1dbc40 WARN                 nvinfer gstnvinfer.cpp:515:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]:useEngineFile(): Failed to read from model engine file
0:00:00.790540980  9634     0x2d1dbc40 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]:initialize(): Trying to create engine from model files
0:00:33.699119988  9634     0x2d1dbc40 INFO                 nvinfer gstnvinfer.cpp:519:gst_nvinfer_logger:<primary-inference> NvDsInferContext[UID 1]:generateTRTModel(): <b>Storing the serialized cuda engine to file at /opt/nvidia/deepstream/deepstream-4.0/samples/models/Primary_Detector/resnet10.caffemodel_b2_int8.engine</b>

Here is the patch

--- a/apps/deepstream-test3/deepstream_test_3.py
+++ b/apps/deepstream-test3/deepstream_test_3.py
@@ -20,6 +20,7 @@
 # DEALINGS IN THE SOFTWARE.
 ################################################################################
 
+import os
 import sys
 sys.path.append('../')
 import gi
@@ -53,6 +54,16 @@ TILED_OUTPUT_HEIGHT=1080
 GST_CAPS_FEATURES_NVMM="memory:NVMM"
 pgie_classes_str= ["Vehicle", "TwoWheeler", "Person","RoadSign"]
 
+
+def engine_filename(engine_basename:str, num_sources:int) -> str:
+    homedir = os.path.expanduser('~')
+    appdir = os.path.join(homedir, '.pyds')
+    os.makedirs(appdir, mode=0o755, exist_ok=True)
+    # the last bit will have to change for Nano
+    engine_filename = f"{engine_basename}_b{num_sources}_int8.engine"
+    return os.path.join(appdir, engine_filename)
+
+
 # tiler_sink_pad_buffer_probe  will extract metadata received on OSD sink pad
 # and update params for drawing rectangle, object information etc.
 def tiler_src_pad_buffer_probe(pad,info,u_data):
@@ -287,6 +298,9 @@ def main(args):
     if(pgie_batch_size != number_sources):
         print("WARNING: Overriding infer-config batch-size",pgie_batch_size," with number of sources ", number_sources," \n")
         pgie.set_property("batch-size",number_sources)
+    model_engine_file = engine_filename("resnet10", number_sources)
+    pgie.set_property('model-engine-file', model_engine_file)
+    print(f"setting model-engine-file property on pgie to {model_engine_file}")
     tiler_rows=int(math.sqrt(number_sources))
     tiler_columns=int(math.ceil((1.0*number_sources)/tiler_rows))
     tiler.set_property("rows",tiler_rows)

Hi,
We put the engine file at

$ ls -all /home/nvidia/resnet10.caffemodel_b1_int8.engine
-rw-rw-r-- 1 nvidia nvidia 4380695  三   4 17:08 /home/nvidia/resnet10.caffemodel_b1_int8.engine

and set

g_object_set (G_OBJECT (pgie), "model-engine-file",
      "/home/nvidia/resnet10.caffemodel_b1_int8.engine", NULL);

It does not re-generate the engine file. Have not tried python code, but it should work the same ideally

That isn’t a problem. If the .engine is already generated, I assume it will load if set to the path at runtime.

Unfortunately, if you apply that patch you will see that the model-engine-file property is not respected for writing the engine file if it needs to be generated. It always tries the model location.

This makes my app design more difficult since the model install path is not writable and I don’t intend on changing that. Best workaround I can see for me right now is to copy the model on app start to my ~/.app/models folder at startup, calculating the mangled .engine filename from the model’s absolute path and trying to use that.

Ideally, model-engine-file would just be a basename and nvinfer could figure out what .engine it needs to generate or load. Alternatively, a model-engine-path parameter which would dump/load the usual generated .engine name of model_bsize_precision.engine. A cache path is what I am hoping for, basically.

Hi,
Please check useEngineFile() in

deepstream_sdk_v4.0.2_jetson\sources\libs\nvdsinfer\nvdsinfer_context_impl.cpp

The default code is

ifstream gieModelFile(initParams.modelEngineFilePath);
if (!gieModelFile.good())
{
    printWarning("Failed to read from model engine file");
    return NVDSINFER_CONFIG_FAILED;
}

You may take a look and customize it to fit your usecase.

Thanks. That’s a big help. Is there a git repository I can clone? I’d like to push changes upstream if possible so people who download my app don’t need a custom version of nvinfer.

Hi,

No. It is installed through SDKManager and currently there is no git repository. Since these are open source code, it should be fine if you push it to your own github.

I will do that and submit a patch via email/pm if I make any changes. For now, however. I think I am just going to have my app ln/copy it’s model into ~/.app/models/ on app start if it doesn’t already exist, then just set the model-engine-file at runtime. Thanks again for your advice and help. I understand a lot more now about how these properties are used.