Hi there. I am wondering if it’s possible to run a Tensorflow-TensorRT inference server with Docker using a JetPack device, e.g. Xavier, Nano, etc. My current approach is slow, since I need to load the model from a script each time I want to run inference.
Having it as an always loaded callable service would be much more productive.
There was a previous thread here about doing this, but the state of current support is still unclear to me. This pull request merged Jetson support, but I’m not sure on the process of how I’d use this in a Dockerfile.
Hi,
First, you can find a Triton server example in this repository:
The pipeline is triggered by the source component.
So you can update it based on your use case.
Thanks.
Thanks, this seems close to what I’m looking for. However, the repo is still not very mature, so I haven’t been able to follow all of the steps.
However, I’ve improved the README and TensorRT conversion script to use a new version of the Docker container, and TF2.0.
You can see it on my fork here. I’ll do a pull request once I get this final kink worked out.
I get through all of the steps, expect running the deepstream-app. It fails with:
$ deepstream-app -c config/source1_primary_faster_rcnn_inception_v2.txt
** ERROR: <create_multi_source_bin:1057>: Failed to create element 'src_bin_muxer'
** ERROR: <create_multi_source_bin:1132>: create_multi_source_bin failed
** ERROR: <create_pipeline:1296>: create_pipeline failed
** ERROR: <main:636>: Failed to create pipeline
Quitting
App run failed
Is there an obvious problem here?
Hi,
Could you clean the cache and try it again?
$ rm ${HOME}/.cache/gstreamer-1.0/registry.aarch64.bin
Thanks.
Hi. I tried that, however got the error:
nvbuf_utils: Could not get EGL display connection
2021-01-06 10:52:05.969154: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudar
t.so.10.2
No EGL Display
nvbufsurftransform: Could not get EGL display connection
nvbuf_utils: Could not get EGL display connection
2021-01-06 10:52:06.537738: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudar
t.so.10.2
No EGL Display
nvbufsurftransform: Could not get EGL display connection
No EGL Display
nvbufsurftransform: Could not get EGL display connection
[repeats a lot ...]
No EGL Display rejected because of wrong authentication.
nvbufsurftransform: Could not get EGL display connection
** ERROR: <create_multi_source_bin:1057>: Failed to create element 'src_bin_muxer'
** ERROR: <create_multi_source_bin:1132>: create_multi_source_bin failed
** ERROR: <create_pipeline:1296>: create_pipeline failed
** ERROR: <main:636>: Failed to create pipeline
Quitting
App run failed
I followed the original instructions to use --net=host flag for running the Docker container, and mounting X11 with -v /tmp/.X11-unix/:/tmp/.X11-unix
I have another trick up my sleeve though, which is prefixing my docker run with xhost +, which is bad security practice, but at least gets me a bit further:
2021-01-06 10:54:42.296824: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
** ERROR: <parse_config_file:513>: parse_config_file failed
** ERROR: <main:627>: Failed to parse config file 'config/source1_primary_faster_rcnn_inception_v2.txt'
Quitting
App run failed
This suggests that the config in the current repo needs some tweaking to be compatible with the most recent versions of the toolchains?
Hi,
The final error indicates that Deepstream cannot parse the source1_primary_faster_rcnn_inception_v2.txt.
Is this a default configure file or a modified version?
Please noted that the Triton server is supported from Deepstream 5.0.
And the Triton server for Jetson should be v20.03.
Thanks.
This is a default configuration file, it hasn’t been modified. Continued to have issues, I think the README on the GitHub has some major issues of correctness.
However, I got further by following the blog-post it is based on.
Running the following steps from the blog (condensed here for easy reproducibility):
-
Pull image: docker pull nvcr.io/nvidia/deepstream-l4t:5.0.1-20.09-samples (latest version).
-
Run image: sudo docker run --runtime=nvidia --network host \ -v $PWD:/workspace/ \ -ti nvcr.io/nvidia/deepstream-l4t:5.0.1-20.09-samples bash -i
-
Download model, prepare configs:
cd samples/trtis_model_repo
cd ssd_inception_v2_coco_2018_01_28/
wget http://download.tensorflow.org/models/object_detection/faster_rcnn_inception_v2_coco_2018_01_28.tar.gz
tar xvf faster_rcnn_inception_v2_coco_2018_01_28.tar.gz
mkdir 1
cp faster_rcnn_inception_v2_coco_2018_01_28/frozen_inference_graph.pb 1/model.graphdef
wget https://raw.githubusercontent.com/NVIDIA-AI-IOT/deepstream_triton_model_deploy/master/faster_rcnn_inception_v2/config/labels.txt
wget https://raw.githubusercontent.com/NVIDIA-AI-IOT/deepstream_triton_model_deploy/master/faster_rcnn_inception_v2/config/config_infer_primary_faster_rcnn_inception_v2.txt
wget https://raw.githubusercontent.com/NVIDIA-AI-IOT/deepstream_triton_model_deploy/master/faster_rcnn_inception_v2/config/source1_primary_faster_rcnn_inception_v2.txt
- Run the model:
deepstream-app -c source1_primary_faster_rcnn_inception_v2.txt
However, running the app fails with the error: deepstream-app: error while loading shared libraries: libnvinfer.so.7: cannot open shared object file: No such file or directory.
If I look in the /usr/lib/aarch64-linux-gnu of the container, I can see that libnvinfer.so.7 isn’t there. But I thought that’s what passing the --runtime=nvidia flag to Docker gives the container access to whatever it needs. The library is present in my base system.
Any idea what the issue could be?
Hi,
Do you find the library once you run the container with --runtime=nvidia?
The plugin will check the csv file located at /etc/nvidia-container-runtime/host-files-for-container.d/ and do the corresponding mapping.
So you can access the library within the container.
Thanks.