Running TensorRT Inference Server without Docker Command

AI & Data Science Deep Learning (Training & Inference) TensorRT
1

Hi all,

I am currently trying to run tensorrt inference server and I followed instructions listed here: [url]Documentation – Pre-release :: NVIDIA Deep Learning Triton Inference Server Documentation

I have successfully built the server from source with correcting a few C++ codes. However, there is literally no instruction about running the server without Docker command. Documentation – Pre-release :: NVIDIA Deep Learning Triton Inference Server Documentation

It doesn’t make sense if you put some instructions about building a software while not giving any info when it comes to run it.

2

Hi doruk898,

I reached out to the TRTIS team about updating the docs for this case (running TRT server on host after building from source), and they created a task to do so.

In the meantime, I believe you can navigate into the trtis/install directory created during the build process as mentioned here: https://docs.nvidia.com/deeplearning/sdk/tensorrt-inference-server-guide/docs/build.html#build-inference-server

And then you can run the executable in there:

# Inside trtis/install directory after building...
./trtserver --model-store=<path_to_models>

This is essentially the same way that you would run it if executing in a docker container:

nvidia-docker run <container_config_flags> ... trtserver --model-store=<path_to_models>

Thanks,
NVIDIA Enterprise Support