# Triton Inference Server (archived) **URL:** https://forums.developer.nvidia.com/c/ai-data-science/deep-learning/triton-inference-server/97.md [Latest](https://forums.developer.nvidia.com/latest.md) · [Categories](https://forums.developer.nvidia.com/categories.md) · [Tags](https://forums.developer.nvidia.com/tags.md) --- ## [Triton Inference Server support available on GitHub](https://forums.developer.nvidia.com/t/triton-inference-server-support-available-on-github/190337)
**Author:** [@TomNVIDIA](https://forums.developer.nvidia.com/u/TomNVIDIA)\ **Replies:** 0\ **Last updated:** [September 28, 2021, 3:45pm UTC](https://forums.developer.nvidia.com/t/triton-inference-server-support-available-on-github/190337 "2021-09-28T15:45:08Z")
To better serve our TIS users, please post your Triton Inference Server issues on our Github instance. This forum is locked as of 9/28/21. --- ## [Triton Inference Server, Model Analyzer](https://forums.developer.nvidia.com/t/triton-inference-server-model-analyzer/284829)
**Author:** [@uss0403](https://forums.developer.nvidia.com/u/uss0403)\ **Replies:** 0\ **Last updated:** [March 4, 2024, 9:34am UTC](https://forums.developer.nvidia.com/t/triton-inference-server-model-analyzer/284829 "2024-03-04T09:34:43Z")
Hello, NVIDIA I am utilizing the excellent features of the NVIDIA Triton Inference Server. In particular, I am utilizing the Model Analyzer in Triton Inference Server to estimate the approximate throughput for models wh… --- ## [Issues with using multiple perf-analyzer processes for Triton Inference Server](https://forums.developer.nvidia.com/t/issues-with-using-multiple-perf-analyzer-processes-for-triton-inference-server/281800)
**Author:** [@adarsh.maurya1](https://forums.developer.nvidia.com/u/adarsh.maurya1)\ **Replies:** 0\ **Last updated:** [February 5, 2024, 6:49am UTC](https://forums.developer.nvidia.com/t/issues-with-using-multiple-perf-analyzer-processes-for-triton-inference-server/281800 "2024-02-05T06:49:06Z")
Description Hi all I am trying to gauge triton-inference-server performance on my hardware with the models I have and the desire is to get as many requests as possible for all of the models that I have. The triton-infer… --- ## [Inference speed of Triton Server](https://forums.developer.nvidia.com/t/inference-speed-of-triton-server/276646)
**Author:** [@danishshres](https://forums.developer.nvidia.com/u/danishshres)\ **Replies:** 0\ **Last updated:** [December 19, 2023, 9:04pm UTC](https://forums.developer.nvidia.com/t/inference-speed-of-triton-server/276646 "2023-12-19T21:04:29Z")
\##Description I am encountering an issue related to latency in the Triton server when performing inference on a model. It appears that the latency is influenced by whether only one model is loaded into the GPU or if the… --- ## [GPUs are underutilized with Triton](https://forums.developer.nvidia.com/t/gpus-are-underutilized-with-triton/273160)
**Author:** [@evan53](https://forums.developer.nvidia.com/u/evan53)\ **Replies:** 2\ **Last updated:** [November 22, 2023, 1:02pm UTC](https://forums.developer.nvidia.com/t/gpus-are-underutilized-with-triton/273160 "2023-11-22T13:02:41Z")
Description We are currently using Triton on EKS in AWS scheduled on a g4dn.xlarge node type (Tesla T4). We were doing a couple of load tests, running requests concurrently and we are noticing that our GPUs are not utili… --- ## [Tensorflow Segmentation Model Deployment On Triton Inference Server](https://forums.developer.nvidia.com/t/tensorflow-segmentation-model-deployment-on-triton-inference-server/187135)
**Author:** [@skilic](https://forums.developer.nvidia.com/u/skilic)\ **Replies:** 1\ **Last updated:** [September 30, 2021, 10:48pm UTC](https://forums.developer.nvidia.com/t/tensorflow-segmentation-model-deployment-on-triton-inference-server/187135 "2021-09-30T22:48:56Z")
Hello everyone, I have Jetson Nano, Jetpack 4.6 and I build Triton Inference Server from build by using latest release(Release 2.12.0). I was able to successfully serve classification models before. Now, I am trying to… --- ## [Convert cuda module (StyleGAN v2 Pytorch) to torchscript (pt)](https://forums.developer.nvidia.com/t/convert-cuda-module-stylegan-v2-pytorch-to-torchscript-pt/187163)
**Author:** [@c.renaudineau](https://forums.developer.nvidia.com/u/c.renaudineau)\ **Replies:** 1\ **Last updated:** [September 30, 2021, 10:31pm UTC](https://forums.developer.nvidia.com/t/convert-cuda-module-stylegan-v2-pytorch-to-torchscript-pt/187163 "2021-09-30T22:31:10Z")
Hello, I want to convert .pth model from StyleGAN v2 (pytorch implementation) to .pt (torchscript model) for triton inference with jit trace. How I could do that ? How I could convert cuda function to torchscript ? fus… --- ## [Trition exiting after every batch inference on cpu](https://forums.developer.nvidia.com/t/trition-exiting-after-every-batch-inference-on-cpu/184712)
**Author:** [@amirkhan](https://forums.developer.nvidia.com/u/amirkhan)\ **Replies:** 5\ **Last updated:** [September 30, 2021, 10:30pm UTC](https://forums.developer.nvidia.com/t/trition-exiting-after-every-batch-inference-on-cpu/184712 "2021-09-30T22:30:21Z")
Hi, I am using onnx language model in trition, my config file content name: “textmodel.onnx”, platform: “onnxruntime\_onnx”, backend: “onnxruntime”, version\_policy: { latest: { num\_versions: 1 } }, max\_batch\_siz… --- ## [Deepstream nvinfer-server with https endpoint](https://forums.developer.nvidia.com/t/deepstream-nvinfer-server-with-https-endpoint/187958)
**Author:** [@dharmil26shah](https://forums.developer.nvidia.com/u/dharmil26shah)\ **Replies:** 1\ **Last updated:** [September 30, 2021, 10:28pm UTC](https://forums.developer.nvidia.com/t/deepstream-nvinfer-server-with-https-endpoint/187958 "2021-09-30T22:28:33Z")
While using Gst-nvinferserver in deepstream pipeline, it uses c-api to communicate with triton inference server. I want to deploy a triton server with multiple models in T4 instance, and deepstream-app to communicate wit… --- ## [TRITON server performance degradation 20.03 =\> 20.07](https://forums.developer.nvidia.com/t/triton-server-performance-degradation-20-03-20-07/188058)
**Author:** [@konstantin.sobolev](https://forums.developer.nvidia.com/u/konstantin.sobolev)\ **Replies:** 1\ **Last updated:** [September 30, 2021, 10:26pm UTC](https://forums.developer.nvidia.com/t/triton-server-performance-degradation-20-03-20-07/188058 "2021-09-30T22:26:54Z")
Hi , after moving big project from triton 20.03 to triton 20.07 we see performance degradation. CPU usage increases from 500% to ~ 1000% ( twice) while stress testing. all models have ‘tensorrt\_plan’ , and system uses… --- ## [Frontend GUI for NVIDIA Triton Inference Server](https://forums.developer.nvidia.com/t/frontend-gui-for-nvidia-triton-inference-server/188062)
**Author:** [@mshafieluru](https://forums.developer.nvidia.com/u/mshafieluru)\ **Replies:** 1\ **Last updated:** [September 30, 2021, 10:26pm UTC](https://forums.developer.nvidia.com/t/frontend-gui-for-nvidia-triton-inference-server/188062 "2021-09-30T22:26:13Z")
Is there an HTTP-based frontend GUI available for NVIDIA Triton Inference Server to do some administrative tasks? Although there is HTTP port 8000 enabled. I could not see an interface. --- ## [Inference on video/audio streams in Triton](https://forums.developer.nvidia.com/t/inference-on-video-audio-streams-in-triton/189019)
**Author:** [@YarivAridor](https://forums.developer.nvidia.com/u/YarivAridor)\ **Replies:** 1\ **Last updated:** [September 30, 2021, 10:23pm UTC](https://forums.developer.nvidia.com/t/inference-on-video-audio-streams-in-triton/189019 "2021-09-30T22:23:40Z")
Where I can find info on how video/ audio streaming is done in Triton and how to deploy models in Triton which gets video/audio steams? what should be done differently than for models which get one time input tensors pe… --- ## [Triton Inference Server Inference Request Error on GPU](https://forums.developer.nvidia.com/t/triton-inference-server-inference-request-error-on-gpu/189416)
**Author:** [@skilic](https://forums.developer.nvidia.com/u/skilic)\ **Replies:** 1\ **Last updated:** [September 29, 2021, 5:30pm UTC](https://forums.developer.nvidia.com/t/triton-inference-server-inference-request-error-on-gpu/189416 "2021-09-29T17:30:05Z")
Hello, I am using Jetson Nano 4GB with Jetpack 4.6. I need to deploy my deeplab semantic segmantation model to Triton Inference Server and then send an inference request to the deployed model. Below you can see my Tri… --- ## [Concurrent Model Execution In Same Client Script](https://forums.developer.nvidia.com/t/concurrent-model-execution-in-same-client-script/189650)
**Author:** [@skilic](https://forums.developer.nvidia.com/u/skilic)\ **Replies:** 1\ **Last updated:** [September 29, 2021, 5:29pm UTC](https://forums.developer.nvidia.com/t/concurrent-model-execution-in-same-client-script/189650 "2021-09-29T17:29:13Z")
Hello, I have 4 deployed models on the Triton Inference server and have 10 images. I need to use the Concurrent Model Execution feature exactly as described. (https://github.com/triton-inference-server/server/blob/main… --- ## [ResourceExhaustedError: OOM when allocating tensor of shape \[128\]](https://forums.developer.nvidia.com/t/resourceexhaustederror-oom-when-allocating-tensor-of-shape-128/189911)
**Author:** [@rahmahtri](https://forums.developer.nvidia.com/u/rahmahtri)\ **Replies:** 1\ **Last updated:** [September 29, 2021, 5:25pm UTC](https://forums.developer.nvidia.com/t/resourceexhaustederror-oom-when-allocating-tensor-of-shape-128/189911 "2021-09-29T17:25:17Z")
I got error (ResourceExhaustedError: OOM when allocating tensor of shape \[128\]) when running this program Human-Pose-Estimation-Benchmarking-and-Action-Recognition/run\_detector.py at master · ChengeYang/Human-Pose-Estima… --- ## [Need help authoring Model configuration for Pytorch MNIST](https://forums.developer.nvidia.com/t/need-help-authoring-model-configuration-for-pytorch-mnist/187205)
**Author:** [@rajivd](https://forums.developer.nvidia.com/u/rajivd)\ **Replies:** 2\ **Last updated:** [September 29, 2021, 5:23pm UTC](https://forums.developer.nvidia.com/t/need-help-authoring-model-configuration-for-pytorch-mnist/187205 "2021-09-29T17:23:35Z")
Hi I am trying to host my first model in triton. I ran the MNIST model from examples/main.py at master · pytorch/examples (github.com) I have saved the pytorch model in torch script format. I tried following config but… --- ## [How to improve ensemble concurrent performance](https://forums.developer.nvidia.com/t/how-to-improve-ensemble-concurrent-performance/190326)
**Author:** [@krzysztof.begiedza](https://forums.developer.nvidia.com/u/krzysztof.begiedza)\ **Replies:** 1\ **Last updated:** [September 28, 2021, 2:59pm UTC](https://forums.developer.nvidia.com/t/how-to-improve-ensemble-concurrent-performance/190326 "2021-09-28T14:59:51Z")
In my scenario many services are using single Triton instance with defined ensemble model. Is there any way to tweak performance of my ensemble model? name: "pipeline" platform: "ensemble" input \[ { name: "RAW\_IM… --- ## [Latency linearly increases when increased batch size or concurrent models Tensorrt](https://forums.developer.nvidia.com/t/latency-linearly-increases-when-increased-batch-size-or-concurrent-models-tensorrt/189243)
**Author:** [@megan1](https://forums.developer.nvidia.com/u/megan1)\ **Replies:** 2\ **Last updated:** [September 24, 2021, 7:43pm UTC](https://forums.developer.nvidia.com/t/latency-linearly-increases-when-increased-batch-size-or-concurrent-models-tensorrt/189243 "2021-09-24T19:43:26Z")
Description We are running a trt model with tensorrt\_platform on triton inference server. When using the perf\_analyzer we see linear increase in latency when we increase batch size: batch size 1: 49.4 infer/sec, latenc… --- ## [Deploying Machine Translation to Triton Inference Server](https://forums.developer.nvidia.com/t/deploying-machine-translation-to-triton-inference-server/188612)
**Author:** [@Vlad.Bondarenko](https://forums.developer.nvidia.com/u/Vlad.Bondarenko)\ **Replies:** 3\ **Last updated:** [September 13, 2021, 8:36pm UTC](https://forums.developer.nvidia.com/t/deploying-machine-translation-to-triton-inference-server/188612 "2021-09-13T20:36:09Z")
Hi, I am trying to deploy a Nemo Machine Translation model to triton inference server. In my understanding the pipeline looks something like this: Raw Text -\> Preprocessing -\> Encoder -\> Decoder -\> Probability for nex… --- ## [Kaldi triton inference server not working on nvidia k80](https://forums.developer.nvidia.com/t/kaldi-triton-inference-server-not-working-on-nvidia-k80/185748)
**Author:** [@muaaz](https://forums.developer.nvidia.com/u/muaaz)\ **Replies:** 0\ **Last updated:** [August 5, 2021, 12:41pm UTC](https://forums.developer.nvidia.com/t/kaldi-triton-inference-server-not-working-on-nvidia-k80/185748 "2021-08-05T12:41:24Z")
It said that nvidia k80 not supported by container --- ## [Optimal Trt inference using threads/processes for peoplenet model for](https://forums.developer.nvidia.com/t/optimal-trt-inference-using-threads-processes-for-peoplenet-model-for/185184)
**Author:** [@kai5gabriel](https://forums.developer.nvidia.com/u/kai5gabriel)\ **Replies:** 1\ **Last updated:** [July 30, 2021, 10:07am UTC](https://forums.developer.nvidia.com/t/optimal-trt-inference-using-threads-processes-for-peoplenet-model-for/185184 "2021-07-30T10:07:54Z")
Observation:Same inference speed when running two parallel processes vs running it as threads inside a process Environment TensorRT Version: 7.2.3.4 GPU Type: Geforce MX130 (2GB) Nvidia Driver Version: 465.19.01 CUDA… --- ## [OpenCV import fails with "Illegal Instruction (Core Dump)"](https://forums.developer.nvidia.com/t/opencv-import-fails-with-illegal-instruction-core-dump/168060)
**Author:** [@liwaa.alnashi](https://forums.developer.nvidia.com/u/liwaa.alnashi)\ **Replies:** 1\ **Last updated:** [July 29, 2021, 12:48pm UTC](https://forums.developer.nvidia.com/t/opencv-import-fails-with-illegal-instruction-core-dump/168060 "2021-07-29T12:48:09Z")
Hi I want to install (using pip) openCV inside a virtual environment (venv) on a Jetson Xavier NX development board. When running import cv2 command I get \>\> Illegal Instruction (Core Dump) on command line? Do I rea… --- ## [Detection is not happening continously in tesla t4 but with the same model the continous detection is happening in nvidia xavier AGX](https://forums.developer.nvidia.com/t/detection-is-not-happening-continously-in-tesla-t4-but-with-the-same-model-the-continous-detection-is-happening-in-nvidia-xavier-agx/184713)
**Author:** [@ranjith.kumar](https://forums.developer.nvidia.com/u/ranjith.kumar)\ **Replies:** 0\ **Last updated:** [July 26, 2021, 9:08am UTC](https://forums.developer.nvidia.com/t/detection-is-not-happening-continously-in-tesla-t4-but-with-the-same-model-the-continous-detection-is-happening-in-nvidia-xavier-agx/184713 "2021-07-26T09:08:46Z")
primary\_inference.txt (1.5 KB) stream.txt (2.2 KB) --- ## [ERROR: ../nvdsinfer/nvdsinfer\_model\_builder.cpp:862 Failed to get cuda engine from custom library API](https://forums.developer.nvidia.com/t/error-nvdsinfer-nvdsinfer-model-builder-cpp-862-failed-to-get-cuda-engine-from-custom-library-api/184711)
**Author:** [@ranjith.kumar](https://forums.developer.nvidia.com/u/ranjith.kumar)\ **Replies:** 0\ **Last updated:** [July 26, 2021, 9:01am UTC](https://forums.developer.nvidia.com/t/error-nvdsinfer-nvdsinfer-model-builder-cpp-862-failed-to-get-cuda-engine-from-custom-library-api/184711 "2021-07-26T09:01:54Z")
while testing the trained mrcnn model the following issue are facing ERROR: …/nvdsinfer/nvdsinfer\_model\_builder.cpp:862 Failed to get cuda engine from custom library API. For full logs: libEGL warning: DRI2: failed to … --- ## [Deploy LPRnet sequence Classifier on Triton Inference Server](https://forums.developer.nvidia.com/t/deploy-lprnet-sequence-classifier-on-triton-inference-server/184293)
**Author:** [@Pritam](https://forums.developer.nvidia.com/u/Pritam)\ **Replies:** 0\ **Last updated:** [July 21, 2021, 9:56am UTC](https://forums.developer.nvidia.com/t/deploy-lprnet-sequence-classifier-on-triton-inference-server/184293 "2021-07-21T09:56:44Z")
Hi, I need some clarity in LPRnet model deployment on Triton Inference Server. Does Triton Inference Server support LPRnet model ? if then how to deploy it on Triton ? --- ## [Opencv error and Gstreamer can't be created](https://forums.developer.nvidia.com/t/opencv-error-and-gstreamer-cant-be-created/163514)
**Author:** [@zahrataleb](https://forums.developer.nvidia.com/u/zahrataleb)\ **Replies:** 1\ **Last updated:** [July 17, 2021, 9:29am UTC](https://forums.developer.nvidia.com/t/opencv-error-and-gstreamer-cant-be-created/163514 "2021-07-17T09:29:35Z")
Hi guys, I am trying to run a video and take license plates from video but I got this error. Any advise would be appreciated! that’s the code: cap = cv2.VideoCapture(‘filesrc location = “IMG\_0580.mp4” ! video/x-raw,f… --- ## [How to use dynamic shapes in TensorRT](https://forums.developer.nvidia.com/t/how-to-use-dynamic-shapes-in-tensorrt/183370)
**Author:** [@1260178344](https://forums.developer.nvidia.com/u/1260178344)\ **Replies:** 0\ **Last updated:** [July 12, 2021, 5:50am UTC](https://forums.developer.nvidia.com/t/how-to-use-dynamic-shapes-in-tensorrt/183370 "2021-07-12T05:50:47Z")
I want to use dynamic shapes in TensorRT, but there is a fully connected layer in my network, and a resizing operation must be performed after entering the network. Then set dynamic batch and dynamic input (nchw) when a… --- ## [Real-time Car/Pedestrian detection models for ADAS(Advance Driver Assistance Systems )](https://forums.developer.nvidia.com/t/real-time-car-pedestrian-detection-models-for-adas-advance-driver-assistance-systems/183296)
**Author:** [@kheirkhah75](https://forums.developer.nvidia.com/u/kheirkhah75)\ **Replies:** 0\ **Last updated:** [July 10, 2021, 2:15pm UTC](https://forums.developer.nvidia.com/t/real-time-car-pedestrian-detection-models-for-adas-advance-driver-assistance-systems/183296 "2021-07-10T14:15:37Z")
Hello everyone, I am trying to develop and ADAS(Advance Driver Assistance Systems ) that detects cars and pedestrians for the purpose of issuing warnings for collisions. What are your recommendations for deep detection… --- ## [What's the proper way to preprocess on Triton with streaming?](https://forums.developer.nvidia.com/t/whats-the-proper-way-to-preprocess-on-triton-with-streaming/182853)
**Author:** [@ming\_wu](https://forums.developer.nvidia.com/u/ming_wu)\ **Replies:** 0\ **Last updated:** [July 6, 2021, 3:22am UTC](https://forums.developer.nvidia.com/t/whats-the-proper-way-to-preprocess-on-triton-with-streaming/182853 "2021-07-06T03:22:20Z")
I have a car flow video that need to detect license plate. Client decode video by OpenCV and send raw BGR data to Triton. I’ve write a simple preprocessing on Triton with OpenCV(Resize and Normalize). This way works b… --- ## [Inconsistant GPU memory utilsation with parallel model instances](https://forums.developer.nvidia.com/t/inconsistant-gpu-memory-utilsation-with-parallel-model-instances/182767)
**Author:** [@sanchit.prakash](https://forums.developer.nvidia.com/u/sanchit.prakash)\ **Replies:** 0\ **Last updated:** [July 5, 2021, 6:59am UTC](https://forums.developer.nvidia.com/t/inconsistant-gpu-memory-utilsation-with-parallel-model-instances/182767 "2021-07-05T06:59:48Z")
Triton Server Version: 2.7.0 Triton Docker image: nvcr.io/nvidia/tritonserver:21.02-py3 GPU used: 1 Tesla T4 16GB TensorFlow version: 2 Experiment objective: Try to load as many parallel instances of a TF model (.pb … [Next page](https://forums.developer.nvidia.com/c/ai-data-science/deep-learning/triton-inference-server/97.md?page=1)