I0105 08:43:08.489437 621925 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f156c000000' with size 67108864 I0105 08:43:08.489911 621925 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864 I0105 08:43:08.489933 621925 cuda_memory_manager.cc:105] CUDA memory pool is created on device 1 with size 67108864 I0105 08:43:08.559571 621925 server.cc:559] +------------------+------+ | Repository Agent | Path | +------------------+------+ +------------------+------+ I0105 08:43:08.559620 621925 server.cc:586] +---------+------+--------+ | Backend | Path | Config | +---------+------+--------+ +---------+------+--------+ I0105 08:43:08.559629 621925 model_repository_manager.cc:704] ModelStates() I0105 08:43:08.559643 621925 server.cc:629] +-------+---------+--------+ | Model | Version | Status | +-------+---------+--------+ +-------+---------+--------+ I0105 08:43:08.593450 621925 metrics.cc:650] Collecting metrics for GPU 0: NVIDIA RTX A4000 I0105 08:43:08.593495 621925 metrics.cc:650] Collecting metrics for GPU 1: NVIDIA RTX A4000 I0105 08:43:08.593763 621925 tritonserver.cc:2176] +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.24.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_shared_memory binary_tensor_data statistics trace | | model_repository_path[0] | /opt/nvidia/deepstream/deepstream-6.1/sources/project/topdownnet-python-triton/triton_model_repo | | model_control_mode | MODE_EXPLICIT | | strict_model_config | 1 | | rate_limit | OFF | | pinned_memory_pool_byte_size | 67108864 | | cuda_memory_pool_byte_size{0} | 67108864 | | cuda_memory_pool_byte_size{1} | 67108864 | | response_cache_byte_size | 0 | | min_supported_compute_capability | 6.0 | | strict_readiness | 1 | | exit_timeout | 30 | +----------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ I0105 08:43:08.593797 621925 model_repository_manager.cc:704] ModelStates() I0105 08:43:08.593829 621925 model_repository_manager.cc:773] GetModel() 'ensemble_python_smoke_16' version -1 I0105 08:43:08.594370 621925 model_config_utils.cc:645] Server side auto-completed config: name: "ensemble_python_smoke_16" platform: "ensemble" input { name: "INPUT" data_type: TYPE_UINT8 dims: 512 dims: 512 dims: 3 } output { name: "OUTPUT" data_type: TYPE_FP16 dims: 1 dims: 256 dims: 256 dims: 1 } ensemble_scheduling { step { model_name: "preprocess_16" model_version: 1 input_map { key: "INPUT" value: "INPUT" } output_map { key: "Image" value: "preprocessed_image" } output_map { key: "InitVector" value: "InitVector" } } step { model_name: "smoke_16" model_version: 1 input_map { key: "Image" value: "preprocessed_image" } input_map { key: "InitVector" value: "InitVector" } output_map { key: "detection_machine_2" value: "detection_machine_2" } } step { model_name: "postprocess_16" model_version: 1 input_map { key: "detection_machine_2" value: "detection_machine_2" } output_map { key: "OUTPUT" value: "OUTPUT" } } } I0105 08:43:08.594523 621925 model_config_utils.cc:645] Server side auto-completed config: name: "postprocess_16" input { name: "detection_machine_2" data_type: TYPE_FP16 dims: 1 dims: 256 dims: 256 dims: 1 } output { name: "OUTPUT" data_type: TYPE_FP16 dims: 1 dims: 256 dims: 256 dims: 1 } instance_group { count: 1 gpus: 0 kind: KIND_GPU } default_model_filename: "model.py" sequence_batching { state { } } backend: "python" I0105 08:43:08.594918 621925 model_config_utils.cc:645] Server side auto-completed config: name: "preprocess_16" input { name: "INPUT" data_type: TYPE_UINT8 dims: 512 dims: 512 dims: 3 } output { name: "Image" data_type: TYPE_FP16 dims: 1 dims: 512 dims: 512 dims: 6 } output { name: "InitVector" data_type: TYPE_FP16 dims: 1 dims: 1 dims: 1 dims: 180 } instance_group { count: 1 gpus: 0 kind: KIND_GPU } default_model_filename: "model.py" backend: "python" I0105 08:43:08.595341 621925 model_config_utils.cc:645] Server side auto-completed config: name: "smoke_16" platform: "tensorrt_plan" input { name: "Image" data_type: TYPE_FP16 dims: 1 dims: 512 dims: 512 dims: 6 } input { name: "InitVector" data_type: TYPE_FP16 dims: 1 dims: 1 dims: 1 dims: 180 } output { name: "detection_machine_2" data_type: TYPE_FP16 dims: 1 dims: 256 dims: 256 dims: 1 } instance_group { count: 1 gpus: 0 kind: KIND_GPU } default_model_filename: "model_smoke_16.onnx.engine" sequence_batching { state { input_name: "PreviousState" output_name: "leaky_re_lu_47" data_type: TYPE_FP16 dims: 1 dims: 128 dims: 128 dims: 180 initial_state { data_type: TYPE_FP16 dims: 1 dims: 128 dims: 128 dims: 180 zero_data: true name: "InitVector" } } } backend: "tensorrt" I0105 08:43:08.595692 621925 model_repository_manager.cc:913] AsyncLoad() 'smoke_16' I0105 08:43:08.595728 621925 model_repository_manager.cc:1151] TriggerNextAction() 'smoke_16' version 1: 1 I0105 08:43:08.595745 621925 model_repository_manager.cc:1187] Load() 'smoke_16' version 1 I0105 08:43:08.595755 621925 model_repository_manager.cc:1206] loading: smoke_16:1 I0105 08:43:08.595771 621925 model_repository_manager.cc:913] AsyncLoad() 'preprocess_16' I0105 08:43:08.595797 621925 model_repository_manager.cc:1151] TriggerNextAction() 'preprocess_16' version 1: 1 I0105 08:43:08.595811 621925 model_repository_manager.cc:1187] Load() 'preprocess_16' version 1 I0105 08:43:08.595821 621925 model_repository_manager.cc:1206] loading: preprocess_16:1 I0105 08:43:08.595795 621925 model_repository_manager.cc:1256] CreateModel() 'smoke_16' version 1 I0105 08:43:08.595867 621925 model_repository_manager.cc:1256] CreateModel() 'preprocess_16' version 1 I0105 08:43:08.595839 621925 model_repository_manager.cc:913] AsyncLoad() 'postprocess_16' I0105 08:43:08.595921 621925 backend_model.cc:292] Adding default backend config setting: default-max-batch-size,4 I0105 08:43:08.595933 621925 model_repository_manager.cc:1151] TriggerNextAction() 'postprocess_16' version 1: 1 I0105 08:43:08.595958 621925 model_repository_manager.cc:1187] Load() 'postprocess_16' version 1 I0105 08:43:08.595974 621925 model_repository_manager.cc:1206] loading: postprocess_16:1 I0105 08:43:08.595935 621925 backend_model.cc:292] Adding default backend config setting: default-max-batch-size,4 I0105 08:43:08.595950 621925 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/tensorrt/libtriton_tensorrt.so I0105 08:43:08.596011 621925 model_repository_manager.cc:1256] CreateModel() 'postprocess_16' version 1 I0105 08:43:08.596071 621925 backend_model.cc:292] Adding default backend config setting: default-max-batch-size,4 I0105 08:43:08.601269 621925 tensorrt.cc:5427] TRITONBACKEND_Initialize: tensorrt I0105 08:43:08.601302 621925 tensorrt.cc:5437] Triton TRITONBACKEND API version: 1.10 I0105 08:43:08.601315 621925 tensorrt.cc:5443] 'tensorrt' TRITONBACKEND API version: 1.10 I0105 08:43:08.601326 621925 tensorrt.cc:5466] Registering TensorRT Plugins I0105 08:43:08.601350 621925 logging.cc:52] Registered plugin creator - ::BatchedNMSDynamic_TRT version 1 I0105 08:43:08.601365 621925 logging.cc:52] Registered plugin creator - ::BatchedNMS_TRT version 1 I0105 08:43:08.601378 621925 logging.cc:52] Registered plugin creator - ::BatchTilePlugin_TRT version 1 I0105 08:43:08.601391 621925 logging.cc:52] Registered plugin creator - ::Clip_TRT version 1 I0105 08:43:08.601403 621925 logging.cc:52] Registered plugin creator - ::CoordConvAC version 1 I0105 08:43:08.601416 621925 logging.cc:52] Registered plugin creator - ::CropAndResizeDynamic version 1 I0105 08:43:08.601429 621925 logging.cc:52] Registered plugin creator - ::CropAndResize version 1 I0105 08:43:08.601442 621925 logging.cc:52] Registered plugin creator - ::DecodeBbox3DPlugin version 1 I0105 08:43:08.601454 621925 logging.cc:52] Registered plugin creator - ::DetectionLayer_TRT version 1 I0105 08:43:08.601466 621925 logging.cc:52] Registered plugin creator - ::EfficientNMS_Explicit_TF_TRT version 1 I0105 08:43:08.601481 621925 logging.cc:52] Registered plugin creator - ::EfficientNMS_Implicit_TF_TRT version 1 I0105 08:43:08.601494 621925 logging.cc:52] Registered plugin creator - ::EfficientNMS_ONNX_TRT version 1 I0105 08:43:08.601507 621925 logging.cc:52] Registered plugin creator - ::EfficientNMS_TRT version 1 I0105 08:43:08.601521 621925 logging.cc:52] Registered plugin creator - ::FlattenConcat_TRT version 1 I0105 08:43:08.601533 621925 logging.cc:52] Registered plugin creator - ::fMHA_V2 version 1 I0105 08:43:08.601546 621925 logging.cc:52] Registered plugin creator - ::fMHCA version 1 I0105 08:43:08.601562 621925 logging.cc:52] Registered plugin creator - ::GenerateDetection_TRT version 1 I0105 08:43:08.601575 621925 logging.cc:52] Registered plugin creator - ::GridAnchor_TRT version 1 I0105 08:43:08.601587 621925 logging.cc:52] Registered plugin creator - ::GridAnchorRect_TRT version 1 I0105 08:43:08.601600 621925 logging.cc:52] Registered plugin creator - ::GroupNorm version 1 I0105 08:43:08.601613 621925 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 1 I0105 08:43:08.601626 621925 logging.cc:52] Registered plugin creator - ::InstanceNormalization_TRT version 2 I0105 08:43:08.601640 621925 logging.cc:52] Registered plugin creator - ::LayerNorm version 1 I0105 08:43:08.601653 621925 logging.cc:52] Registered plugin creator - ::LReLU_TRT version 1 I0105 08:43:08.601665 621925 logging.cc:52] Registered plugin creator - ::MultilevelCropAndResize_TRT version 1 I0105 08:43:08.601678 621925 logging.cc:52] Registered plugin creator - ::MultilevelProposeROI_TRT version 1 I0105 08:43:08.601691 621925 logging.cc:52] Registered plugin creator - ::MultiscaleDeformableAttnPlugin_TRT version 1 I0105 08:43:08.601704 621925 logging.cc:52] Registered plugin creator - ::NMSDynamic_TRT version 1 I0105 08:43:08.601716 621925 logging.cc:52] Registered plugin creator - ::NMS_TRT version 1 I0105 08:43:08.601729 621925 logging.cc:52] Registered plugin creator - ::Normalize_TRT version 1 I0105 08:43:08.601741 621925 logging.cc:52] Registered plugin creator - ::PillarScatterPlugin version 1 I0105 08:43:08.601757 621925 logging.cc:52] Registered plugin creator - ::PriorBox_TRT version 1 I0105 08:43:08.601770 621925 logging.cc:52] Registered plugin creator - ::ProposalDynamic version 1 I0105 08:43:08.601782 621925 logging.cc:52] Registered plugin creator - ::ProposalLayer_TRT version 1 I0105 08:43:08.601794 621925 logging.cc:52] Registered plugin creator - ::Proposal version 1 I0105 08:43:08.601807 621925 logging.cc:52] Registered plugin creator - ::PyramidROIAlign_TRT version 1 I0105 08:43:08.601820 621925 logging.cc:52] Registered plugin creator - ::Region_TRT version 1 I0105 08:43:08.601834 621925 logging.cc:52] Registered plugin creator - ::Reorg_TRT version 1 I0105 08:43:08.601847 621925 logging.cc:52] Registered plugin creator - ::ResizeNearest_TRT version 1 I0105 08:43:08.601859 621925 logging.cc:52] Registered plugin creator - ::ROIAlign_TRT version 1 I0105 08:43:08.601872 621925 logging.cc:52] Registered plugin creator - ::RPROI_TRT version 1 I0105 08:43:08.601884 621925 logging.cc:52] Registered plugin creator - ::ScatterND version 1 I0105 08:43:08.601904 621925 logging.cc:52] Registered plugin creator - ::SeqLen2Spatial version 1 I0105 08:43:08.601919 621925 logging.cc:52] Registered plugin creator - ::SpecialSlice_TRT version 1 I0105 08:43:08.601935 621925 logging.cc:52] Registered plugin creator - ::SplitGeLU version 1 I0105 08:43:08.601949 621925 logging.cc:52] Registered plugin creator - ::Split version 1 I0105 08:43:08.601966 621925 logging.cc:52] Registered plugin creator - ::VoxelGeneratorPlugin version 1 I0105 08:43:08.601980 621925 tensorrt.cc:5486] backend configuration: {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} I0105 08:43:08.602013 621925 tensorrt.cc:5538] TRITONBACKEND_ModelInitialize: smoke_16 (version 1) I0105 08:43:08.602719 621925 model_config_utils.cc:1656] ModelConfig 64-bit fields: I0105 08:43:08.602739 621925 model_config_utils.cc:1658] ModelConfig::dynamic_batching::default_queue_policy::default_timeout_microseconds I0105 08:43:08.602749 621925 model_config_utils.cc:1658] ModelConfig::dynamic_batching::max_queue_delay_microseconds I0105 08:43:08.602769 621925 model_config_utils.cc:1658] ModelConfig::dynamic_batching::priority_queue_policy::value::default_timeout_microseconds I0105 08:43:08.602783 621925 model_config_utils.cc:1658] ModelConfig::ensemble_scheduling::step::model_version I0105 08:43:08.602796 621925 model_config_utils.cc:1658] ModelConfig::input::dims I0105 08:43:08.602814 621925 model_config_utils.cc:1658] ModelConfig::input::reshape::shape I0105 08:43:08.602830 621925 model_config_utils.cc:1658] ModelConfig::instance_group::secondary_devices::device_id I0105 08:43:08.602842 621925 model_config_utils.cc:1658] ModelConfig::model_warmup::inputs::value::dims I0105 08:43:08.602857 621925 model_config_utils.cc:1658] ModelConfig::optimization::cuda::graph_spec::graph_lower_bound::input::value::dim I0105 08:43:08.602871 621925 model_config_utils.cc:1658] ModelConfig::optimization::cuda::graph_spec::input::value::dim I0105 08:43:08.602886 621925 model_config_utils.cc:1658] ModelConfig::output::dims I0105 08:43:08.602900 621925 model_config_utils.cc:1658] ModelConfig::output::reshape::shape I0105 08:43:08.602915 621925 model_config_utils.cc:1658] ModelConfig::sequence_batching::direct::max_queue_delay_microseconds I0105 08:43:08.602929 621925 model_config_utils.cc:1658] ModelConfig::sequence_batching::max_sequence_idle_microseconds I0105 08:43:08.602944 621925 model_config_utils.cc:1658] ModelConfig::sequence_batching::oldest::max_queue_delay_microseconds I0105 08:43:08.602958 621925 model_config_utils.cc:1658] ModelConfig::sequence_batching::state::dims I0105 08:43:08.602972 621925 model_config_utils.cc:1658] ModelConfig::sequence_batching::state::initial_state::dims I0105 08:43:08.602992 621925 model_config_utils.cc:1658] ModelConfig::version_policy::specific::versions I0105 08:43:08.603106 621925 tensorrt.cc:442] model configuration: { "name": "smoke_16", "platform": "tensorrt_plan", "backend": "tensorrt", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 0, "input": [ { "name": "Image", "data_type": "TYPE_FP16", "format": "FORMAT_NONE", "dims": [ 1, 512, 512, 6 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false }, { "name": "InitVector", "data_type": "TYPE_FP16", "format": "FORMAT_NONE", "dims": [ 1, 1, 1, 180 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "detection_machine_2", "data_type": "TYPE_FP16", "dims": [ 1, 256, 256, 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 1000000, "control_input": [], "state": [ { "input_name": "PreviousState", "output_name": "leaky_re_lu_47", "data_type": "TYPE_FP16", "dims": [ 1, 128, 128, 180 ], "initial_state": [ { "data_type": "TYPE_FP16", "dims": [ "1", "128", "128", "180" ], "name": "InitVector", "zero_data": true } ] } ] }, "instance_group": [ { "name": "smoke_16_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "model_smoke_16.onnx.engine", "cc_model_filenames": {}, "metric_tags": {}, "parameters": {}, "model_warmup": [] } I0105 08:43:08.603132 621925 shared_library.cc:108] OpenLibraryHandle: /opt/tritonserver/backends/python/libtriton_python.so I0105 08:43:08.603296 621925 backend_model.cc:179] Overriding execution policy to "TRITONBACKEND_EXECUTION_BLOCKING" for sequence model "smoke_16" I0105 08:43:08.604697 621925 python_be.cc:1523] 'python' TRITONBACKEND API version: 1.10 I0105 08:43:08.604723 621925 python_be.cc:1545] backend configuration: {"cmdline":{"auto-complete-config":"false","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} I0105 08:43:08.604747 621925 python_be.cc:1675] Shared memory configuration is shm-default-byte-size=67108864,shm-growth-byte-size=67108864,stub-timeout-seconds=30 I0105 08:43:08.604867 621925 tensorrt.cc:5587] TRITONBACKEND_ModelInstanceInitialize: smoke_16_0 (GPU device 0) I0105 08:43:08.605245 621925 backend_model_instance.cc:105] Creating instance smoke_16_0 on GPU 0 (8.6) using artifact 'model_smoke_16.onnx.engine' I0105 08:43:08.639146 621925 tensorrt.cc:1615] Zero copy optimization is disabled I0105 08:43:08.938588 621925 logging.cc:49] Loaded engine size: 15 MiB I0105 08:43:09.045309 621925 logging.cc:52] Trying to load shared library libcudnn.so.8 I0105 08:43:09.045431 621925 logging.cc:52] Loaded shared library libcudnn.so.8 I0105 08:43:09.045465 621925 logging.cc:52] Using cuDNN as plugin tactic source I0105 08:43:09.881494 621925 logging.cc:52] Using cuDNN as core library tactic source I0105 08:43:09.881717 621925 logging.cc:49] [MemUsageChange] Init cuDNN: CPU +970, GPU +414, now: CPU 1865, GPU 812 (MiB) W0105 08:43:09.881796 621925 logging.cc:46] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.4.1 I0105 08:43:09.884244 621925 logging.cc:52] Deserialization required 871438 microseconds. I0105 08:43:09.884399 621925 logging.cc:49] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +6, now: CPU 0, GPU 6 (MiB) I0105 08:43:09.885935 621925 tensorrt.cc:390] Created new runtime on GPU device 0, NVDLA core -1 for smoke_16 I0105 08:43:09.885960 621925 tensorrt.cc:397] Created new engine on GPU device 0, NVDLA core -1 for smoke_16 I0105 08:43:09.886459 621925 logging.cc:52] Trying to load shared library libcudnn.so.8 I0105 08:43:09.886504 621925 logging.cc:52] Loaded shared library libcudnn.so.8 I0105 08:43:09.886535 621925 logging.cc:52] Using cuDNN as plugin tactic source I0105 08:43:09.887526 621925 logging.cc:52] Using cuDNN as core library tactic source I0105 08:43:09.887684 621925 logging.cc:49] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 1834, GPU 814 (MiB) W0105 08:43:09.887704 621925 logging.cc:46] TensorRT was linked against cuDNN 8.6.0 but loaded cuDNN 8.4.1 I0105 08:43:09.891290 621925 logging.cc:52] Total per-runner device persistent memory is 2926592 I0105 08:43:09.891316 621925 logging.cc:52] Total per-runner host persistent memory is 445280 I0105 08:43:09.891636 621925 logging.cc:52] Allocated activation device memory of size 165095936 I0105 08:43:09.920688 621925 logging.cc:49] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +160, now: CPU 0, GPU 166 (MiB) W0105 08:43:09.920740 621925 logging.cc:46] CUDA lazy loading is not enabled. Enabling it can significantly reduce device memory usage. See `CUDA_MODULE_LOADING` in https://docs.nvidia.com/cuda/cuda-c-programming-guide/index.html#env-vars I0105 08:43:09.920772 621925 tensorrt.cc:3308] Detected Image as execution binding for smoke_16_0 I0105 08:43:09.920786 621925 tensorrt.cc:3308] Detected InitVector as execution binding for smoke_16_0 I0105 08:43:09.920796 621925 tensorrt.cc:3308] Detected PreviousState as execution binding for smoke_16_0 I0105 08:43:09.920815 621925 tensorrt.cc:3308] Detected leaky_re_lu_47 as execution binding for smoke_16_0 I0105 08:43:09.920826 621925 tensorrt.cc:3308] Detected detection_machine_2 as execution binding for smoke_16_0 I0105 08:43:09.921562 621925 tensorrt.cc:1541] Created instance smoke_16_0 on GPU 0 with stream priority 0 and optimization profile default[0]; I0105 08:43:09.921615 621925 python_be.cc:1723] TRITONBACKEND_ModelInitialize: preprocess_16 (version 1) I0105 08:43:09.921672 621925 backend_model_instance.cc:735] Starting backend thread for smoke_16_0 at nice 0 on device 0... I0105 08:43:09.921737 621925 pinned_memory_manager.cc:161] pinned memory allocation: size 5898240, addr 0x7f156c000090 I0105 08:43:09.922004 621925 python_be.cc:1503] model configuration: { "name": "preprocess_16", "platform": "", "backend": "python", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 0, "input": [ { "name": "INPUT", "data_type": "TYPE_UINT8", "format": "FORMAT_NONE", "dims": [ 512, 512, 3 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "Image", "data_type": "TYPE_FP16", "dims": [ 1, 512, 512, 6 ], "label_filename": "", "is_shape_tensor": false }, { "name": "InitVector", "data_type": "TYPE_FP16", "dims": [ 1, 1, 1, 180 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "instance_group": [ { "name": "preprocess_16_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "model.py", "cc_model_filenames": {}, "metric_tags": {}, "parameters": {}, "model_warmup": [] } I0105 08:43:09.922040 621925 python_be.cc:1723] TRITONBACKEND_ModelInitialize: postprocess_16 (version 1) I0105 08:43:09.922106 621925 sequence_batch_scheduler.cc:1183] Starting Direct sequence-batch scheduler thread 0 at nice 0... I0105 08:43:09.922125 621925 model_repository_manager.cc:1352] successfully loaded 'smoke_16' version 1 I0105 08:43:09.922144 621925 sequence_batch_scheduler.cc:782] Starting sequence-batch reaper thread at nice 10... I0105 08:43:09.922154 621925 model_repository_manager.cc:1151] TriggerNextAction() 'smoke_16' version 1: 0 I0105 08:43:09.922187 621925 sequence_batch_scheduler.cc:876] Reaper: sleeping for 1000000us... I0105 08:43:09.922203 621925 model_repository_manager.cc:1165] no next action, trigger OnComplete() I0105 08:43:09.922255 621925 model_repository_manager.cc:728] VersionStates() 'smoke_16' I0105 08:43:09.922502 621925 python_be.cc:1503] model configuration: { "name": "postprocess_16", "platform": "", "backend": "python", "version_policy": { "latest": { "num_versions": 1 } }, "max_batch_size": 0, "input": [ { "name": "detection_machine_2", "data_type": "TYPE_FP16", "format": "FORMAT_NONE", "dims": [ 1, 256, 256, 1 ], "is_shape_tensor": false, "allow_ragged_batch": false, "optional": false } ], "output": [ { "name": "OUTPUT", "data_type": "TYPE_FP16", "dims": [ 1, 256, 256, 1 ], "label_filename": "", "is_shape_tensor": false } ], "batch_input": [], "batch_output": [], "optimization": { "priority": "PRIORITY_DEFAULT", "input_pinned_memory": { "enable": true }, "output_pinned_memory": { "enable": true }, "gather_kernel_buffer_threshold": 0, "eager_batching": false }, "sequence_batching": { "max_sequence_idle_microseconds": 1000000, "control_input": [], "state": [ { "input_name": "", "output_name": "", "data_type": "TYPE_INVALID", "dims": [], "initial_state": [] } ] }, "instance_group": [ { "name": "postprocess_16_0", "kind": "KIND_GPU", "count": 1, "gpus": [ 0 ], "secondary_devices": [], "profile": [], "passive": false, "host_policy": "" } ], "default_model_filename": "model.py", "cc_model_filenames": {}, "metric_tags": {}, "parameters": {}, "model_warmup": [] } I0105 08:43:09.923263 621925 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: preprocess_16_0 (GPU device 0) I0105 08:43:09.923368 621925 backend_model_instance.cc:105] Creating instance preprocess_16_0 on GPU 0 (8.6) using artifact 'model.py' I0105 08:43:09.990807 622048 stub_launcher.cc:253] Starting Python backend stub: exec /opt/tritonserver/backends/python/triton_python_backend_stub /opt/nvidia/deepstream/deepstream-6.1/sources/project/topdownnet-python-triton/triton_model_repo/preprocess_16/1/model.py triton_python_backend_shm_region_1 67108864 67108864 621925 /opt/tritonserver/backends/python 336 preprocess_16_0 I0105 08:43:10.922301 621925 sequence_batch_scheduler.cc:876] Reaper: sleeping for 1000000us... I0105 08:43:11.003361 621925 python_be.cc:1788] TRITONBACKEND_ModelInstanceInitialize: instance initialization successful preprocess_16_0 (device 0) I0105 08:43:11.003451 621925 python_be.cc:1767] TRITONBACKEND_ModelInstanceInitialize: postprocess_16_0 (GPU device 0) I0105 08:43:11.003476 621925 backend_model_instance.cc:735] Starting backend thread for preprocess_16_0 at nice 0 on device 0... I0105 08:43:11.003562 621925 model_repository_manager.cc:1352] successfully loaded 'preprocess_16' version 1 I0105 08:43:11.003584 621925 model_repository_manager.cc:1151] TriggerNextAction() 'preprocess_16' version 1: 0 I0105 08:43:11.003604 621925 model_repository_manager.cc:1165] no next action, trigger OnComplete() I0105 08:43:11.003642 621925 model_repository_manager.cc:728] VersionStates() 'preprocess_16' I0105 08:43:11.003663 621925 backend_model_instance.cc:105] Creating instance postprocess_16_0 on GPU 0 (8.6) using artifact 'model.py' I0105 08:43:11.037267 622124 stub_launcher.cc:253] Starting Python backend stub: exec /opt/tritonserver/backends/python/triton_python_backend_stub /opt/nvidia/deepstream/deepstream-6.1/sources/project/topdownnet-python-triton/triton_model_repo/postprocess_16/1/model.py triton_python_backend_shm_region_2 67108864 67108864 621925 /opt/tritonserver/backends/python 336 postprocess_16_0 I0105 08:43:11.922438 621925 sequence_batch_scheduler.cc:876] Reaper: sleeping for 1000000us... I0105 08:43:12.000808 621925 python_be.cc:1788] TRITONBACKEND_ModelInstanceInitialize: instance initialization successful postprocess_16_0 (device 0) I0105 08:43:12.000920 621925 backend_model_instance.cc:735] Starting backend thread for postprocess_16_0 at nice 0 on device 0... I0105 08:43:12.001031 621925 sequence_batch_scheduler.cc:1183] Starting Direct sequence-batch scheduler thread 0 at nice 0... I0105 08:43:12.001050 621925 model_repository_manager.cc:1352] successfully loaded 'postprocess_16' version 1 I0105 08:43:12.001058 621925 sequence_batch_scheduler.cc:782] Starting sequence-batch reaper thread at nice 10... I0105 08:43:12.001099 621925 model_repository_manager.cc:1151] TriggerNextAction() 'postprocess_16' version 1: 0 I0105 08:43:12.001127 621925 model_repository_manager.cc:1165] no next action, trigger OnComplete() I0105 08:43:12.001115 621925 sequence_batch_scheduler.cc:876] Reaper: sleeping for 1000000us... I0105 08:43:12.001168 621925 model_repository_manager.cc:728] VersionStates() 'postprocess_16' I0105 08:43:12.001205 621925 model_repository_manager.cc:773] GetModel() 'preprocess_16' version -1 I0105 08:43:12.001244 621925 model_repository_manager.cc:773] GetModel() 'smoke_16' version -1 I0105 08:43:12.001273 621925 model_repository_manager.cc:773] GetModel() 'postprocess_16' version -1 I0105 08:43:12.001295 621925 model_repository_manager.cc:773] GetModel() 'preprocess_16' version -1 I0105 08:43:12.001312 621925 model_repository_manager.cc:773] GetModel() 'smoke_16' version -1 I0105 08:43:12.001332 621925 model_repository_manager.cc:773] GetModel() 'postprocess_16' version -1 I0105 08:43:12.001350 621925 model_repository_manager.cc:773] GetModel() 'preprocess_16' version -1 I0105 08:43:12.001369 621925 model_repository_manager.cc:773] GetModel() 'smoke_16' version -1 I0105 08:43:12.001384 621925 model_repository_manager.cc:773] GetModel() 'postprocess_16' version -1 I0105 08:43:12.001411 621925 model_repository_manager.cc:913] AsyncLoad() 'ensemble_python_smoke_16' I0105 08:43:12.001464 621925 model_repository_manager.cc:1151] TriggerNextAction() 'ensemble_python_smoke_16' version 1: 1 I0105 08:43:12.001479 621925 model_repository_manager.cc:1187] Load() 'ensemble_python_smoke_16' version 1 I0105 08:43:12.001488 621925 model_repository_manager.cc:1206] loading: ensemble_python_smoke_16:1 I0105 08:43:12.001515 621925 model_repository_manager.cc:1256] CreateModel() 'ensemble_python_smoke_16' version 1 I0105 08:43:12.001644 621925 ensemble_model.cc:54] ensemble model for ensemble_python_smoke_16 I0105 08:43:12.001663 621925 model_repository_manager.cc:773] GetModel() 'postprocess_16' version 1 I0105 08:43:12.001677 621925 model_repository_manager.cc:1352] successfully loaded 'ensemble_python_smoke_16' version 1 I0105 08:43:12.001690 621925 model_repository_manager.cc:1151] TriggerNextAction() 'ensemble_python_smoke_16' version 1: 0 I0105 08:43:12.001704 621925 model_repository_manager.cc:1165] no next action, trigger OnComplete() I0105 08:43:12.001722 621925 model_repository_manager.cc:728] VersionStates() 'ensemble_python_smoke_16' I0105 08:43:12.001740 621925 model_repository_manager.cc:728] VersionStates() 'ensemble_python_smoke_16' I0105 08:43:12.001760 621925 model_repository_manager.cc:773] GetModel() 'ensemble_python_smoke_16' version -1 I0105 08:43:12.036066 621925 model_repository_manager.cc:773] GetModel() 'ensemble_python_smoke_16' version -1 I0105 08:43:12.036129 621925 infer_request.cc:713] [request id: 0] prepared: [0x0x561d56125390] request id: 0, model: ensemble_python_smoke_16, requested version: -1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0 original inputs: [0x0x561d56a0d688] input: INPUT, type: UINT8, original shape: [512,512,3], batch + shape: [512,512,3], shape: [512,512,3] override inputs: inputs: [0x0x561d56a0d688] input: INPUT, type: UINT8, original shape: [512,512,3], batch + shape: [512,512,3], shape: [512,512,3] original requested outputs: OUTPUT requested outputs: OUTPUT I0105 08:43:12.036157 621925 model_repository_manager.cc:773] GetModel() 'preprocess_16' version 1 I0105 08:43:12.036174 621925 model_repository_manager.cc:773] GetModel() 'smoke_16' version 1 I0105 08:43:12.036191 621925 model_repository_manager.cc:773] GetModel() 'postprocess_16' version 1 I0105 08:43:12.036222 621925 infer_request.cc:713] [request id: 0] prepared: [0x0x561d5613fee0] request id: 0, model: preprocess_16, requested version: 1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0 original inputs: [0x0x561d565e9948] input: INPUT, type: UINT8, original shape: [512,512,3], batch + shape: [512,512,3], shape: [512,512,3] override inputs: inputs: [0x0x561d565e9948] input: INPUT, type: UINT8, original shape: [512,512,3], batch + shape: [512,512,3], shape: [512,512,3] original requested outputs: Image InitVector requested outputs: Image InitVector I0105 08:43:12.036298 621925 python_be.cc:1009] model preprocess_16, instance preprocess_16_0, executing 1 requests I0105 08:43:12.051219 621925 infer_response.cc:167] add response output: output: Image, type: FP16, shape: [1,512,512,6] I0105 08:43:12.051273 621925 pinned_memory_manager.cc:161] pinned memory allocation: size 3145728, addr 0x7f156c5a00a0 I0105 08:43:12.051284 621925 ensemble_scheduler.cc:540] Internal response allocation: Image, size 3145728, addr 0x7f156c5a00a0, memory type 1, type id 0 I0105 08:43:12.051754 621925 infer_response.cc:167] add response output: output: InitVector, type: FP16, shape: [1,1,1,180] I0105 08:43:12.051792 621925 pinned_memory_manager.cc:161] pinned memory allocation: size 360, addr 0x7f156c8a00b0 I0105 08:43:12.051803 621925 ensemble_scheduler.cc:540] Internal response allocation: InitVector, size 360, addr 0x7f156c8a00b0, memory type 1, type id 0 I0105 08:43:12.051825 621925 ensemble_scheduler.cc:555] Internal response release: size 3145728, addr 0x7f156c5a00a0 I0105 08:43:12.051836 621925 ensemble_scheduler.cc:555] Internal response release: size 360, addr 0x7f156c8a00b0 I0105 08:43:12.051856 621925 infer_request.cc:713] [request id: 0] prepared: [0x0x7f13900052a0] request id: 0, model: smoke_16, requested version: 1, actual version: 1, flags: 0x0, correlation id: 0, batch size: 0, priority: 0, timeout (us): 0 original inputs: [0x0x7f1390005768] input: Image, type: FP16, original shape: [1,512,512,6], batch + shape: [1,512,512,6], shape: [1,512,512,6] [0x0x7f13900055d8] input: InitVector, type: FP16, original shape: [1,1,1,180], batch + shape: [1,1,1,180], shape: [1,1,1,180] override inputs: inputs: [0x0x7f13900055d8] input: InitVector, type: FP16, original shape: [1,1,1,180], batch + shape: [1,1,1,180], shape: [1,1,1,180] [0x0x7f1390005768] input: Image, type: FP16, original shape: [1,512,512,6], batch + shape: [1,512,512,6], shape: [1,512,512,6] original requested outputs: detection_machine_2 requested outputs: detection_machine_2 ERROR: infer_trtis_server.cpp:259 Triton: TritonServer response error received., triton_err_str:Invalid argument, err_msg:in ensemble 'ensemble_python_smoke_16', inference request to model 'smoke_16' must specify a non-zero or non-empty correlation ID ERROR: infer_trtis_backend.cpp:603 Triton server failed to parse response with request-id:0 model: I0105 08:43:12.051915 621925 pinned_memory_manager.cc:190] pinned memory deallocation: addr 0x7f156c5a00a0 ERROR: infer_trtis_backend.cpp:359 failed to specify dims after running inference failed on model:ensemble_python_smoke_16, nvinfer error:NVDSINFER_TRITON_ERROR I0105 08:43:12.051928 621925 pinned_memory_manager.cc:190] pinned memory deallocation: addr 0x7f156c8a00b0 I0105 08:43:12.051992 621925 python_be.cc:1884] TRITONBACKEND_ModelInstanceExecute: model instance name preprocess_16_0 released 1 requests 0:00:03.727830140 621925 0x7f157c0022d0 ERROR   nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 1]: Error in specifyBackendDims() [UID = 1]: failed to specify input dims triton backend for model:ensemble_python_smoke_16, nvinfer error:NVDSINFER_TRITON_ERROR 0:00:03.727863932 621925 0x7f157c0022d0 ERROR   nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 1]: Error in createNNBackend() [UID = 1]: failed to specify triton backend input dims for model:ensemble_python_smoke_16, nvinfer error:NVDSINFER_TRITON_ERROR I0105 08:43:12.052345 621925 model_repository_manager.cc:858] AsyncUnload() 'ensemble_python_smoke_16' I0105 08:43:12.052370 621925 model_repository_manager.cc:1151] TriggerNextAction() 'ensemble_python_smoke_16' version 1: 2 I0105 08:43:12.052387 621925 model_repository_manager.cc:1223] Unload() 'ensemble_python_smoke_16' version 1 I0105 08:43:12.052403 621925 model_repository_manager.cc:1230] unloading: ensemble_python_smoke_16:1 I0105 08:43:12.052479 621925 model_repository_manager.cc:728] VersionStates() 'ensemble_python_smoke_16' I0105 08:43:12.052556 621925 model_repository_manager.cc:1333] OnDestroy callback() 'ensemble_python_smoke_16' version 1 I0105 08:43:12.052590 621925 model_repository_manager.cc:1335] successfully unloaded 'ensemble_python_smoke_16' version 1 I0105 08:43:12.052610 621925 model_repository_manager.cc:1151] TriggerNextAction() 'ensemble_python_smoke_16' version 1: 0 I0105 08:43:12.052636 621925 server.cc:260] Waiting for in-flight requests to complete. I0105 08:43:12.052655 621925 model_repository_manager.cc:660] StopAllModels() I0105 08:43:12.052675 621925 model_repository_manager.cc:679] InflightStatus() I0105 08:43:12.052691 621925 server.cc:276] Timeout 30: Found 0 model versions that have in-flight inferences I0105 08:43:12.052708 621925 model_repository_manager.cc:858] AsyncUnload() 'smoke_16' I0105 08:43:12.052725 621925 model_repository_manager.cc:1151] TriggerNextAction() 'smoke_16' version 1: 2 I0105 08:43:12.052742 621925 model_repository_manager.cc:1223] Unload() 'smoke_16' version 1 I0105 08:43:12.052758 621925 model_repository_manager.cc:1230] unloading: smoke_16:1 I0105 08:43:12.052794 621925 model_repository_manager.cc:858] AsyncUnload() 'postprocess_16' I0105 08:43:12.052811 621925 model_repository_manager.cc:1151] TriggerNextAction() 'postprocess_16' version 1: 2 I0105 08:43:12.052830 621925 model_repository_manager.cc:1223] Unload() 'postprocess_16' version 1 I0105 08:43:12.052831 621925 backend_model_instance.cc:758] Stopping backend thread for smoke_16_0... I0105 08:43:12.052847 621925 model_repository_manager.cc:1230] unloading: postprocess_16:1 I0105 08:43:12.052910 621925 tensorrt.cc:5625] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0105 08:43:12.052982 621925 backend_model_instance.cc:758] Stopping backend thread for postprocess_16_0... I0105 08:43:12.052964 621925 model_repository_manager.cc:858] AsyncUnload() 'preprocess_16' I0105 08:43:12.053033 621925 python_be.cc:1902] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0105 08:43:12.053042 621925 model_repository_manager.cc:1151] TriggerNextAction() 'preprocess_16' version 1: 2 I0105 08:43:12.053082 621925 model_repository_manager.cc:1223] Unload() 'preprocess_16' version 1 I0105 08:43:12.053098 621925 model_repository_manager.cc:1230] unloading: preprocess_16:1 I0105 08:43:12.053136 621925 server.cc:291] All models are stopped, unloading models I0105 08:43:12.053151 621925 model_repository_manager.cc:624] LiveModelStates() I0105 08:43:12.053154 621925 backend_model_instance.cc:758] Stopping backend thread for preprocess_16_0... I0105 08:43:12.053168 621925 server.cc:298] Timeout 30: Found 3 live models and 0 in-flight non-inference requests I0105 08:43:12.053199 621925 server.cc:305] postprocess_16 v1: UNLOADING I0105 08:43:12.053218 621925 server.cc:305] preprocess_16 v1: UNLOADING I0105 08:43:12.053232 621925 server.cc:305] smoke_16 v1: UNLOADING I0105 08:43:12.053219 621925 python_be.cc:1902] TRITONBACKEND_ModelInstanceFinalize: delete instance state I0105 08:43:12.058390 621925 tensorrt.cc:5564] TRITONBACKEND_ModelFinalize: delete model state I0105 08:43:12.070280 621925 sequence_batch_scheduler.cc:882] Stopping sequence-batch reaper thread... I0105 08:43:12.070336 621925 sequence_batch_scheduler.cc:1500] Stopping Direct sequence-batch scheduler thread 0... I0105 08:43:12.070396 621925 pinned_memory_manager.cc:190] pinned memory deallocation: addr 0x7f156c000090 I0105 08:43:12.070433 621925 model_repository_manager.cc:1333] OnDestroy callback() 'smoke_16' version 1 I0105 08:43:12.070452 621925 model_repository_manager.cc:1335] successfully unloaded 'smoke_16' version 1 I0105 08:43:12.070463 621925 model_repository_manager.cc:1151] TriggerNextAction() 'smoke_16' version 1: 0 I0105 08:43:13.001227 621925 sequence_batch_scheduler.cc:876] Reaper: sleeping for 1000000us... I0105 08:43:13.053315 621925 model_repository_manager.cc:624] LiveModelStates() I0105 08:43:13.053470 621925 server.cc:298] Timeout 29: Found 2 live models and 0 in-flight non-inference requests I0105 08:43:13.053491 621925 server.cc:305] postprocess_16 v1: UNLOADING I0105 08:43:13.053510 621925 server.cc:305] preprocess_16 v1: UNLOADING I0105 08:43:13.372784 621925 python_be.cc:1746] TRITONBACKEND_ModelFinalize: delete model state I0105 08:43:13.372879 621925 sequence_batch_scheduler.cc:882] Stopping sequence-batch reaper thread... I0105 08:43:13.372972 621925 sequence_batch_scheduler.cc:1500] Stopping Direct sequence-batch scheduler thread 0... I0105 08:43:13.373028 621925 model_repository_manager.cc:1333] OnDestroy callback() 'postprocess_16' version 1 I0105 08:43:13.373049 621925 model_repository_manager.cc:1335] successfully unloaded 'postprocess_16' version 1 I0105 08:43:13.373065 621925 model_repository_manager.cc:1151] TriggerNextAction() 'postprocess_16' version 1: 0 I0105 08:43:13.533692 621925 python_be.cc:1746] TRITONBACKEND_ModelFinalize: delete model state I0105 08:43:13.533764 621925 model_repository_manager.cc:1333] OnDestroy callback() 'preprocess_16' version 1 I0105 08:43:13.533780 621925 model_repository_manager.cc:1335] successfully unloaded 'preprocess_16' version 1 I0105 08:43:13.533793 621925 model_repository_manager.cc:1151] TriggerNextAction() 'preprocess_16' version 1: 0 I0105 08:43:14.053598 621925 model_repository_manager.cc:624] LiveModelStates() I0105 08:43:14.053647 621925 server.cc:298] Timeout 28: Found 0 live models and 0 in-flight non-inference requests I0105 08:43:14.053661 621925 backend_manager.cc:111] unloading backend 'python' I0105 08:43:14.053678 621925 python_be.cc:1703] TRITONBACKEND_Finalize: Start I0105 08:43:14.053794 621925 python_be.cc:1708] TRITONBACKEND_Finalize: End I0105 08:43:14.053811 621925 backend_manager.cc:111] unloading backend 'tensorrt' 0:00:05.730026876 621925 0x7f157c0022d0 ERROR   nvinferserver gstnvinferserver.cpp:375:gst_nvinfer_server_logger: nvinferserver[UID 1]: Error in initialize() [UID = 1]: create nn-backend failed, check config file settings, nvinfer error:NVDSINFER_TRITON_ERROR 0:00:05.730060086 621925 0x7f157c0022d0 WARN   nvinferserver gstnvinferserver_impl.cpp:547:start: error: Failed to initialize InferTrtIsContext 0:00:05.730094990 621925 0x7f157c0022d0 WARN   nvinferserver gstnvinferserver_impl.cpp:547:start: error: Config file path: /opt/nvidia/deepstream/deepstream-6.1/sources/project/topdownnet-python-triton/config_triton_inferserver_primary_smoke_16.txt 0:00:05.730175509 621925 0x7f157c0022d0 WARN   nvinferserver gstnvinferserver.cpp:473:gst_nvinfer_server_start: error: gstnvinferserver_impl start failed ERROR from primary_gie: Failed to initialize InferTrtIsContext Debug info: gstnvinferserver_impl.cpp(547): start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie: Config file path: /opt/nvidia/deepstream/deepstream-6.1/sources/project/topdownnet-python-triton/config_triton_inferserver_primary_smoke_16.txt ERROR from primary_gie: gstnvinferserver_impl start failed Debug info: gstnvinferserver.cpp(473): gst_nvinfer_server_start (): /GstPipeline:pipeline/GstBin:primary_gie_bin/GstNvInferServer:primary_gie