I0119 10:21:56.049489 17852 pinned_memory_manager.cc:240] Pinned memory pool is created at '0x7f781c000000' with size 268435456 I0119 10:21:56.049993 17852 cuda_memory_manager.cc:105] CUDA memory pool is created on device 0 with size 67108864 I0119 10:21:56.057207 17852 model_repository_manager.cc:1206] loading: yolo:1 2023-01-19 10:21:56.256426: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 I0119 10:21:56.297632 17852 tensorflow.cc:2531] TRITONBACKEND_Initialize: tensorflow I0119 10:21:56.297677 17852 tensorflow.cc:2541] Triton TRITONBACKEND API version: 1.10 I0119 10:21:56.297691 17852 tensorflow.cc:2547] 'tensorflow' TRITONBACKEND API version: 1.10 I0119 10:21:56.297704 17852 tensorflow.cc:2571] backend configuration: {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","backend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} I0119 10:21:56.297776 17852 tensorflow.cc:2637] TRITONBACKEND_ModelInitialize: yolo (version 1) 2023-01-19 10:21:56.301186: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /data/models/yolo/1/model.savedmodel 2023-01-19 10:21:56.366570: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve } 2023-01-19 10:21:56.417237: I tensorflow/core/platform/profile_utils/cpu_utils.cc:109] CPU Frequency: 2100000000 Hz 2023-01-19 10:21:56.425935: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f77ad22cdb0 initialized for platform Host (this does not guarantee that XLA will be used). Devices: 2023-01-19 10:21:56.425974: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version 2023-01-19 10:21:56.427731: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1 2023-01-19 10:21:56.429041: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7f77affd7410 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices: 2023-01-19 10:21:56.429060: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Quadro RTX 8000, Compute Capability 7.5 2023-01-19 10:21:56.429522: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1666] Found device 0 with properties: name: Quadro RTX 8000 major: 7 minor: 5 memoryClockRate(GHz): 1.62 pciBusID: 0000:5e:00.0 2023-01-19 10:21:56.429546: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2023-01-19 10:21:56.434980: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2023-01-19 10:21:56.467536: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2023-01-19 10:21:56.467884: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2023-01-19 10:21:56.468696: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2023-01-19 10:21:56.469727: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2023-01-19 10:21:56.469891: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2023-01-19 10:21:56.470425: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1794] Adding visible gpu devices: 0 2023-01-19 10:21:56.877388: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-01-19 10:21:56.877439: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0 2023-01-19 10:21:56.877451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N 2023-01-19 10:21:56.878131: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 43026 MB memory) -> physical GPU (device: 0, name: Quadro RTX 8000, pci bus id: 0000:5e:00.0, compute capability: 7.5) 2023-01-19 10:21:56.902807: I tensorflow/cc/saved_model/loader.cc:274] Restoring SavedModel bundle. 2023-01-19 10:21:57.100119: I tensorflow/cc/saved_model/loader.cc:218] Running initialization op on SavedModel bundle at path: /data/models/yolo/1/model.savedmodel 2023-01-19 10:21:57.311538: I tensorflow/cc/saved_model/loader.cc:402] SavedModel load for tags { serve }; Status: success. Took 1010356 microseconds. I0119 10:21:57.320907 17852 tensorflow.cc:2686] TRITONBACKEND_ModelInstanceInitialize: yolo (GPU device 0) 2023-01-19 10:21:57.321217: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /data/models/yolo/1/model.savedmodel 2023-01-19 10:21:57.349031: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve } 2023-01-19 10:21:57.354918: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1666] Found device 0 with properties: name: Quadro RTX 8000 major: 7 minor: 5 memoryClockRate(GHz): 1.62 pciBusID: 0000:5e:00.0 2023-01-19 10:21:57.354948: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0 2023-01-19 10:21:57.354969: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11 2023-01-19 10:21:57.354980: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10 2023-01-19 10:21:57.354989: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10 2023-01-19 10:21:57.354999: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.11 2023-01-19 10:21:57.355008: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11 2023-01-19 10:21:57.355017: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2023-01-19 10:21:57.355467: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1794] Adding visible gpu devices: 0 2023-01-19 10:21:57.355495: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1206] Device interconnect StreamExecutor with strength 1 edge matrix: 2023-01-19 10:21:57.355503: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1212] 0 2023-01-19 10:21:57.355510: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1225] 0: N 2023-01-19 10:21:57.356011: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1351] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 43026 MB memory) -> physical GPU (device: 0, name: Quadro RTX 8000, pci bus id: 0000:5e:00.0, compute capability: 7.5) 2023-01-19 10:21:57.365342: I tensorflow/cc/saved_model/loader.cc:274] Restoring SavedModel bundle. 2023-01-19 10:21:57.552840: I tensorflow/cc/saved_model/loader.cc:218] Running initialization op on SavedModel bundle at path: /data/models/yolo/1/model.savedmodel 2023-01-19 10:21:57.751883: I tensorflow/cc/saved_model/loader.cc:402] SavedModel load for tags { serve }; Status: success. Took 430670 microseconds. I0119 10:21:57.752175 17852 model_repository_manager.cc:1352] successfully loaded 'yolo' version 1 I0119 10:21:57.752318 17852 server.cc:559] +------------------+------+ | Repository Agent | Path | +------------------+------+ +------------------+------+ I0119 10:21:57.752430 17852 server.cc:586] +------------+-----------------------------------------------------------------+----------------------------------------------------------------------------------+ | Backend | Path | Config | +------------+-----------------------------------------------------------------+----------------------------------------------------------------------------------+ | tensorflow | /opt/tritonserver/backends/tensorflow1/libtriton_tensorflow1.so | {"cmdline":{"auto-complete-config":"true","min-compute-capability":"6.000000","b | | | | ackend-directory":"/opt/tritonserver/backends","default-max-batch-size":"4"}} | +------------+-----------------------------------------------------------------+----------------------------------------------------------------------------------+ I0119 10:21:57.752502 17852 server.cc:629] +-------+---------+--------+ | Model | Version | Status | +-------+---------+--------+ | yolo | 1 | READY | +-------+---------+--------+ I0119 10:21:57.806747 17852 metrics.cc:650] Collecting metrics for GPU 0: Quadro RTX 8000 I0119 10:21:57.807074 17852 tritonserver.cc:2176] +----------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | Option | Value | +----------------------------------+------------------------------------------------------------------------------------------------------------------------------+ | server_id | triton | | server_version | 2.24.0 | | server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shar | | | ed_memory cuda_shared_memory binary_tensor_data statistics trace | | model_repository_path[0] | /data/models/ | | model_control_mode | MODE_NONE | | strict_model_config | 0 | | rate_limit | OFF | | pinned_memory_pool_byte_size | 268435456 | | cuda_memory_pool_byte_size{0} | 67108864 | | response_cache_byte_size | 0 | | min_supported_compute_capability | 6.0 | | strict_readiness | 1 | | exit_timeout | 30 | +----------------------------------+------------------------------------------------------------------------------------------------------------------------------+ I0119 10:21:57.808396 17852 grpc_server.cc:4608] Started GRPCInferenceService at 0.0.0.0:8001 I0119 10:21:57.808660 17852 http_server.cc:3312] Started HTTPService at 0.0.0.0:8000 I0119 10:21:57.850155 17852 http_server.cc:178] Started Metrics Service at 0.0.0.0:8002 2023-01-19 10:22:06.596371: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8 2023-01-19 10:22:08.433564: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11