INFO:__main__:Program started. INFO:app.App:Runtime env settings: {'input_folder': '/input', 'output_folder': '/output', 'logs_folder': '/logs', 'publish_folder': '/publish', 'series_selection_folder': '/series_selection', 'trtis_uri': '10.107.68.1:8000', 'config_inference_path': 'app_base_inference_v2/config/config_inference.json', 'nii_extension': '.nii'} INFO:app.App:Config details: {'name': 'LoadNifti', 'args': {'fields': 'image'}} INFO:app.App:Config details: {'name': 'ScaleToShape', 'args': {'fields': 'image', 'target_shape': [1, 256, 256]}} INFO:app.App:Config details: {'name': 'ConvertToChannelsFirst', 'args': {'fields': 'image'}} INFO:app.App:Config details: {'name': 'ScaleIntensityRange', 'args': {'fields': 'image', 'a_min': 1600, 'a_max': 3000, 'b_min': 0.0, 'b_max': 1.0, 'clip': True}} INFO:app.App:Config details: {'name': 'ArgmaxAcrossChannels', 'args': {'fields': 'model'}} INFO:app.App:Config details: {'name': 'RestoreOriginalShape', 'args': {'field': 'model', 'src_field': 'image', 'is_label': True}} INFO:app.App:Config details: {'name': 'CopyProperties', 'args': {'fields': ['model'], 'from_field': 'image', 'properties': ['affine']}} INFO:app.App:Model name:segmentation_chestxray_v1 INFO:app.App:Trying to check TRTIS server health ... INFO:model_loaders.trtis_config.TRTISConfig:output_type set to InferContext.ResultFormat.RAW: (1,) INFO:model_loaders.trtis_config.TRTISConfig:TRTIS Config {"ip": "10.107.68.1", "port": "8000", "protocol": 0, "verbose": false, "streaming": false, "output_type": [1], "shmem": "no", "use_cupy": false} WARNING:app.App:model_loader config item is not used. INFO:app.App:Found only one supported file. INFO:app.App:Selected input file: /input/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.mhd, and its extension: .mhd INFO:root:MHD metadata[ITK_InputFilterName]: MetaImageIO INFO:app.App:Input file converted for transforms: /tmp/tmpkrq_y7te/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.nii. Extracted properties: {'Spacing': (0.143, 0.143, 1.0), 'Origin': (0.0, 0.0, 0.0), 'Direction': (1.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 1.0)}. INFO:app.App:File list for transforms: [{'image': '/tmp/tmpkrq_y7te/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.nii'}] INFO:app.App:Writer dtype "uint8", field "model", and path "/output/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.mhd" INFO:app.App:Generated Output file name: /output/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.mhd INFO:TRTISScanWindowInferer:model_input_values: {'image': array([[[[[0. , 0. , 0. , ..., 0. , 0. , 0. ], [0. , 0. , 0. , ..., 0. , 0. , 0. ], [0. , 0. , 0. , ..., 0. , 0. , 0. ], ..., [0.32441354, 0.32441354, 0.32441354, ..., 0.32441354, 0.32441354, 0.32441354], [0.32635567, 0.32635567, 0.32635567, ..., 0.32635567, 0.32635567, 0.32635567], [0.3296378 , 0.3296378 , 0.3296378 , ..., 0.3296378 , 0.3296378 , 0.3296378 ]]]]], dtype=float32), 'image.affine': array([[[-0.14300001, 0. , 0. , -0. ], [ 0. , -0.14300001, 0. , -0. ], [ 0. , 0. , 1. , 0. ], [ 0. , 0. , 0. , 1. ]]]), 'image.original_affine': array([[[-0.14300001, 0. , 0. , -0. ], [ 0. , -0.14300001, 0. , -0. ], [ 0. , 0. , 1. , 0. ], [ 0. , 0. , 0. , 1. ]]]), 'image.file_name': array(['/tmp/tmpkrq_y7te/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.nii'], dtype=' NumClasses: 1 => Shape: (1, 1, 1, 256, 256) INFO:TRTISScanWindowInferer:Number of slices: 1 Failed to update context stat: [ 0] INTERNAL - Timer not set correctly. WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/Meta/src/itkMetaImageIO.cxx, line 659 MetaImageIO (0x4386fd0): Unsupported or empty metaData item ITK_FileNotes of type Ssfound, won't be written to image file WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/Meta/src/itkMetaImageIO.cxx, line 659 MetaImageIO (0x4386fd0): Unsupported or empty metaData item aux_file of type Ssfound, won't be written to image file WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/Meta/src/itkMetaImageIO.cxx, line 659 MetaImageIO (0x4386fd0): Unsupported or empty metaData item descrip of type Ssfound, won't be written to image file WARNING: In /tmp/SimpleITK-build/ITK/Modules/IO/Meta/src/itkMetaImageIO.cxx, line 659 MetaImageIO (0x4386fd0): Unsupported or empty metaData item intent_name of type Ssfound, won't be written to image file Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner self.run() File "/usr/lib/python3.6/threading.py", line 864, in run self._target(*self._args, **self._kwargs) File "/app_base_inference_v2/inferers/trtis_sw_inferer.py", line 501, in collect_response output_roi = infer_ctx.get_async_run_results(request_id) File "/usr/local/lib/python3.6/dist-packages/tensorrtserver/api/__init__.py", line 1724, in get_async_run_results self._last_request_id = _raise_if_error(err) File "/usr/local/lib/python3.6/dist-packages/tensorrtserver/api/__init__.py", line 261, in _raise_if_error raise ex tensorrtserver.api.InferenceServerException: [inference:0 13] NodeDef mentions attr 'explicit_paddings' not in Op output:T; attr=T:type,default=DT_FLOAT,allowed=[DT_HALF, DT_BFLOAT16, DT_FLOAT, DT_DOUBLE, DT_INT32, DT_INT64, DT_UINT8, DT_INT16, DT_INT8, DT_UINT16, DT_QINT8]; attr=ksize:list(int),min=4; attr=strides:list(int),min=4; attr=padding:string,allowed=["SAME", "VALID"]; attr=data_format:string,default="NHWC",allowed=["NHWC", "NCHW", "NCHW_VECT_C"]>; NodeDef: {{node model/max_pooling2d/MaxPool}}. (Check whether your GraphDef-interpreting binary is up to date with your GraphDef-generating binary.). [[model/max_pooling2d/MaxPool]] /app_base_inference_v2/inferers/trtis_sw_inferer.py:584: RuntimeWarning: invalid value encountered in true_divide output_dict[output_key] /= count_dict[output_key] INFO:app.App:Copying image files "/input/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.mhd" to "/publish/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.mhd" INFO:app.App:Copying source raw file /input/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.raw to destination /publish/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.raw INFO:app.App:Copying image files "/output/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.mhd" to "/publish/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.output.mhd" INFO:app.App:Copying source raw file /output/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.raw to destination /publish/1.3.12.2.1107.5.4.4.1486.30000016062023324415600000018.output.raw {"event": {"category": "operator", "name":"processing_started", "level": "info", "timestamp": "20211126T043834.662Z"}, "message": "AI-base_inference"} [PERF] AI-base_inference Start Time: 1637901514663 {"event": {"category": "operator", "name":"processing_started", "level": "info", "timestamp": "20211126T043835.683Z", "stage": "application setup"}, "message": "AI-base_inference Application setup"} [PERF] AI-base_inference Application setup Start Time: 1637901515683 {"event": {"category": "operator","name":"processing_ended", "level": "info", "timestamp": "20211126T043858.648Z", "elapsed_time": 22966, "stage": "application setup"},"message":"AI-base_inference Application setup"} [PERF] AI-base_inference Application setup End Time: 1637901538649 [PERF] AI-base_inference Application setup Elapsed Time (ms): 22966 {"event": {"category": "operator", "name":"processing_started", "level": "info", "timestamp": "20211126T043858.648Z", "stage": "application execute"}, "message": "AI-base_inference Application execute"} [PERF] AI-base_inference Application execute Start Time: 1637901538649 [driver] nvidia_clara_cpd__wait_for_completion() is deprecated. {"event": {"category": "operator", "name":"processing_started", "level": "info", "timestamp": "20211126T043858.648Z", "stage": "input selection and conversion"}, "message": "AI-base_inference Input selection and conversion"} Clara Pipeline Driver no longer expects applications to wait for completion. (0, is_err=True) [PERF] AI-base_inference Input selection and conversion Start Time: 1637901538649 INFO:__main__:Program exited. {"event": {"category": "operator","name":"processing_ended", "level": "info", "timestamp": "20211126T043859.146Z", "elapsed_time": 498, "stage": "input selection and conversion"},"message":"AI-base_inference Input selection and conversion"} [PERF] AI-base_inference Input selection and conversion End Time: 1637901539147 [PERF] AI-base_inference Input selection and conversion Elapsed Time (ms): 498 {"event": {"category": "operator", "name":"processing_started", "level": "info", "timestamp": "20211126T043859.146Z", "stage": "transforms and inference"}, "message": "AI-base_inference Transforms and Inference"} [PERF] AI-base_inference Transforms and Inference Start Time: 1637901539147 Previously evaluated: 0 ; To be evaluated: 1 Batch 1 / 1: 32.04s; pre-process: 29.72s; infer: 1.22s; post-process: 1.10s Total Inference Time: 1.2170994281768799s {"event": {"category": "operator","name":"processing_ended", "level": "info", "timestamp": "20211126T043931.186Z", "elapsed_time": 32040, "stage": "transforms and inference"},"message":"AI-base_inference Transforms and Inference"} [PERF] AI-base_inference Transforms and Inference End Time: 1637901571187 [PERF] AI-base_inference Transforms and Inference Elapsed Time (ms): 32040 {"event": {"category": "operator","name":"processing_ended", "level": "info", "timestamp": "20211126T043931.733Z", "elapsed_time": 33085, "stage": "application execute"},"message":"AI-base_inference Application execute"} [PERF] AI-base_inference Application execute End Time: 1637901571733 [PERF] AI-base_inference Application execute Elapsed Time (ms): 33085 {"event": {"category": "operator", "name":"processing_ended", "level": "info", "timestamp": "20211126T043931.733Z", "elapsed_time": 57071}, "message": "AI-base_inference"} [PERF] AI-base_inference End Time: 1637901571734 [PERF] AI-base_inference Elapsed Time (ms): 57071