AttributeError while loading the model into AIAA server

I’ve pulled the docker image clara-train-sdk:v1.0-py3 and ran the container. The AIAA server is setup but when I try to upload the model using API endpoints
I’m getting this error on terminal:

> <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
> <title>500 Internal Server Error</title>
> <h1>Internal Server Error</h1>
> <p>The server encountered an internal error and was unable to complete your request.  Either the server is overloaded or there is an error in the application.</p>

further checking the logs, I came to know that ‘LoadNifty’ attribute is missing

[2020-07-15 06:45:24.120] INFO - Reading Config from Form Params
[2020-07-15 06:45:24.120] INFO - Input Model Config:
{‘type’: ‘segmentation’, ‘labels’: [‘spleen’], ‘writer’: {‘args’: {‘dtype’: ‘uint8’, ‘field’: ‘model’}, ‘name’: ‘WriteNifti’}, ‘inference’: {‘image’: ‘image’, ‘batch_size’: 3, ‘tf’: {‘input_nodes’: {‘image’: ‘NV_MODEL_INPUT’}, ‘output_nodes’: {‘model’: ‘NV_MODEL_OUTPUT’}}, ‘trtis’: {‘gpu_instance_count’: 1, ‘input_channels’: {‘image’: 1}, ‘output_channels’: {‘model’: 2}, ‘max_batch_size’: 8}, ‘roi’: [160, 160, 160], ‘scanning_window’: True}, ‘version’: ‘2’, ‘post_transforms’: [{‘args’: {‘fields’: ‘model’}, ‘name’: ‘ArgmaxAcrossChannels’}, {‘args’: {‘label_field’: ‘model’, ‘image_field’: ‘image’, ‘points’: ‘points’}, ‘name’: ‘FetchExtremePoints’}, {‘args’: {‘from_field’: ‘image’, ‘properties’: [‘affine’], ‘fields’: [‘model’]}, ‘name’: ‘CopyProperties’}, {‘args’: {‘src_field’: ‘image’, ‘field’: ‘model’, ‘is_label’: True}, ‘name’: ‘RestoreOriginalShape’}], ‘description’: ‘A pre-trained model for volumetric (3D) segmentation of the spleen from CT image’, ‘pre_transforms’: [{‘args’: {‘fields’: ‘image’}, ‘name’: ‘LoadNifti’}, {‘args’: {‘fields’: ‘image’}, ‘name’: ‘ConvertToChannelsFirst’}, {‘args’: {‘target_resolution’: [1.0, 1.0, 1.0], ‘fields’: ‘image’}, ‘name’: ‘ScaleByResolution’}, {‘args’: {‘b_min’: 0.0, ‘a_min’: -57, ‘fields’: ‘image’, ‘a_max’: 164, ‘b_max’: 1.0, ‘clip’: True}, ‘name’: ‘ScaleIntensityRange’}]}
[2020-07-15 06:45:25.318] INFO - 157.32.254.174 - - [15/Jul/2020 06:45:25] “e[37mGET /logs?lines=40 HTTP/1.1e[0m” 200 -
[2020-07-15 06:45:25.346] INFO - Saved archive file as: /var/nvidia/aiaa/downloads/host-5000/clara_ct_seg_spleen_amp_1.zip
[2020-07-15 06:45:25.346] INFO - Model: byom_seg_spleen
[2020-07-15 06:45:25.346] INFO - Unpack Archive: /var/nvidia/aiaa/downloads/host-5000/clara_ct_seg_spleen_amp_1.zip
[2020-07-15 06:45:28.460] INFO - 157.32.254.174 - - [15/Jul/2020 06:45:28] “e[37mGET /logs?lines=40 HTTP/1.1e[0m” 200 -
[2020-07-15 06:45:30.740] INFO - 157.32.254.174 - - [15/Jul/2020 06:45:30] “e[37mGET /logs?lines=40 HTTP/1.1e[0m” 200 -
[2020-07-15 06:45:35.027] INFO - Check for model at: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/resources
[2020-07-15 06:45:35.027] INFO - Check for model at: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/commands
[2020-07-15 06:45:35.027] INFO - Check for model at: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/models
[2020-07-15 06:45:35.027] INFO - Check for model at: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/config
[2020-07-15 06:45:35.027] INFO - Check for model at: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/docs
[2020-07-15 06:45:35.027] INFO - Model Dir: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/models
[2020-07-15 06:45:35.027] INFO - ++ Model Format: TRT
[2020-07-15 06:45:35.028] INFO - ++ Model File: model
[2020-07-15 06:45:35.028] INFO - No need of trt_export as downloaded data has: /var/nvidia/aiaa/downloads/host-5000/byom_seg_spleen/models/model.trt.pb
[2020-07-15 06:45:35.220] INFO - TRT Model for serving: /var/nvidia/aiaa/models/byom_seg_spleen.trt.pb
[2020-07-15 06:45:35.220] INFO - Init Model: byom_seg_spleen
[2020-07-15 06:45:35.221] INFO - Class Path: medical.tlt2.src.components.transforms.transforms.LoadNifti
[2020-07-15 06:45:35.221] INFO - Class Args: {‘fields’: ‘image’}
[2020-07-15 06:45:35.221]ERROR - Exception on /admin/model/byom_seg_spleen [PUT]
Traceback (most recent call last):
File “tlt2/src/utils/class_utils.py”, line 11, in get_class
AttributeError: module ‘medical.tlt2.src.components.transforms.transforms’ has no attribute ‘LoadNifti’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/usr/local/lib/python3.5/dist-packages/flask/app.py”, line 2311, in wsgi_app
response = self.full_dispatch_request()
File “/usr/local/lib/python3.5/dist-packages/flask/app.py”, line 1834, in full_dispatch_request
rv = self.handle_user_exception(e)
File “/usr/local/lib/python3.5/dist-packages/flask/app.py”, line 1737, in handle_user_exception
reraise(exc_type, exc_value, tb)
File “/usr/local/lib/python3.5/dist-packages/flask/_compat.py”, line 36, in reraise
raise value
File “/usr/local/lib/python3.5/dist-packages/flask/app.py”, line 1832, in full_dispatch_request
rv = self.dispatch_request()
File “/usr/local/lib/python3.5/dist-packages/flask/app.py”, line 1818, in dispatch_request
return self.view_functionsrule.endpoint
File “/usr/local/lib/python3.5/dist-packages/flask_monitoringdashboard/core/measurement.py”, line 61, in wrapper
result = fun(*args, **kwargs)
File “tlt2/src/apps/aas/server.py”, line 340, in admin_model_load
File “tlt2/src/apps/aas/server_context.py”, line 25, in put_model
File “tlt2/src/apps/aas/actions/inference.py”, line 35, in init
File “tlt2/src/apps/aas/actions/inference.py”, line 197, in _init_transform
File “tlt2/src/apps/aas/actions/inference.py”, line 197, in
File “tlt2/src/apps/aas/actions/inference.py”, line 211, in _build_transform
File “tlt2/src/utils/class_utils.py”, line 31, in instantiate_class
File “tlt2/src/utils/class_utils.py”, line 13, in get_class
ValueError: Class medical.tlt2.src.components.transforms.transforms.LoadNifti does not exist

How can I resolve this issue?

Hi
You are using a year old container clara-train-sdk:v1.0-py3 . Our SDK has improved significantly since then. we are on V3.0 now.

Please download and use the latest one from NGC and try again. https://ngc.nvidia.com/catalog/containers/nvidia:clara-train-sdk
docker pull nvcr.io/nvidia/clara-train-sdk:v3.0

Also please note the configs has changed a bit since V2 so you would need to re-download the mmars or convert them manually following https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v3.0/nvmidl/appendix/mapping_of_old_to_new.html

1 Like