annotation not visible on docker images

I run the docker v2 of clara train( https://ngc.nvidia.com/catalog/containers/nvidia:clara-train-sdk)
I don’t found any indication on how to import tha models. I f i use the comand describe on the manual (ngc comand) the data are located on temp but are not avaible for the server
Could you please someone help me?

Hi

Thanks for your interest in Clara Train and AIAA. Not exactly clear to me which (ngc) command you’re using; however the information on how to directly load your model once the AIAA server is stood up can be found here:

https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v2.0/aiaa/loading_models.html

Hope this helps.

Thanks.

thanks so much!!

I have use this comand but when I try to search models I dont’ found any model.

MODEL_NAME=clara_mri_seg_brain_tumors_br16_full_amp
VERSION=1

ngc registry model download-version nvidia/med/$MODEL_NAME:$VERSION --dest /var/tmp

I have also try to perform the tutorial but I have this error:

curl -v -X POST "http://0.0.0.0:5000/v1/segmentation?model=segmentation_2d_brain&output=image" \
>      -H "accept: multipart/form-data" \
>      -H "Content-Type: multipart/form-data" \
>      -F "params={}" \
>      -F "datapoint=@TCGA_CS_4944.png" \
>      -o result.png
Warning: setting file TCGA_CS_4944.png  failed!
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 0.0.0.0...
* TCP_NODELAY set
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* Connected to 0.0.0.0 (127.0.0.1) port 5000 (#0)
> POST /v1/segmentation?model=segmentation_2d_brain&output=image HTTP/1.1
> Host: 0.0.0.0:5000
> User-Agent: curl/7.58.0
> Transfer-Encoding: chunked
> accept: multipart/form-data
> Content-Type: multipart/form-data; boundary=------------------------c3dd80edd0ebfbb6
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
} [253 bytes data]
* read function returned funny value
* stopped the pause stream!
100   253    0     0    0   253      0   247k --:--:-- --:--:-- --:--:--  247k
* Closing connection 0
curl: (26) read function returned funny value

Hi there,

Thanks for your interest!

For your first question:

After u start your AIAA
Can you try:

curl -X PUT "http://0.0.0.0:5000/admin/model/clara_mri_seg_brain_tumors_br16_full_amp"      -H "accept: application/json"      -H "Content-Type: application/json"      -d '{"path":"nvidia/med/clara_mri_seg_brain_tumors_br16_full_amp","version":"1"}'

And check http://0.0.0.0:5000/v1/models
There should be one model there.
I just try this and it worked.

For the second question,
you say you are trying the tutorial,
You need to transform the model using nvidia pytorch 19.09 container

I have one here for you: https://drive.google.com/open?id=1aMUvnGtCHJCQpssOHb9Ife3vMr3TPXFz
Download this and then run through steps 1,3,4,5,6,7 in
https://docs.nvidia.com/clara/tlt-mi/clara-train-sdk-v2.0/aiaa/pytorch_tutorial.html

You should see a “result.png”

I have tried but I wan unable to import: Here What I obtained:

curl -X PUT "http://0.0.0.0:5000/admin/model/clara_mri_seg_brain_tumors_br16_full_amp"      -H "accept: application/json"      -H "Content-Type: application/json"      -d '{"path":"nvidia/med/clara_mri_seg_brain_tumors_br16_full_amp","version":"1"}'
Model/Config Not Found OR Failed to import model into AIAA
Check NGC Path OR Try to load from file-system with valid model+config
curl -X PUT "http://0.0.0.0:5000/admin/model/segmentation_2d_brain" \
>      -F "config=@segmentation_2d_brain.json;type=application/json" \
>      -F "data=@unet.pt"
Model/Config Not Found OR Failed to import model into AIAA
Load with valid model+config

log

nput_node_names "NV_MODEL_INPUT" --output_node_names "NV_MODEL_OUTPUT" --trt_min_seg_size 50
[2020-01-23 07:15:30.756][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Creating a regular frozen graph from Checkpoint at '/workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models' ...
[2020-01-23 07:15:30.756][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:28.775010: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:15:30.791][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Loaded meta graph file '/workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.ckpt.meta
[2020-01-23 07:15:30.792][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.731752: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:15:33.123][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Striping unused nodes with NVIDIA fix...
[2020-01-23 07:15:33.123][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.0-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[2020-01-23 07:15:33.198][ INFO](GenericUtils:run_command) - TRT_EXPORT:: 888 ops in the final graph.
[2020-01-23 07:15:33.198][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 
[2020-01-23 07:15:33.204][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Creating TRT-optimized graph ...
[2020-01-23 07:15:33.204][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.0-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[2020-01-23 07:15:33.744][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Validating TRT-optimized graph ...
[2020-01-23 07:15:33.744][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 
[2020-01-23 07:15:34.025][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Saving the TRT-optimized graph ...
[2020-01-23 07:15:34.025][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
[2020-01-23 07:15:34.103][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Frozen File Generated: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.fzn.pb
[2020-01-23 07:15:34.103][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Instructions for updating:
[2020-01-23 07:15:34.103][ INFO](GenericUtils:run_command) - TRT_EXPORT:: TRT File Generated: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.trt.pb
[2020-01-23 07:15:34.103][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Use standard file APIs to check for files with this prefix.
[2020-01-23 07:15:34.449][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.806735: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
[2020-01-23 07:15:34.449][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.827978: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.449][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.828182: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
[2020-01-23 07:15:34.449][ERROR](GenericUtils:run_command) - TRT_EXPORT:: name: Quadro RTX 4000 major: 7 minor: 5 memoryClockRate(GHz): 1.545
[2020-01-23 07:15:34.449][ERROR](GenericUtils:run_command) - TRT_EXPORT:: pciBusID: 0000:01:00.0
[2020-01-23 07:15:34.449][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.828193: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.828220: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.828232: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.828245: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.830091: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.830677: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.830700: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.830747: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.830959: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.831151: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.863392: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3600000000 Hz
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.864504: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5b534e0 executing computations on platform Host. Devices:
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.864515: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): <undefined>, <undefined>
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.939117: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.939519: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5b9b270 executing computations on platform CUDA. Devices:
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.939530: I tensorflow/compiler/xla/service/service.cc:175]   StreamExecutor device (0): Quadro RTX 4000, Compute Capability 7.5
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.939691: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940538: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: name: Quadro RTX 4000 major: 7 minor: 5 memoryClockRate(GHz): 1.545
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: pciBusID: 0000:01:00.0
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940580: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940658: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
[2020-01-23 07:15:34.450][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940673: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940699: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940709: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940718: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940726: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.940830: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.941158: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:30.941344: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:31.452166: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:31.452189: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:31.452193: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:31.452594: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:31.453059: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:31.453245: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1511 MB memory) -> physical GPU (device: 0, name: Quadro RTX 4000, pci bus id: 0000:01:00.0, compute capability: 7.5)
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:226: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Instructions for updating:
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Use `tf.compat.v1.graph_util.convert_variables_to_constants`
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/graph_util_impl.py:270: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
[2020-01-23 07:15:34.451][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Instructions for updating:
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Use `tf.compat.v1.graph_util.extract_sub_graph`
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From utils/strip_unused_lib.py:120: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Instructions for updating:
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Use tf.gfile.GFile.
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From apps/export.py:213: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From apps/export.py:214: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.395429: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.395719: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 1
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396131: I tensorflow/core/grappler/clusters/single_machine.cc:359] Starting new session
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396570: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396778: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: name: Quadro RTX 4000 major: 7 minor: 5 memoryClockRate(GHz): 1.545
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: pciBusID: 0000:01:00.0
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396797: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396825: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396835: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396844: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396855: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396865: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
[2020-01-23 07:15:34.452][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396874: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.396916: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397081: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397216: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397234: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397238: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397240: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397292: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397456: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.397594: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 1511 MB memory) -> physical GPU (device: 0, name: Quadro RTX 4000, pci bus id: 0000:01:00.0, compute capability: 7.5)
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.431064: W tensorflow/core/framework/allocator.cc:107] Allocation of 102760448 exceeds 10% of system memory.
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.642321: I tensorflow/compiler/tf2tensorrt/segment/segment.cc:460] There are 63 ops of 15 different types in the graph that are not converted to TensorRT: Merge, Cast, GreaterEqual, Conv3DBackpropInputV2, RandomUniform, Identity, StridedSlice, Placeholder, NoOp, Pack, Const, Switch, Conv3D, PlaceholderWithDefault, Shape, (For more information see https://docs.nvidia.com/deeplearning/dgx/tf-trt-user-guide/index.html#supported-ops).
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.657422: I tensorflow/compiler/tf2tensorrt/convert/convert_graph.cc:735] Number of TensorRT candidate segments: 0
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.657626: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.667600: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:752] Optimization results for grappler item: tf_graph
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.667640: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   constant folding: Graph size after: 751 nodes (-137), 856 edges (-140), time = 73.977ms.
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.667644: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   layout: Graph size after: 751 nodes (0), 856 edges (0), time = 21.423ms.
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.667646: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   constant folding: Graph size after: 751 nodes (0), 856 edges (0), time = 34.08ms.
[2020-01-23 07:15:34.453][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:15:33.667648: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:754]   TensorRTOptimizer: Graph size after: 751 nodes (0), 856 edges (0), time = 46.02ms.
[2020-01-23 07:15:34.462][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Return code: 0
[2020-01-23 07:15:34.462][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - Frozen File Generated: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.fzn.pb
[2020-01-23 07:15:34.462][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - TRT File Generated: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.trt.pb
[2020-01-23 07:15:34.484][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - TRT Model for serving: /workspace/data/storia//models/clara_mri_seg_brain_tumors_br16_full_amp.trt.pb
[2020-01-23 07:15:34.484][ INFO](nvmidl.apps.aas.actions.model_import_trtis:_import_model_trtis) - generating trtis model dir on /workspace/data/storia//trtis_models with model clara_mri_seg_brain_tumors_br16_full_amp
[2020-01-23 07:16:06.526][ERROR](nvmidl.apps.aas.actions.model_import_trtis:import_model) - Gen TRTIS model failed: Unable to get status for 'clara_mri_seg_brain_tumors_br16_full_amp'
[2020-01-23 07:16:06.544][ INFO](werkzeug:_log) - 127.0.0.1 - - [23/Jan/2020 07:16:06] "e[33mPUT /admin/model/clara_mri_seg_brain_tumors_br16_full_amp HTTP/1.1e[0m" 404 -
[2020-01-23 07:17:45.594][ INFO](nvmidl.apps.aas.server:admin_model_load) - Model: clara_mri_seg_brain_tumors_br16_full_amp
[2020-01-23 07:17:45.594][ INFO](nvmidl.apps.aas.server:check_for_access) - Request Host: 0.0.0.0:5000
[2020-01-23 07:17:45.595][ INFO](nvmidl.apps.aas.server:check_for_access) - Remote  Addr: 127.0.0.1
[2020-01-23 07:17:45.595][ INFO](nvmidl.apps.aas.actions.model_import:model_import_ngc) - Model: clara_mri_seg_brain_tumors_br16_full_amp; Path: nvidia/med/clara_mri_seg_brain_tumors_br16_full_amp; Version: 1
[2020-01-23 07:17:45.595][ INFO](nvmidl.apps.aas.actions.model_import:_ngc_download) - Trying to download model from NGC...
[2020-01-23 07:17:45.595][ INFO](nvmidl.apps.aas.actions.model_import:_ngc_download) - Running command: 
ngc registry model download-version nvidia/med/clara_mri_seg_brain_tumors_br16_full_amp:1 --dest /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp
[2020-01-23 07:17:45.624][ INFO](GenericUtils:run_command) - Running Command: ngc registry model download-version nvidia/med/clara_mri_seg_brain_tumors_br16_full_amp:1 --dest /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp
[2020-01-23 07:17:53.497][ INFO](GenericUtils:run_command) - NGC:: 
[2020-01-23 07:18:40.167][ INFO](schedule:run) - Running job Every 5 minutes do cleanup_docs() (last run: 2020-01-23 07:13:40, next run: 2020-01-23 07:18:40)
[2020-01-23 07:19:04.904][ INFO](GenericUtils:run_command) - NGC:: Return code: 0
[2020-01-23 07:19:04.904][ INFO](nvmidl.apps.aas.actions.model_import:_ngc_download) - Download Complete!!!
[2020-01-23 07:19:04.904][ INFO](nvmidl.apps.aas.actions.model_import:model_import_ngc) - Check for model+config at: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1
[2020-01-23 07:19:04.904][ INFO](nvmidl.apps.aas.actions.model_import:model_import_ngc) - CWD Download Dir: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1
[2020-01-23 07:19:04.904][ INFO](nvmidl.apps.aas.actions.model_import:_get_model_config) - Using Model Config File: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/config/config_aiaa.json
[2020-01-23 07:19:04.904][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - Model Dir: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models
[2020-01-23 07:19:04.905][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - ++ Model Format: CKPT
[2020-01-23 07:19:04.905][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - ++ Model File: model
[2020-01-23 07:19:04.905][ INFO](nvmidl.apps.aas.actions.model_import:import_model) - Executing TRT_EXPORT
[2020-01-23 07:19:04.932][ INFO](GenericUtils:run_command) - Running Command: python3 -u -m nvmidl.apps.export --model_file_format CKPT --model_file_path /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models --model_name model --input_node_names "NV_MODEL_INPUT" --output_node_names "NV_MODEL_OUTPUT" --trt_min_seg_size 50
[2020-01-23 07:19:07.126][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Creating a regular frozen graph from Checkpoint at '/workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models' ...
[2020-01-23 07:19:07.126][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:05.221642: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:19:07.164][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Loaded meta graph file '/workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.ckpt.meta
[2020-01-23 07:19:07.164][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.106226: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:19:09.304][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Striping unused nodes with NVIDIA fix...
[2020-01-23 07:19:09.304][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.0-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:117: The name tf.global_variables is deprecated. Please use tf.compat.v1.global_variables instead.
[2020-01-23 07:19:09.379][ INFO](GenericUtils:run_command) - TRT_EXPORT:: 888 ops in the final graph.
[2020-01-23 07:19:09.379][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 
[2020-01-23 07:19:09.384][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Creating TRT-optimized graph ...
[2020-01-23 07:19:09.384][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/horovod-0.18.0-py3.6-linux-x86_64.egg/horovod/tensorflow/__init__.py:143: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
[2020-01-23 07:19:09.933][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Validating TRT-optimized graph ...
[2020-01-23 07:19:09.933][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 
[2020-01-23 07:19:10.243][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Saving the TRT-optimized graph ...
[2020-01-23 07:19:10.243][ERROR](GenericUtils:run_command) - TRT_EXPORT:: WARNING:tensorflow:From /usr/local/lib/python3.6/dist-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
[2020-01-23 07:19:10.329][ INFO](GenericUtils:run_command) - TRT_EXPORT:: Frozen File Generated: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.fzn.pb
[2020-01-23 07:19:10.329][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Instructions for updating:
[2020-01-23 07:19:10.329][ INFO](GenericUtils:run_command) - TRT_EXPORT:: TRT File Generated: /workspace/data/storia//downloads/host-5000/clara_mri_seg_brain_tumors_br16_full_amp/clara_mri_seg_brain_tumors_br16_full_amp_v1/models/model.trt.pb
[2020-01-23 07:19:10.329][ERROR](GenericUtils:run_command) - TRT_EXPORT:: Use standard file APIs to check for files with this prefix.
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.170772: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.192226: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.192434: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: name: Quadro RTX 4000 major: 7 minor: 5 memoryClockRate(GHz): 1.545
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: pciBusID: 0000:01:00.0
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.192447: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.1
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.192476: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.192489: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.192501: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.193770: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.194536: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.194580: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.194665: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.194994: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1005] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.195227: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
[2020-01-23 07:19:10.700][ERROR](GenericUtils:run_command) - TRT_EXPORT:: 2020-01-23 07:19:07.219440: I tensorflow/co

Hi can you try to start AIAA with this additional option

“–engine AIAA”

then try just the first example