Trt_convert converter.convert() gets killed without errors

Hello, I am trying to convert this model (I saved the model as “saved_model.pb” using this script".

The script am using for the conversion is as follows:
“”"
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from absl import app, flags, logging
from absl.flags import FLAGS
import argparse
import tensorflow as tf
import os

ap = argparse.ArgumentParser()
ap.add_argument(“-i”, “–input_saved_model_dir”, type=str, default=“./serving/yolov3/1”,
help=“path to saved model”)
args = vars(ap.parse_args())

model_name=‘yolov3’
trt_model = os.path.join(os.path.dirname(os.path.abspath(file)),(model_name+“_trt”))
if not os.path.exists(trt_model):
os.mkdir(trt_model)

print(trt_model)
print(“\nconverting to trt-model”)
converter = trt.TrtGraphConverterV2(input_saved_model_dir=args[‘input_saved_model_dir’])

print(“\nconverter.convert”)
converter.convert()
print(“\nconverter.save”)
converter.save(trt_model)

print(“trt-model saved under: “,trt_model)
“””
which produces the following output:
“””
2020-08-24 15:25:39.506148: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
/home/ucesfpa/Desktop/yolov3-tf2/yolov3_trt

converting to trt-model
2020-08-24 15:25:53.502740: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer.so.7

converter.convert
2020-08-24 15:26:27.039598: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-24 15:26:27.113939: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:27.114249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.109GHz coreCount: 6 deviceMemorySize: 7.58GiB deviceMemoryBandwidth: 66.10GiB/s
2020-08-24 15:26:27.114432: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:26:27.114603: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-24 15:26:27.264814: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-24 15:26:27.483716: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-24 15:26:27.747521: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-24 15:26:27.893602: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-24 15:26:27.893984: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-24 15:26:27.894504: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:27.895026: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:27.895218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-24 15:26:27.931248: W tensorflow/core/platform/profile_utils/cpu_utils.cc:106] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2020-08-24 15:26:27.932736: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d10f950 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-24 15:26:27.933128: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-08-24 15:26:28.167276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.167894: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3e5ca540 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-24 15:26:28.168032: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Xavier, Compute Capability 7.2
2020-08-24 15:26:28.225125: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.225565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.109GHz coreCount: 6 deviceMemorySize: 7.58GiB deviceMemoryBandwidth: 66.10GiB/s
2020-08-24 15:26:28.225756: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:26:28.225869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-24 15:26:28.225958: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-24 15:26:28.226043: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-24 15:26:28.226126: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-24 15:26:28.226245: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-24 15:26:28.227439: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-24 15:26:28.227980: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.228561: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.228792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-24 15:26:28.229031: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:26:46.202330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 15:26:46.202493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-08-24 15:26:46.202555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-08-24 15:26:46.203367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:46.203831: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:46.204219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2931 MB memory) → physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2020-08-24 15:29:29.424069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:29.467059: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-08-24 15:29:29.561791: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-08-24 15:29:29.944459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:29.945085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.109GHz coreCount: 6 deviceMemorySize: 7.58GiB deviceMemoryBandwidth: 66.10GiB/s
2020-08-24 15:29:30.133039: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:29:30.390899: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-24 15:29:30.700141: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-24 15:29:30.724874: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-24 15:29:30.771131: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-24 15:29:30.817725: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-24 15:29:30.818107: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-24 15:29:30.818518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.818992: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.819307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-24 15:29:30.873255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 15:29:30.873354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-08-24 15:29:30.873450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-08-24 15:29:30.874762: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.875229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.875476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2931 MB memory) → physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2020-08-24 15:29:32.654548: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:824] Optimization results for grappler item: graph_to_optimize
2020-08-24 15:29:32.654711: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:826] function_optimizer: Graph size after: 1775 nodes (1403), 2812 edges (2437), time = 332.379ms.
2020-08-24 15:29:32.654811: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:826] function_optimizer: function_optimizer did nothing. time = 2.455ms.
Killed
“”"
As there is no error I am not able to debug any further.
Can you please help me?

Best regards,
Fabio

Hi,

Killed is usually caused by out of memory.
Would you mind to check the memory status with tegrastats and share the result with us first?

$ sudo tegrastats

Thanks.

The following is the output of the $ sudo tegrastats command:

RAM 1999/7764MB (lfb 1094x4MB) SWAP 0/3882MB (cached 0MB) CPU [23%@1190,8%@1190,off,off,off,off] EMC_FREQ 1%@1600 GR3D_FREQ 0%@408 APE 150 MTS fg 0% bg 5% AO@37.5C GPU@37C PMIC@100C AUX@38.5C CPU@39C thermal@38.2C VDD_IN 3587/3798 VDD_CPU_GPU_CV 366/741 VDD_SOC 1261/1285

Hi,

Please help to monitor the memory usage (RAM 1999/7764MB) when running the inference.
Does the occupied memory remains roughly as 1999 among the app execution time?

Thanks.

Hi, thank you for your help. As you can see from the screenshot I think you are right.
What is your recommendation to overcome the problem?

Hi,

Would you mind to create some swap space to see if help?
You will need an extra space for the swap file.

Ex.

sudo fallocate -l 8G [/media/mySSD/swapfile]
sudo chmod 600  [/media/mySSD/swapfile]
sudo mkswap  [/media/mySSD/swapfile]
sudo /bin/sh -c 'echo  "[/media/mySSD/swapfile] \t none \t swap \t defaults \t 0 \t 0" >> /etc/fstab'
sudo swapon -a

Thanks.

does it overlap with zram swap existent in pre-installed OS envoronment on NX after flashing the jetson?
or co-exists with zram swap?
will the instruction require disabling the zram swap before creating the swap file?

Hi,

They are different and can co-exist on the Xavier NX.

Swap memory can be turn off via swapoff command.
Please find more information with man swapoff

Thanks.