Trt_convert converter.convert() gets killed without errors

Autonomous Machines Jetson & Embedded Systems Jetson Xavier NX
#1

Hello, I am trying to convert this model (I saved the model as “saved_model.pb” using this script".

The script am using for the conversion is as follows:
“”"
from tensorflow.python.compiler.tensorrt import trt_convert as trt
from absl import app, flags, logging
from absl.flags import FLAGS
import argparse
import tensorflow as tf
import os

ap = argparse.ArgumentParser()
ap.add_argument("-i", “–input_saved_model_dir”, type=str, default="./serving/yolov3/1",
help=“path to saved model”)
args = vars(ap.parse_args())

model_name=‘yolov3’
trt_model = os.path.join(os.path.dirname(os.path.abspath(file)),(model_name+"_trt"))
if not os.path.exists(trt_model):
os.mkdir(trt_model)

print(trt_model)
print("\nconverting to trt-model")
converter = trt.TrtGraphConverterV2(input_saved_model_dir=args[‘input_saved_model_dir’])

print("\nconverter.convert")
converter.convert()
print("\nconverter.save")
converter.save(trt_model)

print(“trt-model saved under: “,trt_model)
“””
which produces the following output:
“””
2020-08-24 15:25:39.506148: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
/home/ucesfpa/Desktop/yolov3-tf2/yolov3_trt

converting to trt-model
2020-08-24 15:25:53.502740: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libnvinfer.so.7

converter.convert
2020-08-24 15:26:27.039598: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-24 15:26:27.113939: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:27.114249: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.109GHz coreCount: 6 deviceMemorySize: 7.58GiB deviceMemoryBandwidth: 66.10GiB/s
2020-08-24 15:26:27.114432: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:26:27.114603: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-24 15:26:27.264814: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-24 15:26:27.483716: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-24 15:26:27.747521: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-24 15:26:27.893602: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-24 15:26:27.893984: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-24 15:26:27.894504: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:27.895026: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:27.895218: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-24 15:26:27.931248: W tensorflow/core/platform/profile_utils/cpu_utils.cc:106] Failed to find bogomips or clock in /proc/cpuinfo; cannot determine CPU frequency
2020-08-24 15:26:27.932736: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3d10f950 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-08-24 15:26:27.933128: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-08-24 15:26:28.167276: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.167894: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3e5ca540 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-08-24 15:26:28.168032: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Xavier, Compute Capability 7.2
2020-08-24 15:26:28.225125: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.225565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.109GHz coreCount: 6 deviceMemorySize: 7.58GiB deviceMemoryBandwidth: 66.10GiB/s
2020-08-24 15:26:28.225756: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:26:28.225869: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-24 15:26:28.225958: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-24 15:26:28.226043: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-24 15:26:28.226126: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-24 15:26:28.226245: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-24 15:26:28.227439: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-24 15:26:28.227980: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.228561: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:28.228792: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-24 15:26:28.229031: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:26:46.202330: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 15:26:46.202493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-08-24 15:26:46.202555: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-08-24 15:26:46.203367: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:46.203831: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:26:46.204219: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2931 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2020-08-24 15:29:29.424069: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:29.467059: I tensorflow/core/grappler/devices.cc:55] Number of eligible GPUs (core count >= 8, compute capability >= 0.0): 0
2020-08-24 15:29:29.561791: I tensorflow/core/grappler/clusters/single_machine.cc:356] Starting new session
2020-08-24 15:29:29.944459: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:29.945085: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1561] Found device 0 with properties:
pciBusID: 0000:00:00.0 name: Xavier computeCapability: 7.2
coreClock: 1.109GHz coreCount: 6 deviceMemorySize: 7.58GiB deviceMemoryBandwidth: 66.10GiB/s
2020-08-24 15:29:30.133039: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.2
2020-08-24 15:29:30.390899: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-24 15:29:30.700141: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-24 15:29:30.724874: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-24 15:29:30.771131: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-24 15:29:30.817725: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-24 15:29:30.818107: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.8
2020-08-24 15:29:30.818518: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.818992: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.819307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1703] Adding visible gpu devices: 0
2020-08-24 15:29:30.873255: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-08-24 15:29:30.873354: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1108] 0
2020-08-24 15:29:30.873450: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1121] 0: N
2020-08-24 15:29:30.874762: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.875229: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:948] ARM64 does not support NUMA - returning NUMA node zero
2020-08-24 15:29:30.875476: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1247] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2931 MB memory) -> physical GPU (device: 0, name: Xavier, pci bus id: 0000:00:00.0, compute capability: 7.2)
2020-08-24 15:29:32.654548: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:824] Optimization results for grappler item: graph_to_optimize
2020-08-24 15:29:32.654711: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:826] function_optimizer: Graph size after: 1775 nodes (1403), 2812 edges (2437), time = 332.379ms.
2020-08-24 15:29:32.654811: I tensorflow/core/grappler/optimizers/meta_optimizer.cc:826] function_optimizer: function_optimizer did nothing. time = 2.455ms.
Killed
“”"
As there is no error I am not able to debug any further.
Can you please help me?

Best regards,
Fabio

#3

Hi,

Killed is usually caused by out of memory.
Would you mind to check the memory status with tegrastats and share the result with us first?

$ sudo tegrastats

Thanks.

#4

The following is the output of the $ sudo tegrastats command:

RAM 1999/7764MB (lfb 1094x4MB) SWAP 0/3882MB (cached 0MB) CPU [23%@1190,8%@1190,off,off,off,off] EMC_FREQ 1%@1600 GR3D_FREQ 0%@408 APE 150 MTS fg 0% bg 5% AO@37.5C GPU@37C PMIC@100C AUX@38.5C CPU@39C thermal@38.2C VDD_IN 3587/3798 VDD_CPU_GPU_CV 366/741 VDD_SOC 1261/1285

#5

Hi,

Please help to monitor the memory usage (RAM 1999/7764MB) when running the inference.
Does the occupied memory remains roughly as 1999 among the app execution time?

Thanks.

#6

Hi, thank you for your help. As you can see from the screenshot I think you are right.
What is your recommendation to overcome the problem?

Convert_trt
Convert_trt1911×963 407 KB

#7

Hi,

Would you mind to create some swap space to see if help?
You will need an extra space for the swap file.

Ex.

sudo fallocate -l 8G [/media/mySSD/swapfile]
sudo chmod 600  [/media/mySSD/swapfile]
sudo mkswap  [/media/mySSD/swapfile]
sudo /bin/sh -c 'echo  "[/media/mySSD/swapfile] \t none \t swap \t defaults \t 0 \t 0" >> /etc/fstab'
sudo swapon -a

Thanks.

Faster rcnn memory consumption
Deepstream out of memory when transfer the onnx model into engine on Jetson Nano board
Given there is .engine file & h5, how to incorporate it into Deepstream?
Cannot build a TensorRT engine for DLA from a large ONNX file
Mxnet installation for Jetson Nano
Custom Yolov3-tiny ota model update, segmentation fault
Run a UNet segmentation model on Jetson Nano / Convert pb to TensorRT
Couldn't install detectron2 on jetson nano
#9

does it overlap with zram swap existent in pre-installed OS envoronment on NX after flashing the jetson?
or co-exists with zram swap?
will the instruction require disabling the zram swap before creating the swap file?

#11

Hi,

They are different and can co-exist on the Xavier NX.

Swap memory can be turn off via swapoff command.
Please find more information with man swapoff

Thanks.