Faster rcnn memory consumption

Hi,

I have a jetson TX2 development kit and a Xavier NX development kit, both flashed JetPack 4.4 DP. And I ran a Faster rcnn inference based on VGG16 and caffe on both boards I found that the Faster rcnn on Xavier NX consumed much more memory than on TX2. On TX2 it consumed approximate 3.5G memory, but approximate 6G on NX. I tried JetPack 4.4 on Xavier NX as well, which has a much worse result the all memory has been totally consumed.

Could someone kindly explain why this happened? what is the different between TX2 and NX memory management?

There is my faster rcnn output on NX, same as output on TX2.
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0819 20:14:33.696404 14956 net.cpp:49] Initializing net from parameters:
name: “VGG_ILSVRC_16_layers”
input: “data”
input: “im_info”
state {
phase: TEST
}
input_shape {
dim: 1
dim: 3
dim: 224
dim: 224
}
input_shape {
dim: 1
dim: 3
}
layer {
name: “conv1_1”
type: “Convolution”
bottom: “data”
top: “conv1_1”
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
}
}
layer {
name: “relu1_1”
type: “ReLU”
bottom: “conv1_1”
top: “conv1_1”
}
layer {
name: “conv1_2”
type: “Convolution”
bottom: “conv1_1”
top: “conv1_2”
convolution_param {
num_output: 64
pad: 1
kernel_size: 3
}
}
layer {
name: “relu1_2”
type: “ReLU”
bottom: “conv1_2”
top: “conv1_2”
}
layer {
name: “pool1”
type: “Pooling”
bottom: “conv1_2”
top: “pool1”
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: “conv2_1”
type: “Convolution”
bottom: “pool1”
top: “conv2_1”
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
}
}
layer {
name: “relu2_1”
type: “ReLU”
bottom: “conv2_1”
top: “conv2_1”
}
layer {
name: “conv2_2”
type: “Convolution”
bottom: “conv2_1”
top: “conv2_2”
convolution_param {
num_output: 128
pad: 1
kernel_size: 3
}
}
layer {
name: “relu2_2”
type: “ReLU”
bottom: “conv2_2”
top: “conv2_2”
}
layer {
name: “pool2”
type: “Pooling”
bottom: “conv2_2”
top: “pool2”
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: “conv3_1”
type: “Convolution”
bottom: “pool2”
top: “conv3_1”
convolution_param {
num_output: 256
pad: 1
kernel_size: 3
}
}
layer {
name: “relu3_1”
type: “ReLU”
bottom: “conv3_1”
top: “conv3_1”
}
layer {
name: “conv3_2”
type: “Convolution”
bottom: “conv3_1”
top: “conv3_2”
convolution_param {
num_output: 256
pad: 1
kernel_size: 3
}
}
layer {
name: “relu3_2”
type: “ReLU”
bottom: “conv3_2”
top: “conv3_2”
}
layer {
name: “conv3_3”
type: “Convolution”
bottom: “conv3_2”
top: “conv3_3”
convolution_param {
num_output: 256
pad: 1
kernel_size: 3
}
}
layer {
name: “relu3_3”
type: “ReLU”
bottom: “conv3_3”
top: “conv3_3”
}
layer {
name: “pool3”
type: “Pooling”
bottom: “conv3_3”
top: “pool3”
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: “conv4_1”
type: “Convolution”
bottom: “pool3”
top: “conv4_1”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layer {
name: “relu4_1”
type: “ReLU”
bottom: “conv4_1”
top: “conv4_1”
}
layer {
name: “conv4_2”
type: “Convolution”
bottom: “conv4_1”
top: “conv4_2”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layer {
name: “relu4_2”
type: “ReLU”
bottom: “conv4_2”
top: “conv4_2”
}
layer {
name: “conv4_3”
type: “Convolution”
bottom: “conv4_2”
top: “conv4_3”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layer {
name: “relu4_3”
type: “ReLU”
bottom: “conv4_3”
top: “conv4_3”
}
layer {
name: “pool4”
type: “Pooling”
bottom: “conv4_3”
top: “pool4”
pooling_param {
pool: MAX
kernel_size: 2
stride: 2
}
}
layer {
name: “conv5_1”
type: “Convolution”
bottom: “pool4”
top: “conv5_1”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layer {
name: “relu5_1”
type: “ReLU”
bottom: “conv5_1”
top: “conv5_1”
}
layer {
name: “conv5_2”
type: “Convolution”
bottom: “conv5_1”
top: “conv5_2”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layer {
name: “relu5_2”
type: “ReLU”
bottom: “conv5_2”
top: “conv5_2”
}
layer {
name: “conv5_3”
type: “Convolution”
bottom: “conv5_2”
top: “conv5_3”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
}
}
layer {
name: “relu5_3”
type: “ReLU”
bottom: “conv5_3”
top: “conv5_3”
}
layer {
name: “rpn_conv/3x3”
type: “Convolution”
bottom: “conv5_3”
top: “rpn/output”
convolution_param {
num_output: 512
pad: 1
kernel_size: 3
stride: 1
}
}
layer {
name: “rpn_relu/3x3”
type: “ReLU”
bottom: “rpn/output”
top: “rpn/output”
}
layer {
name: “rpn_cls_score”
type: “Convolution”
bottom: “rpn/output”
top: “rpn_cls_score”
convolution_param {
num_output: 30
pad: 0
kernel_size: 1
stride: 1
}
}
layer {
name: “rpn_bbox_pred”
type: “Convolution”
bottom: “rpn/output”
top: “rpn_bbox_pred”
convolution_param {
num_output: 60
pad: 0
kernel_size: 1
stride: 1
}
}
layer {
name: “rpn_cls_score_reshape”
type: “Reshape”
bottom: “rpn_cls_score”
top: “rpn_cls_score_reshape”
reshape_param {
shape {
dim: 0
dim: 2
dim: -1
dim: 0
}
}
}
layer {
name: “rpn_cls_prob”
type: “Softmax”
bottom: “rpn_cls_score_reshape”
top: “rpn_cls_prob”
}
layer {
name: “rpn_cls_prob_reshape”
type: “Reshape”
bottom: “rpn_cls_prob”
top: “rpn_cls_prob_reshape”
reshape_param {
shape {
dim: 0
dim: 30
dim: -1
dim: 0
}
}
}
layer {
name: “proposal”
type: “Python”
bottom: “rpn_cls_prob_reshape”
bottom: “rpn_bbox_pred”
bottom: “im_info”
top: “rois”
python_param {
module: “proposal_layer”
layer: “ProposalLayer”
param_str: “‘feat_stride’: 16”
}
}
layer {
name: “roi_pool5”
type: “ROIPooling”
bottom: “conv5_3”
bottom: “rois”
top: “pool5”
roi_pooling_param {
pooled_h: 7
pooled_w: 7
spatial_scale: 0.0625
}
}
layer {
name: “fc6”
type: “InnerProduct”
bottom: “pool5”
top: “fc6”
inner_product_param {
num_output: 4096
}
}
layer {
name: “relu6”
type: “ReLU”
bottom: “fc6”
top: “fc6”
}
layer {
name: “fc7”
type: “InnerProduct”
bottom: “fc6”
top: “fc7”
inner_product_param {
num_output: 4096
}
}
layer {
name: “relu7”
type: “ReLU”
bottom: “fc7”
top: “fc7”
}
layer {
name: “cls_score”
type: “InnerProduct”
bottom: “fc7”
top: “cls_score”
inner_product_param {
num_output: 16
}
}
layer {
name: “bbox_pred”
type: “InnerProduct”
bottom: “fc7”
top: “bbox_pred”
inner_product_param {
num_output: 64
}
}
layer {
name: “cls_prob”
type: “Softmax”
bottom: “cls_score”
top: “cls_prob”
}
I0819 20:14:33.699580 14956 net.cpp:413] Input 0 -> data
I0819 20:14:33.975554 14956 net.cpp:413] Input 1 -> im_info
I0819 20:14:33.975764 14956 layer_factory.hpp:77] Creating layer conv1_1
I0819 20:14:33.975873 14956 net.cpp:106] Creating Layer conv1_1
I0819 20:14:33.975917 14956 net.cpp:454] conv1_1 <- data
I0819 20:14:33.975972 14956 net.cpp:411] conv1_1 -> conv1_1
I0819 20:14:38.397877 14956 net.cpp:150] Setting up conv1_1
I0819 20:14:38.397984 14956 net.cpp:157] Top shape: 1 64 224 224 (3211264)
I0819 20:14:38.398042 14956 net.cpp:165] Memory required for data: 12845056
I0819 20:14:38.398145 14956 layer_factory.hpp:77] Creating layer relu1_1
I0819 20:14:38.398203 14956 net.cpp:106] Creating Layer relu1_1
I0819 20:14:38.398265 14956 net.cpp:454] relu1_1 <- conv1_1
I0819 20:14:38.398303 14956 net.cpp:397] relu1_1 -> conv1_1 (in-place)
I0819 20:14:38.401821 14956 net.cpp:150] Setting up relu1_1
I0819 20:14:38.401906 14956 net.cpp:157] Top shape: 1 64 224 224 (3211264)
I0819 20:14:38.401962 14956 net.cpp:165] Memory required for data: 25690112
I0819 20:14:38.402063 14956 layer_factory.hpp:77] Creating layer conv1_2
I0819 20:14:38.402145 14956 net.cpp:106] Creating Layer conv1_2
I0819 20:14:38.402210 14956 net.cpp:454] conv1_2 <- conv1_1
I0819 20:14:38.402249 14956 net.cpp:411] conv1_2 -> conv1_2
I0819 20:14:38.417649 14956 net.cpp:150] Setting up conv1_2
I0819 20:14:38.417743 14956 net.cpp:157] Top shape: 1 64 224 224 (3211264)
I0819 20:14:38.417794 14956 net.cpp:165] Memory required for data: 38535168
I0819 20:14:38.417918 14956 layer_factory.hpp:77] Creating layer relu1_2
I0819 20:14:38.417964 14956 net.cpp:106] Creating Layer relu1_2
I0819 20:14:38.418042 14956 net.cpp:454] relu1_2 <- conv1_2
I0819 20:14:38.418081 14956 net.cpp:397] relu1_2 -> conv1_2 (in-place)
I0819 20:14:38.421238 14956 net.cpp:150] Setting up relu1_2
I0819 20:14:38.421310 14956 net.cpp:157] Top shape: 1 64 224 224 (3211264)
I0819 20:14:38.421360 14956 net.cpp:165] Memory required for data: 51380224
I0819 20:14:38.421391 14956 layer_factory.hpp:77] Creating layer pool1
I0819 20:14:38.421445 14956 net.cpp:106] Creating Layer pool1
I0819 20:14:38.421478 14956 net.cpp:454] pool1 <- conv1_2
I0819 20:14:38.421514 14956 net.cpp:411] pool1 -> pool1
I0819 20:14:38.421785 14956 net.cpp:150] Setting up pool1
I0819 20:14:38.421856 14956 net.cpp:157] Top shape: 1 64 112 112 (802816)
I0819 20:14:38.421895 14956 net.cpp:165] Memory required for data: 54591488
I0819 20:14:38.421952 14956 layer_factory.hpp:77] Creating layer conv2_1
I0819 20:14:38.421994 14956 net.cpp:106] Creating Layer conv2_1
I0819 20:14:38.422022 14956 net.cpp:454] conv2_1 <- pool1
I0819 20:14:38.422055 14956 net.cpp:411] conv2_1 -> conv2_1
I0819 20:14:38.442656 14956 net.cpp:150] Setting up conv2_1
I0819 20:14:38.442764 14956 net.cpp:157] Top shape: 1 128 112 112 (1605632)
I0819 20:14:38.442819 14956 net.cpp:165] Memory required for data: 61014016
I0819 20:14:38.442867 14956 layer_factory.hpp:77] Creating layer relu2_1
I0819 20:14:38.442970 14956 net.cpp:106] Creating Layer relu2_1
I0819 20:14:38.443006 14956 net.cpp:454] relu2_1 <- conv2_1
I0819 20:14:38.443039 14956 net.cpp:397] relu2_1 -> conv2_1 (in-place)
I0819 20:14:38.445365 14956 net.cpp:150] Setting up relu2_1
I0819 20:14:38.445438 14956 net.cpp:157] Top shape: 1 128 112 112 (1605632)
I0819 20:14:38.445523 14956 net.cpp:165] Memory required for data: 67436544
I0819 20:14:38.445556 14956 layer_factory.hpp:77] Creating layer conv2_2
I0819 20:14:38.445657 14956 net.cpp:106] Creating Layer conv2_2
I0819 20:14:38.445698 14956 net.cpp:454] conv2_2 <- conv2_1
I0819 20:14:38.445735 14956 net.cpp:411] conv2_2 -> conv2_2
I0819 20:14:38.464427 14956 net.cpp:150] Setting up conv2_2
I0819 20:14:38.464517 14956 net.cpp:157] Top shape: 1 128 112 112 (1605632)
I0819 20:14:38.464568 14956 net.cpp:165] Memory required for data: 73859072
I0819 20:14:38.464643 14956 layer_factory.hpp:77] Creating layer relu2_2
I0819 20:14:38.464690 14956 net.cpp:106] Creating Layer relu2_2
I0819 20:14:38.464716 14956 net.cpp:454] relu2_2 <- conv2_2
I0819 20:14:38.464757 14956 net.cpp:397] relu2_2 -> conv2_2 (in-place)
I0819 20:14:38.467469 14956 net.cpp:150] Setting up relu2_2
I0819 20:14:38.467561 14956 net.cpp:157] Top shape: 1 128 112 112 (1605632)
I0819 20:14:38.467613 14956 net.cpp:165] Memory required for data: 80281600
I0819 20:14:38.467643 14956 layer_factory.hpp:77] Creating layer pool2
I0819 20:14:38.467691 14956 net.cpp:106] Creating Layer pool2
I0819 20:14:38.467725 14956 net.cpp:454] pool2 <- conv2_2
I0819 20:14:38.467761 14956 net.cpp:411] pool2 -> pool2
I0819 20:14:38.467958 14956 net.cpp:150] Setting up pool2
I0819 20:14:38.467991 14956 net.cpp:157] Top shape: 1 128 56 56 (401408)
I0819 20:14:38.468026 14956 net.cpp:165] Memory required for data: 81887232
I0819 20:14:38.468053 14956 layer_factory.hpp:77] Creating layer conv3_1
I0819 20:14:38.468161 14956 net.cpp:106] Creating Layer conv3_1
I0819 20:14:38.468194 14956 net.cpp:454] conv3_1 <- pool2
I0819 20:14:38.468250 14956 net.cpp:411] conv3_1 -> conv3_1
I0819 20:14:38.482343 14956 net.cpp:150] Setting up conv3_1
I0819 20:14:38.482448 14956 net.cpp:157] Top shape: 1 256 56 56 (802816)
I0819 20:14:38.482501 14956 net.cpp:165] Memory required for data: 85098496
I0819 20:14:38.482560 14956 layer_factory.hpp:77] Creating layer relu3_1
I0819 20:14:38.482618 14956 net.cpp:106] Creating Layer relu3_1
I0819 20:14:38.482648 14956 net.cpp:454] relu3_1 <- conv3_1
I0819 20:14:38.482679 14956 net.cpp:397] relu3_1 -> conv3_1 (in-place)
I0819 20:14:38.486246 14956 net.cpp:150] Setting up relu3_1
I0819 20:14:38.486349 14956 net.cpp:157] Top shape: 1 256 56 56 (802816)
I0819 20:14:38.486440 14956 net.cpp:165] Memory required for data: 88309760
I0819 20:14:38.486472 14956 layer_factory.hpp:77] Creating layer conv3_2
I0819 20:14:38.486539 14956 net.cpp:106] Creating Layer conv3_2
I0819 20:14:38.486570 14956 net.cpp:454] conv3_2 <- conv3_1
I0819 20:14:38.486604 14956 net.cpp:411] conv3_2 -> conv3_2
I0819 20:14:38.515075 14956 net.cpp:150] Setting up conv3_2
I0819 20:14:38.515188 14956 net.cpp:157] Top shape: 1 256 56 56 (802816)
I0819 20:14:38.515257 14956 net.cpp:165] Memory required for data: 91521024
I0819 20:14:38.515326 14956 layer_factory.hpp:77] Creating layer relu3_2
I0819 20:14:38.515367 14956 net.cpp:106] Creating Layer relu3_2
I0819 20:14:38.515398 14956 net.cpp:454] relu3_2 <- conv3_2
I0819 20:14:38.515430 14956 net.cpp:397] relu3_2 -> conv3_2 (in-place)
I0819 20:14:38.521358 14956 net.cpp:150] Setting up relu3_2
I0819 20:14:38.522444 14956 net.cpp:157] Top shape: 1 256 56 56 (802816)
I0819 20:14:38.522799 14956 net.cpp:165] Memory required for data: 94732288
I0819 20:14:38.522876 14956 layer_factory.hpp:77] Creating layer conv3_3
I0819 20:14:38.523169 14956 net.cpp:106] Creating Layer conv3_3
I0819 20:14:38.523314 14956 net.cpp:454] conv3_3 <- conv3_2
I0819 20:14:38.523479 14956 net.cpp:411] conv3_3 -> conv3_3
I0819 20:14:38.551084 14956 net.cpp:150] Setting up conv3_3
I0819 20:14:38.551185 14956 net.cpp:157] Top shape: 1 256 56 56 (802816)
I0819 20:14:38.551265 14956 net.cpp:165] Memory required for data: 97943552
I0819 20:14:38.551316 14956 layer_factory.hpp:77] Creating layer relu3_3
I0819 20:14:38.551384 14956 net.cpp:106] Creating Layer relu3_3
I0819 20:14:38.551412 14956 net.cpp:454] relu3_3 <- conv3_3
I0819 20:14:38.551443 14956 net.cpp:397] relu3_3 -> conv3_3 (in-place)
I0819 20:14:38.561141 14956 net.cpp:150] Setting up relu3_3
I0819 20:14:38.561266 14956 net.cpp:157] Top shape: 1 256 56 56 (802816)
I0819 20:14:38.561321 14956 net.cpp:165] Memory required for data: 101154816
I0819 20:14:38.561349 14956 layer_factory.hpp:77] Creating layer pool3
I0819 20:14:38.561426 14956 net.cpp:106] Creating Layer pool3
I0819 20:14:38.561463 14956 net.cpp:454] pool3 <- conv3_3
I0819 20:14:38.561497 14956 net.cpp:411] pool3 -> pool3
I0819 20:14:38.561699 14956 net.cpp:150] Setting up pool3
I0819 20:14:38.561731 14956 net.cpp:157] Top shape: 1 256 28 28 (200704)
I0819 20:14:38.561825 14956 net.cpp:165] Memory required for data: 101957632
I0819 20:14:38.561856 14956 layer_factory.hpp:77] Creating layer conv4_1
I0819 20:14:38.561902 14956 net.cpp:106] Creating Layer conv4_1
I0819 20:14:38.561928 14956 net.cpp:454] conv4_1 <- pool3
I0819 20:14:38.561956 14956 net.cpp:411] conv4_1 -> conv4_1
I0819 20:14:38.597697 14956 net.cpp:150] Setting up conv4_1
I0819 20:14:38.597798 14956 net.cpp:157] Top shape: 1 512 28 28 (401408)
I0819 20:14:38.597844 14956 net.cpp:165] Memory required for data: 103563264
I0819 20:14:38.597893 14956 layer_factory.hpp:77] Creating layer relu4_1
I0819 20:14:38.597934 14956 net.cpp:106] Creating Layer relu4_1
I0819 20:14:38.597993 14956 net.cpp:454] relu4_1 <- conv4_1
I0819 20:14:38.598026 14956 net.cpp:397] relu4_1 -> conv4_1 (in-place)
I0819 20:14:38.602872 14956 net.cpp:150] Setting up relu4_1
I0819 20:14:38.603055 14956 net.cpp:157] Top shape: 1 512 28 28 (401408)
I0819 20:14:38.603135 14956 net.cpp:165] Memory required for data: 105168896
I0819 20:14:38.603168 14956 layer_factory.hpp:77] Creating layer conv4_2
I0819 20:14:38.603233 14956 net.cpp:106] Creating Layer conv4_2
I0819 20:14:38.603266 14956 net.cpp:454] conv4_2 <- conv4_1
I0819 20:14:38.603300 14956 net.cpp:411] conv4_2 -> conv4_2
I0819 20:14:38.643654 14956 net.cpp:150] Setting up conv4_2
I0819 20:14:38.643745 14956 net.cpp:157] Top shape: 1 512 28 28 (401408)
I0819 20:14:38.643841 14956 net.cpp:165] Memory required for data: 106774528
I0819 20:14:38.643898 14956 layer_factory.hpp:77] Creating layer relu4_2
I0819 20:14:38.643955 14956 net.cpp:106] Creating Layer relu4_2
I0819 20:14:38.643991 14956 net.cpp:454] relu4_2 <- conv4_2
I0819 20:14:38.644044 14956 net.cpp:397] relu4_2 -> conv4_2 (in-place)
I0819 20:14:38.650077 14956 net.cpp:150] Setting up relu4_2
I0819 20:14:38.650291 14956 net.cpp:157] Top shape: 1 512 28 28 (401408)
I0819 20:14:38.650350 14956 net.cpp:165] Memory required for data: 108380160
I0819 20:14:38.650383 14956 layer_factory.hpp:77] Creating layer conv4_3
I0819 20:14:38.650445 14956 net.cpp:106] Creating Layer conv4_3
I0819 20:14:38.650477 14956 net.cpp:454] conv4_3 <- conv4_2
I0819 20:14:38.650522 14956 net.cpp:411] conv4_3 -> conv4_3
I0819 20:14:38.683059 14956 net.cpp:150] Setting up conv4_3
I0819 20:14:38.683172 14956 net.cpp:157] Top shape: 1 512 28 28 (401408)
I0819 20:14:38.683223 14956 net.cpp:165] Memory required for data: 109985792
I0819 20:14:38.683271 14956 layer_factory.hpp:77] Creating layer relu4_3
I0819 20:14:38.683312 14956 net.cpp:106] Creating Layer relu4_3
I0819 20:14:38.683339 14956 net.cpp:454] relu4_3 <- conv4_3
I0819 20:14:38.683372 14956 net.cpp:397] relu4_3 -> conv4_3 (in-place)
I0819 20:14:38.695237 14956 net.cpp:150] Setting up relu4_3
I0819 20:14:38.695367 14956 net.cpp:157] Top shape: 1 512 28 28 (401408)
I0819 20:14:38.695425 14956 net.cpp:165] Memory required for data: 111591424
I0819 20:14:38.695456 14956 layer_factory.hpp:77] Creating layer pool4
I0819 20:14:38.695546 14956 net.cpp:106] Creating Layer pool4
I0819 20:14:38.695587 14956 net.cpp:454] pool4 <- conv4_3
I0819 20:14:38.695639 14956 net.cpp:411] pool4 -> pool4
I0819 20:14:38.695883 14956 net.cpp:150] Setting up pool4
I0819 20:14:38.695920 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.695961 14956 net.cpp:165] Memory required for data: 111992832
I0819 20:14:38.696025 14956 layer_factory.hpp:77] Creating layer conv5_1
I0819 20:14:38.696071 14956 net.cpp:106] Creating Layer conv5_1
I0819 20:14:38.696105 14956 net.cpp:454] conv5_1 <- pool4
I0819 20:14:38.696141 14956 net.cpp:411] conv5_1 -> conv5_1
I0819 20:14:38.739487 14956 net.cpp:150] Setting up conv5_1
I0819 20:14:38.739578 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.739624 14956 net.cpp:165] Memory required for data: 112394240
I0819 20:14:38.739672 14956 layer_factory.hpp:77] Creating layer relu5_1
I0819 20:14:38.739714 14956 net.cpp:106] Creating Layer relu5_1
I0819 20:14:38.739742 14956 net.cpp:454] relu5_1 <- conv5_1
I0819 20:14:38.739778 14956 net.cpp:397] relu5_1 -> conv5_1 (in-place)
I0819 20:14:38.745735 14956 net.cpp:150] Setting up relu5_1
I0819 20:14:38.745829 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.745878 14956 net.cpp:165] Memory required for data: 112795648
I0819 20:14:38.745908 14956 layer_factory.hpp:77] Creating layer conv5_2
I0819 20:14:38.745971 14956 net.cpp:106] Creating Layer conv5_2
I0819 20:14:38.746003 14956 net.cpp:454] conv5_2 <- conv5_1
I0819 20:14:38.746039 14956 net.cpp:411] conv5_2 -> conv5_2
I0819 20:14:38.787648 14956 net.cpp:150] Setting up conv5_2
I0819 20:14:38.787739 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.787782 14956 net.cpp:165] Memory required for data: 113197056
I0819 20:14:38.787830 14956 layer_factory.hpp:77] Creating layer relu5_2
I0819 20:14:38.787884 14956 net.cpp:106] Creating Layer relu5_2
I0819 20:14:38.787909 14956 net.cpp:454] relu5_2 <- conv5_2
I0819 20:14:38.787941 14956 net.cpp:397] relu5_2 -> conv5_2 (in-place)
I0819 20:14:38.793911 14956 net.cpp:150] Setting up relu5_2
I0819 20:14:38.794131 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.794205 14956 net.cpp:165] Memory required for data: 113598464
I0819 20:14:38.794242 14956 layer_factory.hpp:77] Creating layer conv5_3
I0819 20:14:38.794324 14956 net.cpp:106] Creating Layer conv5_3
I0819 20:14:38.794359 14956 net.cpp:454] conv5_3 <- conv5_2
I0819 20:14:38.794430 14956 net.cpp:411] conv5_3 -> conv5_3
I0819 20:14:38.838101 14956 net.cpp:150] Setting up conv5_3
I0819 20:14:38.838201 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.838258 14956 net.cpp:165] Memory required for data: 113999872
I0819 20:14:38.838348 14956 layer_factory.hpp:77] Creating layer relu5_3
I0819 20:14:38.838403 14956 net.cpp:106] Creating Layer relu5_3
I0819 20:14:38.838441 14956 net.cpp:454] relu5_3 <- conv5_3
I0819 20:14:38.838524 14956 net.cpp:397] relu5_3 -> conv5_3 (in-place)
I0819 20:14:38.843271 14956 net.cpp:150] Setting up relu5_3
I0819 20:14:38.843370 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.843458 14956 net.cpp:165] Memory required for data: 114401280
I0819 20:14:38.843492 14956 layer_factory.hpp:77] Creating layer conv5_3_relu5_3_0_split
I0819 20:14:38.843566 14956 net.cpp:106] Creating Layer conv5_3_relu5_3_0_split
I0819 20:14:38.843606 14956 net.cpp:454] conv5_3_relu5_3_0_split <- conv5_3
I0819 20:14:38.843662 14956 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_0
I0819 20:14:38.843752 14956 net.cpp:411] conv5_3_relu5_3_0_split -> conv5_3_relu5_3_0_split_1
I0819 20:14:38.843945 14956 net.cpp:150] Setting up conv5_3_relu5_3_0_split
I0819 20:14:38.843979 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.844015 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.844045 14956 net.cpp:165] Memory required for data: 115204096
I0819 20:14:38.844099 14956 layer_factory.hpp:77] Creating layer rpn_conv/3x3
I0819 20:14:38.844142 14956 net.cpp:106] Creating Layer rpn_conv/3x3
I0819 20:14:38.844177 14956 net.cpp:454] rpn_conv/3x3 <- conv5_3_relu5_3_0_split_0
I0819 20:14:38.844216 14956 net.cpp:411] rpn_conv/3x3 -> rpn/output
I0819 20:14:38.886345 14956 net.cpp:150] Setting up rpn_conv/3x3
I0819 20:14:38.886451 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.886507 14956 net.cpp:165] Memory required for data: 115605504
I0819 20:14:38.886555 14956 layer_factory.hpp:77] Creating layer rpn_relu/3x3
I0819 20:14:38.886610 14956 net.cpp:106] Creating Layer rpn_relu/3x3
I0819 20:14:38.886641 14956 net.cpp:454] rpn_relu/3x3 <- rpn/output
I0819 20:14:38.886675 14956 net.cpp:397] rpn_relu/3x3 -> rpn/output (in-place)
I0819 20:14:38.893805 14956 net.cpp:150] Setting up rpn_relu/3x3
I0819 20:14:38.893905 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.893970 14956 net.cpp:165] Memory required for data: 116006912
I0819 20:14:38.894003 14956 layer_factory.hpp:77] Creating layer rpn/output_rpn_relu/3x3_0_split
I0819 20:14:38.894078 14956 net.cpp:106] Creating Layer rpn/output_rpn_relu/3x3_0_split
I0819 20:14:38.894116 14956 net.cpp:454] rpn/output_rpn_relu/3x3_0_split <- rpn/output
I0819 20:14:38.894155 14956 net.cpp:411] rpn/output_rpn_relu/3x3_0_split -> rpn/output_rpn_relu/3x3_0_split_0
I0819 20:14:38.894197 14956 net.cpp:411] rpn/output_rpn_relu/3x3_0_split -> rpn/output_rpn_relu/3x3_0_split_1
I0819 20:14:38.894446 14956 net.cpp:150] Setting up rpn/output_rpn_relu/3x3_0_split
I0819 20:14:38.894485 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.894533 14956 net.cpp:157] Top shape: 1 512 14 14 (100352)
I0819 20:14:38.894610 14956 net.cpp:165] Memory required for data: 116809728
I0819 20:14:38.894639 14956 layer_factory.hpp:77] Creating layer rpn_cls_score
I0819 20:14:38.894696 14956 net.cpp:106] Creating Layer rpn_cls_score
I0819 20:14:38.894762 14956 net.cpp:454] rpn_cls_score <- rpn/output_rpn_relu/3x3_0_split_0
I0819 20:14:38.894863 14956 net.cpp:411] rpn_cls_score -> rpn_cls_score
I0819 20:14:38.919548 14956 net.cpp:150] Setting up rpn_cls_score
I0819 20:14:38.919739 14956 net.cpp:157] Top shape: 1 30 14 14 (5880)
I0819 20:14:38.919786 14956 net.cpp:165] Memory required for data: 116833248
I0819 20:14:38.919829 14956 layer_factory.hpp:77] Creating layer rpn_bbox_pred
I0819 20:14:38.919879 14956 net.cpp:106] Creating Layer rpn_bbox_pred
I0819 20:14:38.919909 14956 net.cpp:454] rpn_bbox_pred <- rpn/output_rpn_relu/3x3_0_split_1
I0819 20:14:38.919946 14956 net.cpp:411] rpn_bbox_pred -> rpn_bbox_pred
I0819 20:14:38.940076 14956 net.cpp:150] Setting up rpn_bbox_pred
I0819 20:14:38.940213 14956 net.cpp:157] Top shape: 1 60 14 14 (11760)
I0819 20:14:38.940290 14956 net.cpp:165] Memory required for data: 116880288
I0819 20:14:38.940379 14956 layer_factory.hpp:77] Creating layer rpn_cls_score_reshape
I0819 20:14:38.940513 14956 net.cpp:106] Creating Layer rpn_cls_score_reshape
I0819 20:14:38.940589 14956 net.cpp:454] rpn_cls_score_reshape <- rpn_cls_score
I0819 20:14:38.940641 14956 net.cpp:411] rpn_cls_score_reshape -> rpn_cls_score_reshape
I0819 20:14:38.940837 14956 net.cpp:150] Setting up rpn_cls_score_reshape
I0819 20:14:38.940870 14956 net.cpp:157] Top shape: 1 2 210 14 (5880)
I0819 20:14:38.940901 14956 net.cpp:165] Memory required for data: 116903808
I0819 20:14:38.940924 14956 layer_factory.hpp:77] Creating layer rpn_cls_prob
I0819 20:14:38.940956 14956 net.cpp:106] Creating Layer rpn_cls_prob
I0819 20:14:38.940979 14956 net.cpp:454] rpn_cls_prob <- rpn_cls_score_reshape
I0819 20:14:38.941009 14956 net.cpp:411] rpn_cls_prob -> rpn_cls_prob
I0819 20:14:38.950731 14956 net.cpp:150] Setting up rpn_cls_prob
I0819 20:14:38.950824 14956 net.cpp:157] Top shape: 1 2 210 14 (5880)
I0819 20:14:38.950875 14956 net.cpp:165] Memory required for data: 116927328
I0819 20:14:38.951004 14956 layer_factory.hpp:77] Creating layer rpn_cls_prob_reshape
I0819 20:14:38.951107 14956 net.cpp:106] Creating Layer rpn_cls_prob_reshape
I0819 20:14:38.951143 14956 net.cpp:454] rpn_cls_prob_reshape <- rpn_cls_prob
I0819 20:14:38.951206 14956 net.cpp:411] rpn_cls_prob_reshape -> rpn_cls_prob_reshape
I0819 20:14:38.951345 14956 net.cpp:150] Setting up rpn_cls_prob_reshape
I0819 20:14:38.951426 14956 net.cpp:157] Top shape: 1 30 14 14 (5880)
I0819 20:14:38.951467 14956 net.cpp:165] Memory required for data: 116950848
I0819 20:14:38.951489 14956 layer_factory.hpp:77] Creating layer proposal
I0819 20:14:38.951634 14956 net.cpp:106] Creating Layer proposal
I0819 20:14:38.951671 14956 net.cpp:454] proposal <- rpn_cls_prob_reshape
I0819 20:14:38.951701 14956 net.cpp:454] proposal <- rpn_bbox_pred
I0819 20:14:38.951731 14956 net.cpp:454] proposal <- im_info
I0819 20:14:38.951773 14956 net.cpp:411] proposal -> rois
I0819 20:14:38.955376 14956 net.cpp:150] Setting up proposal
I0819 20:14:38.955478 14956 net.cpp:157] Top shape: 1 5 (5)
I0819 20:14:38.955518 14956 net.cpp:165] Memory required for data: 116950868
I0819 20:14:38.955549 14956 layer_factory.hpp:77] Creating layer roi_pool5
I0819 20:14:38.955591 14956 net.cpp:106] Creating Layer roi_pool5
I0819 20:14:38.955619 14956 net.cpp:454] roi_pool5 <- conv5_3_relu5_3_0_split_1
I0819 20:14:38.955651 14956 net.cpp:454] roi_pool5 <- rois
I0819 20:14:38.955682 14956 net.cpp:411] roi_pool5 -> pool5
I0819 20:14:38.955724 14956 roi_pooling_layer.cpp:30] Spatial scale: 0.0625
I0819 20:14:38.955883 14956 net.cpp:150] Setting up roi_pool5
I0819 20:14:38.955916 14956 net.cpp:157] Top shape: 1 512 7 7 (25088)
I0819 20:14:38.955950 14956 net.cpp:165] Memory required for data: 117051220
I0819 20:14:38.956012 14956 layer_factory.hpp:77] Creating layer fc6
I0819 20:14:38.956044 14956 net.cpp:106] Creating Layer fc6
I0819 20:14:38.956069 14956 net.cpp:454] fc6 <- pool5
I0819 20:14:38.956128 14956 net.cpp:411] fc6 -> fc6
I0819 20:14:39.452057 14956 net.cpp:150] Setting up fc6
I0819 20:14:39.452199 14956 net.cpp:157] Top shape: 1 4096 (4096)
I0819 20:14:39.452267 14956 net.cpp:165] Memory required for data: 117067604
I0819 20:14:39.452332 14956 layer_factory.hpp:77] Creating layer relu6
I0819 20:14:39.452426 14956 net.cpp:106] Creating Layer relu6
I0819 20:14:39.452463 14956 net.cpp:454] relu6 <- fc6
I0819 20:14:39.452503 14956 net.cpp:397] relu6 -> fc6 (in-place)
I0819 20:14:39.458786 14956 net.cpp:150] Setting up relu6
I0819 20:14:39.458875 14956 net.cpp:157] Top shape: 1 4096 (4096)
I0819 20:14:39.459013 14956 net.cpp:165] Memory required for data: 117083988
I0819 20:14:39.459095 14956 layer_factory.hpp:77] Creating layer fc7
I0819 20:14:39.459156 14956 net.cpp:106] Creating Layer fc7
I0819 20:14:39.459197 14956 net.cpp:454] fc7 <- fc6
I0819 20:14:39.459265 14956 net.cpp:411] fc7 -> fc7
I0819 20:14:39.555208 14956 net.cpp:150] Setting up fc7
I0819 20:14:39.555320 14956 net.cpp:157] Top shape: 1 4096 (4096)
I0819 20:14:39.555382 14956 net.cpp:165] Memory required for data: 117100372
I0819 20:14:39.555475 14956 layer_factory.hpp:77] Creating layer relu7
I0819 20:14:39.555524 14956 net.cpp:106] Creating Layer relu7
I0819 20:14:39.555557 14956 net.cpp:454] relu7 <- fc7
I0819 20:14:39.555635 14956 net.cpp:397] relu7 -> fc7 (in-place)
I0819 20:14:39.561326 14956 net.cpp:150] Setting up relu7
I0819 20:14:39.561399 14956 net.cpp:157] Top shape: 1 4096 (4096)
I0819 20:14:39.561447 14956 net.cpp:165] Memory required for data: 117116756
I0819 20:14:39.561478 14956 layer_factory.hpp:77] Creating layer fc7_relu7_0_split
I0819 20:14:39.561563 14956 net.cpp:106] Creating Layer fc7_relu7_0_split
I0819 20:14:39.561604 14956 net.cpp:454] fc7_relu7_0_split <- fc7
I0819 20:14:39.561655 14956 net.cpp:411] fc7_relu7_0_split -> fc7_relu7_0_split_0
I0819 20:14:39.561719 14956 net.cpp:411] fc7_relu7_0_split -> fc7_relu7_0_split_1
I0819 20:14:39.561916 14956 net.cpp:150] Setting up fc7_relu7_0_split
I0819 20:14:39.561961 14956 net.cpp:157] Top shape: 1 4096 (4096)
I0819 20:14:39.562000 14956 net.cpp:157] Top shape: 1 4096 (4096)
I0819 20:14:39.562024 14956 net.cpp:165] Memory required for data: 117149524
I0819 20:14:39.562050 14956 layer_factory.hpp:77] Creating layer cls_score
I0819 20:14:39.562094 14956 net.cpp:106] Creating Layer cls_score
I0819 20:14:39.562121 14956 net.cpp:454] cls_score <- fc7_relu7_0_split_0
I0819 20:14:39.562157 14956 net.cpp:411] cls_score -> cls_score
I0819 20:14:39.562885 14956 net.cpp:150] Setting up cls_score
I0819 20:14:39.562968 14956 net.cpp:157] Top shape: 1 16 (16)
I0819 20:14:39.563006 14956 net.cpp:165] Memory required for data: 117149588
I0819 20:14:39.563081 14956 layer_factory.hpp:77] Creating layer bbox_pred
I0819 20:14:39.563125 14956 net.cpp:106] Creating Layer bbox_pred
I0819 20:14:39.563158 14956 net.cpp:454] bbox_pred <- fc7_relu7_0_split_1
I0819 20:14:39.563218 14956 net.cpp:411] bbox_pred -> bbox_pred
I0819 20:14:39.565522 14956 net.cpp:150] Setting up bbox_pred
I0819 20:14:39.565577 14956 net.cpp:157] Top shape: 1 64 (64)
I0819 20:14:39.565611 14956 net.cpp:165] Memory required for data: 117149844
I0819 20:14:39.565652 14956 layer_factory.hpp:77] Creating layer cls_prob
I0819 20:14:39.565697 14956 net.cpp:106] Creating Layer cls_prob
I0819 20:14:39.565733 14956 net.cpp:454] cls_prob <- cls_score
I0819 20:14:39.565768 14956 net.cpp:411] cls_prob -> cls_prob
I0819 20:14:39.572201 14956 net.cpp:150] Setting up cls_prob
I0819 20:14:39.572296 14956 net.cpp:157] Top shape: 1 16 (16)
I0819 20:14:39.572391 14956 net.cpp:165] Memory required for data: 117149908
I0819 20:14:39.572432 14956 net.cpp:228] cls_prob does not need backward computation.
I0819 20:14:39.572465 14956 net.cpp:228] bbox_pred does not need backward computation.
I0819 20:14:39.572536 14956 net.cpp:228] cls_score does not need backward computation.
I0819 20:14:39.572583 14956 net.cpp:228] fc7_relu7_0_split does not need backward computation.
I0819 20:14:39.572611 14956 net.cpp:228] relu7 does not need backward computation.
I0819 20:14:39.572638 14956 net.cpp:228] fc7 does not need backward computation.
I0819 20:14:39.572674 14956 net.cpp:228] relu6 does not need backward computation.
I0819 20:14:39.572705 14956 net.cpp:228] fc6 does not need backward computation.
I0819 20:14:39.572736 14956 net.cpp:228] roi_pool5 does not need backward computation.
I0819 20:14:39.572768 14956 net.cpp:228] proposal does not need backward computation.
I0819 20:14:39.572803 14956 net.cpp:228] rpn_cls_prob_reshape does not need backward computation.
I0819 20:14:39.572829 14956 net.cpp:228] rpn_cls_prob does not need backward computation.
I0819 20:14:39.572877 14956 net.cpp:228] rpn_cls_score_reshape does not need backward computation.
I0819 20:14:39.572911 14956 net.cpp:228] rpn_bbox_pred does not need backward computation.
I0819 20:14:39.572940 14956 net.cpp:228] rpn_cls_score does not need backward computation.
I0819 20:14:39.572976 14956 net.cpp:228] rpn/output_rpn_relu/3x3_0_split does not need backward computation.
I0819 20:14:39.572999 14956 net.cpp:228] rpn_relu/3x3 does not need backward computation.
I0819 20:14:39.573029 14956 net.cpp:228] rpn_conv/3x3 does not need backward computation.
I0819 20:14:39.573058 14956 net.cpp:228] conv5_3_relu5_3_0_split does not need backward computation.
I0819 20:14:39.573086 14956 net.cpp:228] relu5_3 does not need backward computation.
I0819 20:14:39.573112 14956 net.cpp:228] conv5_3 does not need backward computation.
I0819 20:14:39.573139 14956 net.cpp:228] relu5_2 does not need backward computation.
I0819 20:14:39.573164 14956 net.cpp:228] conv5_2 does not need backward computation.
I0819 20:14:39.573190 14956 net.cpp:228] relu5_1 does not need backward computation.
I0819 20:14:39.573217 14956 net.cpp:228] conv5_1 does not need backward computation.
I0819 20:14:39.573254 14956 net.cpp:228] pool4 does not need backward computation.
I0819 20:14:39.573319 14956 net.cpp:228] relu4_3 does not need backward computation.
I0819 20:14:39.573351 14956 net.cpp:228] conv4_3 does not need backward computation.
I0819 20:14:39.573376 14956 net.cpp:228] relu4_2 does not need backward computation.
I0819 20:14:39.573400 14956 net.cpp:228] conv4_2 does not need backward computation.
I0819 20:14:39.573457 14956 net.cpp:228] relu4_1 does not need backward computation.
I0819 20:14:39.573483 14956 net.cpp:228] conv4_1 does not need backward computation.
I0819 20:14:39.573509 14956 net.cpp:228] pool3 does not need backward computation.
I0819 20:14:39.573539 14956 net.cpp:228] relu3_3 does not need backward computation.
I0819 20:14:39.573565 14956 net.cpp:228] conv3_3 does not need backward computation.
I0819 20:14:39.573602 14956 net.cpp:228] relu3_2 does not need backward computation.
I0819 20:14:39.573632 14956 net.cpp:228] conv3_2 does not need backward computation.
I0819 20:14:39.573700 14956 net.cpp:228] relu3_1 does not need backward computation.
I0819 20:14:39.573731 14956 net.cpp:228] conv3_1 does not need backward computation.
I0819 20:14:39.573758 14956 net.cpp:228] pool2 does not need backward computation.
I0819 20:14:39.573786 14956 net.cpp:228] relu2_2 does not need backward computation.
I0819 20:14:39.573817 14956 net.cpp:228] conv2_2 does not need backward computation.
I0819 20:14:39.573845 14956 net.cpp:228] relu2_1 does not need backward computation.
I0819 20:14:39.573873 14956 net.cpp:228] conv2_1 does not need backward computation.
I0819 20:14:39.573915 14956 net.cpp:228] pool1 does not need backward computation.
I0819 20:14:39.573947 14956 net.cpp:228] relu1_2 does not need backward computation.
I0819 20:14:39.573977 14956 net.cpp:228] conv1_2 does not need backward computation.
I0819 20:14:39.574012 14956 net.cpp:228] relu1_1 does not need backward computation.
I0819 20:14:39.574041 14956 net.cpp:228] conv1_1 does not need backward computation.
I0819 20:14:39.574071 14956 net.cpp:270] This network produces output bbox_pred
I0819 20:14:39.574105 14956 net.cpp:270] This network produces output cls_prob
I0819 20:14:39.574242 14956 net.cpp:283] Network initialization done.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:604] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
[libprotobuf WARNING google/protobuf/io/coded_stream.cc:81] The total number of bytes read was 1027477531
I0819 20:14:52.118567 14956 net.cpp:816] Ignoring source layer data
I0819 20:14:52.118643 14956 net.cpp:816] Ignoring source layer rois_data_1_split
I0819 20:14:52.140255 14956 net.cpp:816] Ignoring source layer roi_pool5_readonly
I0819 20:14:52.140470 14956 net.cpp:816] Ignoring source layer fc6_readonly
I0819 20:14:52.140514 14956 net.cpp:816] Ignoring source layer relu6_readonly
I0819 20:14:52.140544 14956 net.cpp:816] Ignoring source layer drop6_readonly
I0819 20:14:52.140619 14956 net.cpp:816] Ignoring source layer fc7_readonly
I0819 20:14:52.140650 14956 net.cpp:816] Ignoring source layer relu7_readonly
I0819 20:14:52.140671 14956 net.cpp:816] Ignoring source layer drop7_readonly
I0819 20:14:52.140755 14956 net.cpp:816] Ignoring source layer fc7_readonly_drop7_readonly_0_split
I0819 20:14:52.140777 14956 net.cpp:816] Ignoring source layer cls_score_readonly
I0819 20:14:52.140805 14956 net.cpp:816] Ignoring source layer bbox_pred_readonly
I0819 20:14:52.140861 14956 net.cpp:816] Ignoring source layer cls_prob_readonly
I0819 20:14:52.140905 14956 net.cpp:816] Ignoring source layer hard_roi_mining
I0819 20:14:52.239678 14956 net.cpp:816] Ignoring source layer drop6
I0819 20:14:52.254129 14956 net.cpp:816] Ignoring source layer drop7
I0819 20:14:52.254225 14956 net.cpp:816] Ignoring source layer fc7_drop7_0_split
I0819 20:14:52.254992 14956 net.cpp:816] Ignoring source layer loss_cls
I0819 20:14:52.255066 14956 net.cpp:816] Ignoring source layer loss_bbox
I0819 20:14:52.258828 14956 net.cpp:816] Ignoring source layer silence_rpn_cls_score
I0819 20:14:52.259049 14956 net.cpp:816] Ignoring source layer silence_rpn_bbox_pred

Hi,

It’s recommended to upgrade your device to our JetPack4.4 product release (GA) software first.

The memory comes from the algorithm used by TensorRT when inferencing.
XavierNX has tensor core hardware so there are much more algorithm can be chosen in TensorRT.
Some algorithm will occupy more memory for a better performance.

If memory is critical for you, TensorRT also provide an API that can control the maximal memory usage.
https://docs.nvidia.com/deeplearning/tensorrt/api/c_api/classnvinfer1_1_1_i_builder_config.html#a2f49b62b833446b99d1d178208e476d5

/usr/src/tensorrt/bin/trtexec --workspace=2048 ....

Thanks.

Hi,

Thanks for your answer, I will try the TensorRT API to limit the maximal memory usage.

BR

@AastaLLL
I ran the trtexec to limit the workspace size, but I got a failed result:
sudo /usr/src/tensorrt/bin/trtexec --model=./model.caffemodel --deploy=./model.prototxt --output=bbox_pred --output=cls_prob --workspace=6144
&&&& RUNNING TensorRT.trtexec # /usr/src/tensorrt/bin/trtexec --model=./model.caffemodel --deploy=./model.prototxt --output=bbox_pred --output=cls_prob --workspace=6144
[08/23/2020-18:53:56] [I] === Model Options ===
[08/23/2020-18:53:56] [I] Format: Caffe
[08/23/2020-18:53:56] [I] Model: ./model.caffemodel
[08/23/2020-18:53:56] [I] Prototxt: ./model.prototxt
[08/23/2020-18:53:56] [I] Output: cls_prob bbox_pred
[08/23/2020-18:53:56] [I] === Build Options ===
[08/23/2020-18:53:56] [I] Max batch: 1
[08/23/2020-18:53:56] [I] Workspace: 6144 MB
[08/23/2020-18:53:56] [I] minTiming: 1
[08/23/2020-18:53:56] [I] avgTiming: 8
[08/23/2020-18:53:56] [I] Precision: FP32
[08/23/2020-18:53:56] [I] Calibration:
[08/23/2020-18:53:56] [I] Safe mode: Disabled
[08/23/2020-18:53:56] [I] Save engine:
[08/23/2020-18:53:56] [I] Load engine:
[08/23/2020-18:53:56] [I] Builder Cache: Enabled
[08/23/2020-18:53:56] [I] NVTX verbosity: 0
[08/23/2020-18:53:56] [I] Inputs format: fp32:CHW
[08/23/2020-18:53:56] [I] Outputs format: fp32:CHW
[08/23/2020-18:53:56] [I] Input build shapes: model
[08/23/2020-18:53:56] [I] Input calibration shapes: model
[08/23/2020-18:53:56] [I] === System Options ===
[08/23/2020-18:53:56] [I] Device: 0
[08/23/2020-18:53:56] [I] DLACore:
[08/23/2020-18:53:56] [I] Plugins:
[08/23/2020-18:53:56] [I] === Inference Options ===
[08/23/2020-18:53:56] [I] Batch: 1
[08/23/2020-18:53:56] [I] Input inference shapes: model
[08/23/2020-18:53:56] [I] Iterations: 10
[08/23/2020-18:53:56] [I] Duration: 3s (+ 200ms warm up)
[08/23/2020-18:53:56] [I] Sleep time: 0ms
[08/23/2020-18:53:56] [I] Streams: 1
[08/23/2020-18:53:56] [I] ExposeDMA: Disabled
[08/23/2020-18:53:56] [I] Spin-wait: Disabled
[08/23/2020-18:53:56] [I] Multithreading: Disabled
[08/23/2020-18:53:56] [I] CUDA Graph: Disabled
[08/23/2020-18:53:56] [I] Skip inference: Disabled
[08/23/2020-18:53:56] [I] Inputs:
[08/23/2020-18:53:56] [I] === Reporting Options ===
[08/23/2020-18:53:56] [I] Verbose: Disabled
[08/23/2020-18:53:56] [I] Averages: 10 inferences
[08/23/2020-18:53:56] [I] Percentile: 99
[08/23/2020-18:53:56] [I] Dump output: Disabled
[08/23/2020-18:53:56] [I] Profile: Disabled
[08/23/2020-18:53:56] [I] Export timing to JSON file:
[08/23/2020-18:53:56] [I] Export output to JSON file:
[08/23/2020-18:53:56] [I] Export profile to JSON file:
[08/23/2020-18:53:56] [I]
[libprotobuf FATAL /externals/protobuf/aarch64/10.0/include/google/protobuf/repeated_field.h:1078] CHECK failed: (index) < (current_size_):
terminate called after throwing an instance of ‘google_private::protobuf::FatalException’
what(): CHECK failed: (index) < (current_size_):
Aborted

I alsl ran this model on GTX 1050 ti, it has a very excellent memory consumption and speed performance!

Hi,

workspace=6144 is very large for XavierNX since it only have 8G memory.
Suppose you don’t need so much memory for a standard faster rcnn model.

If the issue goes on, would you mind to add some swap memory when compiling to see if helps.

Thanks.