I got error (ResourceExhaustedError: OOM when allocating tensor of shape [128]) when running this program Human-Pose-Estimation-Benchmarking-and-Action-Recognition/run_detector.py at master · ChengeYang/Human-Pose-Estimation-Benchmarking-and-Action-Recognition · GitHub on jetson nano 4GB. Please help me to solve this problem.
This is description of error :
2021-09-23 10:33:16.733972: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/mobilenet/mobilenet.py:369: The name tf.nn.avg_pool is deprecated. Please use tf.nn.avg_pool2d instead.
[2021-09-23 10:33:32,853] [TfPoseEstimator] [INFO] loading graph from /home/jetson/action-rec/src/githubs/tf-pose-estimation/models/graph/cmu/graph_opt.pb(default size=432x368)
[2021-09-23 10:33:32,853] [TfPoseEstimator] [INFO] loading graph from /home/jetson/action-rec/src/githubs/tf-pose-estimation/models/graph/cmu/graph_opt.pb(default size=432x368)
2021-09-23 10:33:32,853 INFO loading graph from /home/jetson/action-rec/src/githubs/tf-pose-estimation/models/graph/cmu/graph_opt.pb(default size=432x368)
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:310: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
2021-09-23 10:33:32,853 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:310: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:311: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
2021-09-23 10:33:32,854 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:311: The name tf.GraphDef is deprecated. Please use tf.compat.v1.GraphDef instead.
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:314: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
2021-09-23 10:33:36,292 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:314: The name tf.get_default_graph is deprecated. Please use tf.compat.v1.get_default_graph instead.
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:316: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-09-23 10:33:40,524 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:316: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2021-09-23 10:33:40.757522: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcuda.so.1
2021-09-23 10:33:40.956802: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:40.957052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-09-23 10:33:40.957214: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-09-23 10:33:41.077869: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2021-09-23 10:33:41.204955: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2021-09-23 10:33:41.368657: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2021-09-23 10:33:41.857237: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2021-09-23 10:33:42.029029: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2021-09-23 10:33:42.438350: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-09-23 10:33:42.438864: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:42.439309: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:42.439417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2021-09-23 10:33:42.637044: W tensorflow/core/platform/profile_utils/cpu_utils.cc:98] Failed to find bogomips in /proc/cpuinfo; cannot determine CPU frequency
2021-09-23 10:33:42.637666: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c728510 executing computations on platform Host. Devices:
2021-09-23 10:33:42.637737: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): ,
2021-09-23 10:33:42.797954: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:42.798276: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x3c374bf0 executing computations on platform CUDA. Devices:
2021-09-23 10:33:42.798339: I tensorflow/compiler/xla/service/service.cc:175] StreamExecutor device (0): NVIDIA Tegra X1, Compute Capability 5.3
2021-09-23 10:33:42.799136: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:42.799277: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: NVIDIA Tegra X1 major: 5 minor: 3 memoryClockRate(GHz): 0.9216
pciBusID: 0000:00:00.0
2021-09-23 10:33:42.799368: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-09-23 10:33:42.799602: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcublas.so.10.0
2021-09-23 10:33:42.799751: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcufft.so.10.0
2021-09-23 10:33:42.799876: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcurand.so.10.0
2021-09-23 10:33:42.799998: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusolver.so.10.0
2021-09-23 10:33:42.800116: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcusparse.so.10.0
2021-09-23 10:33:42.800222: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudnn.so.7
2021-09-23 10:33:42.800560: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:42.800920: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:42.801019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2021-09-23 10:33:42.824342: I tensorflow/stream_executor/platform/default/dso_loader.cc:42] Successfully opened dynamic library libcudart.so.10.0
2021-09-23 10:33:54.675214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2021-09-23 10:33:54.675324: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187] 0
2021-09-23 10:33:54.675363: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0: N
2021-09-23 10:33:54.697677: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:54.698208: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:972] ARM64 does not support NUMA - returning NUMA node zero
2021-09-23 10:33:54.698431: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 73 MB memory) → physical GPU (device: 0, name: NVIDIA Tegra X1, pci bus id: 0000:00:00.0, compute capability: 5.3)
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:327: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
2021-09-23 10:33:54,765 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:327: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:328: The name tf.image.resize_area is deprecated. Please use tf.compat.v1.image.resize_area instead.
2021-09-23 10:33:54,770 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:328: The name tf.image.resize_area is deprecated. Please use tf.compat.v1.image.resize_area instead.
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/tensblur/smoother.py:92: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
2021-09-23 10:33:55,002 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/tensblur/smoother.py:92: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.
WARNING:tensorflow:From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:337: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2021-09-23 10:33:55,110 WARNING From /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:337: add_dispatch_support..wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.where in 2.0, which has the same broadcast rule as np.where
2021-09-23 10:33:57.924403: W tensorflow/compiler/jit/mark_for_compilation_pass.cc:1412] (One-time warning): Not using XLA:CPU for cluster because envvar TF_XLA_FLAGS=–tf_xla_cpu_global_jit was not set. If you want XLA:CPU, either set that envvar, or use experimental_jit_scope to enable XLA:CPU. To confirm that XLA is active, pass --vmodule=xla_compilation_cache=1 (as a proper command-line flag, not via TF_XLA_FLAGS) or set the envvar XLA_FLAGS=–xla_hlo_profile.
2021-09-23 10:34:18.251760: W tensorflow/core/framework/allocator.cc:107] Allocation of 9437184 exceeds 10% of system memory.
2021-09-23 10:35:01.590871: W tensorflow/core/framework/allocator.cc:107] Allocation of 9437184 exceeds 10% of system memory.
2021-09-23 10:35:06.153590: W tensorflow/core/framework/allocator.cc:107] Allocation of 9437184 exceeds 10% of system memory.
2021-09-23 10:35:13.848946: W tensorflow/core/framework/allocator.cc:107] Allocation of 9437184 exceeds 10% of system memory.
2021-09-23 10:35:18.693541: W tensorflow/core/framework/allocator.cc:107] Allocation of 9437184 exceeds 10% of system memory.
2021-09-23 10:41:39.576674: W tensorflow/core/common_runtime/bfc_allocator.cc:314] Allocator (GPU_0_bfc) ran out of memory trying to allocate 512B (rounded to 512). Current allocation summary follows.
2021-09-23 10:41:40.008797: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (256): Total Chunks: 5, Chunks in use: 5. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 532B client-requested in use in bin.
2021-09-23 10:41:40.031799: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (512): Total Chunks: 27, Chunks in use: 27. 13.5KiB allocated for chunks. 13.5KiB in use in bin. 13.5KiB client-requested in use in bin.
2021-09-23 10:41:40.031879: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1024): Total Chunks: 1, Chunks in use: 1. 1.2KiB allocated for chunks. 1.2KiB in use in bin. 1.0KiB client-requested in use in bin.
2021-09-23 10:41:40.031909: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2048): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.031937: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4096): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.031962: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8192): Total Chunks: 3, Chunks in use: 3. 28.5KiB allocated for chunks. 28.5KiB in use in bin. 28.5KiB client-requested in use in bin.
2021-09-23 10:41:40.031987: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16384): Total Chunks: 2, Chunks in use: 2. 38.0KiB allocated for chunks. 38.0KiB in use in bin. 38.0KiB client-requested in use in bin.
2021-09-23 10:41:40.032016: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (32768): Total Chunks: 2, Chunks in use: 2. 93.0KiB allocated for chunks. 93.0KiB in use in bin. 92.8KiB client-requested in use in bin.
2021-09-23 10:41:40.032042: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (65536): Total Chunks: 5, Chunks in use: 5. 320.0KiB allocated for chunks. 320.0KiB in use in bin. 320.0KiB client-requested in use in bin.
2021-09-23 10:41:40.032530: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (131072): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032570: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (262144): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032684: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (524288): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032710: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (1048576): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032742: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (2097152): Total Chunks: 18, Chunks in use: 18. 55.24MiB allocated for chunks. 55.24MiB in use in bin. 55.12MiB client-requested in use in bin.
2021-09-23 10:41:40.032772: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (4194304): Total Chunks: 4, Chunks in use: 4. 17.71MiB allocated for chunks. 17.71MiB in use in bin. 17.71MiB client-requested in use in bin.
2021-09-23 10:41:40.032795: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (8388608): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032829: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (16777216): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032857: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (33554432): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032884: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032911: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (134217728): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032937: I tensorflow/core/common_runtime/bfc_allocator.cc:764] Bin (268435456): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2021-09-23 10:41:40.032992: I tensorflow/core/common_runtime/bfc_allocator.cc:780] Bin for 512B was 512B, Chunk State:
2021-09-23 10:41:40.033018: I tensorflow/core/common_runtime/bfc_allocator.cc:793] Next region of size 77000704
2021-09-23 10:41:40.033147: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850000 next 1 of size 1280
2021-09-23 10:41:40.033183: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00850500 next 2 of size 47616
2021-09-23 10:41:40.033213: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0085bf00 next 3 of size 47616
2021-09-23 10:41:40.033242: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00867900 next 4 of size 256
2021-09-23 10:41:40.033273: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00867a00 next 5 of size 9728
2021-09-23 10:41:40.033304: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0086a000 next 6 of size 512
2021-09-23 10:41:40.033335: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0086a200 next 7 of size 65536
2021-09-23 10:41:40.033366: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0087a200 next 8 of size 512
2021-09-23 10:41:40.033397: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0087a400 next 9 of size 3211264
2021-09-23 10:41:40.033428: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00b8a400 next 10 of size 512
2021-09-23 10:41:40.033460: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00b8a600 next 11 of size 3211264
2021-09-23 10:41:40.033491: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00e9a600 next 12 of size 512
2021-09-23 10:41:40.033522: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf00e9a800 next 13 of size 3211264
2021-09-23 10:41:40.033553: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf011aa800 next 14 of size 512
2021-09-23 10:41:40.033585: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf011aaa00 next 15 of size 3211264
2021-09-23 10:41:40.033616: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf014baa00 next 16 of size 512
2021-09-23 10:41:40.033648: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf014bac00 next 17 of size 4641280
2021-09-23 10:41:40.033680: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01927e00 next 18 of size 256
2021-09-23 10:41:40.033712: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01927f00 next 19 of size 19456
2021-09-23 10:41:40.033744: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0192cb00 next 20 of size 512
2021-09-23 10:41:40.033775: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0192cd00 next 21 of size 65536
2021-09-23 10:41:40.033806: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0193cd00 next 22 of size 512
2021-09-23 10:41:40.033837: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0193cf00 next 23 of size 3211264
2021-09-23 10:41:40.033868: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01c4cf00 next 24 of size 512
2021-09-23 10:41:40.033899: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01c4d100 next 25 of size 3211264
2021-09-23 10:41:40.033930: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01f5d100 next 26 of size 512
2021-09-23 10:41:40.033962: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf01f5d300 next 27 of size 3211264
2021-09-23 10:41:40.033994: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0226d300 next 28 of size 512
2021-09-23 10:41:40.034026: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0226d500 next 29 of size 3211264
2021-09-23 10:41:40.034058: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0257d500 next 30 of size 512
2021-09-23 10:41:40.034090: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0257d700 next 31 of size 4641280
2021-09-23 10:41:40.034121: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf029ea900 next 32 of size 256
2021-09-23 10:41:40.034153: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf029eaa00 next 33 of size 9728
2021-09-23 10:41:40.034186: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf029ed000 next 34 of size 512
2021-09-23 10:41:40.034219: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf029ed200 next 35 of size 65536
2021-09-23 10:41:40.034252: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf029fd200 next 36 of size 512
2021-09-23 10:41:40.034283: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf029fd400 next 37 of size 3211264
2021-09-23 10:41:40.034314: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02d0d400 next 38 of size 512
2021-09-23 10:41:40.034346: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf02d0d600 next 39 of size 3211264
2021-09-23 10:41:40.034378: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0301d600 next 40 of size 512
2021-09-23 10:41:40.034411: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0301d800 next 41 of size 3211264
2021-09-23 10:41:40.034442: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0332d800 next 42 of size 512
2021-09-23 10:41:40.034474: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0332da00 next 43 of size 3211264
2021-09-23 10:41:40.034505: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0363da00 next 44 of size 512
2021-09-23 10:41:40.034536: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf0363dc00 next 45 of size 4641280
2021-09-23 10:41:40.034568: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03aaae00 next 46 of size 256
2021-09-23 10:41:40.034601: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03aaaf00 next 47 of size 19456
2021-09-23 10:41:40.034633: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03aafb00 next 48 of size 512
2021-09-23 10:41:40.034664: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03aafd00 next 49 of size 65536
2021-09-23 10:41:40.034695: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03abfd00 next 50 of size 512
2021-09-23 10:41:40.034728: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03abff00 next 51 of size 3211264
2021-09-23 10:41:40.034759: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03dcff00 next 52 of size 512
2021-09-23 10:41:40.034790: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf03dd0100 next 53 of size 3211264
2021-09-23 10:41:40.034821: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf040e0100 next 54 of size 512
2021-09-23 10:41:40.034854: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf040e0300 next 55 of size 3211264
2021-09-23 10:41:40.034886: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf043f0300 next 56 of size 512
2021-09-23 10:41:40.034918: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf043f0500 next 57 of size 3211264
2021-09-23 10:41:40.034949: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04700500 next 58 of size 512
2021-09-23 10:41:40.034980: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04700700 next 59 of size 4641280
2021-09-23 10:41:40.035011: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04b6d900 next 60 of size 256
2021-09-23 10:41:40.035044: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04b6da00 next 61 of size 9728
2021-09-23 10:41:40.035117: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04b70000 next 62 of size 512
2021-09-23 10:41:40.035155: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04b70200 next 63 of size 65536
2021-09-23 10:41:40.035187: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04b80200 next 64 of size 512
2021-09-23 10:41:40.035220: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04b80400 next 65 of size 3211264
2021-09-23 10:41:40.035252: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04e90400 next 66 of size 512
2021-09-23 10:41:40.035287: I tensorflow/core/common_runtime/bfc_allocator.cc:800] InUse at 0xf04e90600 next 18446744073709551615 of size 3336704
2021-09-23 10:41:40.035318: I tensorflow/core/common_runtime/bfc_allocator.cc:809] Summary of in-use Chunks by size:
2021-09-23 10:41:40.035357: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 256 totalling 1.2KiB
2021-09-23 10:41:40.035394: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 27 Chunks of size 512 totalling 13.5KiB
2021-09-23 10:41:40.035429: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 1280 totalling 1.2KiB
2021-09-23 10:41:40.035464: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 3 Chunks of size 9728 totalling 28.5KiB
2021-09-23 10:41:40.035498: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 19456 totalling 38.0KiB
2021-09-23 10:41:40.035536: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 2 Chunks of size 47616 totalling 93.0KiB
2021-09-23 10:41:40.035572: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 5 Chunks of size 65536 totalling 320.0KiB
2021-09-23 10:41:40.035607: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 17 Chunks of size 3211264 totalling 52.06MiB
2021-09-23 10:41:40.035640: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 1 Chunks of size 3336704 totalling 3.18MiB
2021-09-23 10:41:40.035676: I tensorflow/core/common_runtime/bfc_allocator.cc:812] 4 Chunks of size 4641280 totalling 17.71MiB
2021-09-23 10:41:40.035710: I tensorflow/core/common_runtime/bfc_allocator.cc:816] Sum Total of in-use chunks: 73.43MiB
2021-09-23 10:41:40.035743: I tensorflow/core/common_runtime/bfc_allocator.cc:818] total_region_allocated_bytes_: 77000704 memory_limit_: 77000704 available bytes: 0 curr_region_allocation_bytes_: 154001408
2021-09-23 10:41:40.035815: I tensorflow/core/common_runtime/bfc_allocator.cc:824] Stats:
Limit: 77000704
InUse: 77000704
MaxInUse: 77000704
NumAllocs: 67
MaxAllocSize: 4641280
2021-09-23 10:41:40.035869: W tensorflow/core/common_runtime/bfc_allocator.cc:319] ****************************************************************************************************
2021-09-23 10:41:40.264204: W tensorflow/core/framework/op_kernel.cc:1479] OP_REQUIRES failed at constant_op.cc:77 : Resource exhausted: OOM when allocating tensor of shape [128] and type float
2021-09-23 10:41:40.582526: E tensorflow/core/common_runtime/executor.cc:648] Executor failed to create kernel. Resource exhausted: OOM when allocating tensor of shape [128] and type float
[[{{node TfPoseEstimator/Mconv3_stage4_L2/biases}}]]
Traceback (most recent call last):
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1356, in _do_call
return fn(*args)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [128] and type float
[[{{node TfPoseEstimator/Mconv3_stage4_L2/biases}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “run_detector.py”, line 270, in
my_detector = SkeletonDetector()
File “run_detector.py”, line 140, in init
e = TfPoseEstimator(get_graph_path(self.model), target_size=(w, h))
File “/home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py”, line 352, in init
self.upsample_size: [target_size[1], target_size[0]]
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 950, in run
run_metadata_ptr)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1173, in _run
feed_dict_tensor, options, run_metadata)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1350, in _do_run
run_metadata)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/client/session.py”, line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor of shape [128] and type float
[[node TfPoseEstimator/Mconv3_stage4_L2/biases (defined at /home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py:315) ]]
Original stack trace for ‘TfPoseEstimator/Mconv3_stage4_L2/biases’:
File “run_detector.py”, line 270, in
my_detector = SkeletonDetector()
File “run_detector.py”, line 140, in init
e = TfPoseEstimator(get_graph_path(self.model), target_size=(w, h))
File “/home/jetson/action-rec/src/githubs/tf-pose-estimation/tf_pose/estimator.py”, line 315, in init
tf.import_graph_def(graph_def, name=‘TfPoseEstimator’)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py”, line 507, in new_func
return func(*args, **kwargs)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/framework/importer.py”, line 443, in import_graph_def
_ProcessNewOps(graph)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/framework/importer.py”, line 236, in _ProcessNewOps
for new_op in graph._add_new_tf_operations(compute_devices=False): # pylint: disable=protected-access
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 3751, in _add_new_tf_operations
for c_op in c_api_util.new_tf_operations(self)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 3751, in
for c_op in c_api_util.new_tf_operations(self)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 3641, in _create_op_from_tf_operation
ret = Operation(c_op, self)
File “/home/jetson/py3cv4/lib/python3.6/site-packages/tensorflow/python/framework/ops.py”, line 2005, in init
self._traceback = tf_stack.extract_stack()
Thank you