ball segmentation crashes

ball segmentation on sdk2 crashes even after recent gpu upgrade to rtx 2080 super, nvidia-xorg updated

waiting for samples samples: 111
waiting for samples samples: 128
waiting for samples samples: 142
waiting for samples samples: 158
waiting for samples samples: 178
waiting for samples samples: 192
waiting for samples samples: 208
waiting for samples samples: 222
waiting for samples samples: 238
waiting for samples samples: 254
waiting for samples samples: 270
waiting for samples samples: 287
waiting for samples samples: 303
waiting for samples samples: 319
waiting for samples samples: 335
waiting for samples samples: 351
waiting for samples samples: 365
waiting for samples samples: 379
waiting for samples samples: 396
waiting for samples samples: 412
waiting for samples samples: 427
waiting for samples samples: 441
waiting for samples samples: 456
waiting for samples samples: 470
waiting for samples samples: 483
Starting training
2019-12-18 20:58:14.297092: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2019-12-18 20:58:15.339279: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-18 20:58:15.344088: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6a12c00 executing computations on platform CUDA. Devices:
2019-12-18 20:58:15.344142: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce RTX 2080 SUPER, Compute Capability 7.5
2019-12-18 20:58:15.541784: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3194400000 Hz
2019-12-18 20:58:15.542694: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6ac7150 executing computations on platform Host. Devices:
2019-12-18 20:58:15.542854: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-12-18 20:58:15.549201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.83
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 6.77GiB
2019-12-18 20:58:15.549689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 20:58:20.780846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 20:58:20.780912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-12-18 20:58:20.780947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-12-18 20:58:20.800837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
image: -1.0 1.0
label: 0.0 1.0
Start training from step: 0
2019-12-18 21:03:51.522104: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.437769: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613812: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613882: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630239: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630316: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.727920: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728028: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728272: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728451: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2019-12-18 21:04:25,488 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:25,635 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:35.703217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 21:04:36.817058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 21:04:36.817121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-12-18 21:04:36.817137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-12-18 21:04:36.897029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
2019-12-18 21:04:47,746 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2019-12-18 21:04:47,747 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
Saved frozen model at /tmp/ball_segmentation/ckpts/model-0-frozen.pb
Exiting due to training data stall
2019-12-18 21:05:46.934 INFO    engine/alice/application.cpp@219: Stopping application 'ball_segmentation_training' (instance UUID: 'a7619038-21d0-11ea-8e00-437676dba1c8') ...
2019-12-18 21:05:47.108 INFO    engine/alice/backend/backend.cpp@115: Backend is shutting down...
2019-12-18 21:05:47.141 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'actor_group_rigid_command_publisher'
2019-12-18 21:05:47.141 INFO    engine/alice/components/TcpPublisher.cpp@108: Stopping TCP publisher on port 5004
2019-12-18 21:05:47.141 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_teleportation'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'camera_teleportation'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_navigation_training_samples'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'tensor_synchronizer'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_encoder'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_visualizer'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_visualizer'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_encoder'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'sim_camera'
2019-12-18 21:05:47.248 INFO    engine/alice/components/TcpSubscriber.cpp@85: Stopping TCP receiver for (localhost:5003)...
2019-12-18 21:05:47.249 INFO    engine/alice/components/TcpSubscriber.cpp@102: Stopping TCP receiver for (localhost:5003)... DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_interactive_markers_bridge'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_pose_tree_bridge'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_statistics'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_statistics/NodeStatistics' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_statistics/NodeStatistics' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_config_bridge'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'websight'
2019-12-18 21:05:47.249 ERROR   engine/alice/components/TcpSubscriber.cpp@188: Socket failed to receive data. Will try to reconnect.
2019-12-18 21:05:47.330 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/isaac.sight.AliceSight' ...
2019-12-18 21:05:47.418 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/isaac.sight.AliceSight' DONE
2019-12-18 21:05:47.428 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/WebsightServer' ...
2019-12-18 21:05:47.630 INFO    packages/sight/WebsightServer.cpp@117: Disconnected
2019-12-18 21:05:48.133 WARNING engine/gems/scheduler/execution_groups.cpp@219: waitForJobDestruction: Job: Name 'NodeQueue' | Handle 'lu' is not stopping...
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@395: =====================================================================================
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@396: |                             Websight Statistics Report                            |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@397: =====================================================================================
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@398: | Name                                            | Bandwidth |    Total |    Count |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@399: -------------------------------------------------------------------------------------
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..ualizer/isaac.viewers.ColorCameraViewer/Color |      0.00 |    60.84 |     3660 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..saac.viewers.SegmentationCameraViewer/Classes |      0.00 |    58.30 |     3271 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..ac.viewers.SegmentationCameraViewer/Instances |      0.00 |    30.42 |     3271 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..aining_samples/isaac.ml.SampleAccumulator/sps |      0.00 |     0.20 |     3351 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..amples/isaac.ml.SampleAccumulator/buffer_size |      0.00 |     0.15 |     3351 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@418: =====================================================================================
2019-12-18 21:05:48.244 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/WebsightServer' DONE
2019-12-18 21:05:48.254 INFO    engine/alice/backend/asio_backend.cpp@49: Stopping ASIO service
2019-12-18 21:05:48.265 DEBUG   engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __BlockerGroup__...
2019-12-18 21:05:48.265 DEBUG   engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@547: |================================================================================================|
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@548: |                                 Job Statistics Report (blocking)                               |
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@549: |================================================================================================|
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@550: | Name                                               | Job Mode          |   #Run |     Avg Time |
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@551: |------------------------------------------------------------------------------------------------|
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@564: | connectToRemote                                    | Blocking One Shot |      1 |      1.13 ms |
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@564: | websight/WebsightServer                            | Blocking          |  91186 |      5.31 ms |
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@564: | ASIO Service                                       | Blocking          |      1 | 484771.98 ms |
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@564: | NodeQueue                                          | Blocking One Shot |      1 | 484761.78 ms |
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@566: |================================================================================================|
2019-12-18 21:05:48.358 DEBUG   engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __WorkerGroup__...
2019-12-18 21:05:48.358 DEBUG   engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@628: 
|=========================================================================================================================================================|
|                                                             Job Statistics Report (regular)                                                             |
|==============================================================================================================================waiting for samples samples: 111
waiting for samples samples: 128
waiting for samples samples: 142
waiting for samples samples: 158
waiting for samples samples: 178
waiting for samples samples: 192
waiting for samples samples: 208
waiting for samples samples: 222
waiting for samples samples: 238
waiting for samples samples: 254
waiting for samples samples: 270
waiting for samples samples: 287
waiting for samples samples: 303
waiting for samples samples: 319
waiting for samples samples: 335
waiting for samples samples: 351
waiting for samples samples: 365
waiting for samples samples: 379
waiting for samples samples: 396
waiting for samples samples: 412
waiting for samples samples: 427
waiting for samples samples: 441
waiting for samples samples: 456
waiting for samples samples: 470
waiting for samples samples: 483
Starting training
2019-12-18 20:58:14.297092: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2019-12-18 20:58:15.339279: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-18 20:58:15.344088: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6a12c00 executing computations on platform CUDA. Devices:
2019-12-18 20:58:15.344142: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): GeForce RTX 2080 SUPER, Compute Capability 7.5
2019-12-18 20:58:15.541784: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3194400000 Hz
2019-12-18 20:58:15.542694: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6ac7150 executing computations on platform Host. Devices:
2019-12-18 20:58:15.542854: I tensorflow/compiler/xla/service/service.cc:158]   StreamExecutor device (0): <undefined>, <undefined>
2019-12-18 20:58:15.549201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties: 
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.83
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 6.77GiB
2019-12-18 20:58:15.549689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 20:58:20.780846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 20:58:20.780912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-12-18 20:58:20.780947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-12-18 20:58:20.800837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
image: -1.0 1.0
label: 0.0 1.0
Start training from step: 0
2019-12-18 21:03:51.522104: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.437769: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613812: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613882: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630239: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630316: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.727920: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728028: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728272: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728451: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2019-12-18 21:04:25,488 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:25,635 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:35.703217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 21:04:36.817058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 21:04:36.817121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990]      0 
2019-12-18 21:04:36.817137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0:   N 
2019-12-18 21:04:36.897029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
2019-12-18 21:04:47,746 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2019-12-18 21:04:47,747 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
Saved frozen model at /tmp/ball_segmentation/ckpts/model-0-frozen.pb
Exiting due to training data stall
2019-12-18 21:05:46.934 INFO    engine/alice/application.cpp@219: Stopping application 'ball_segmentation_training' (instance UUID: 'a7619038-21d0-11ea-8e00-437676dba1c8') ...
2019-12-18 21:05:47.108 INFO    engine/alice/backend/backend.cpp@115: Backend is shutting down...
2019-12-18 21:05:47.141 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'actor_group_rigid_command_publisher'
2019-12-18 21:05:47.141 INFO    engine/alice/components/TcpPublisher.cpp@108: Stopping TCP publisher on port 5004
2019-12-18 21:05:47.141 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_teleportation'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'camera_teleportation'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_navigation_training_samples'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'tensor_synchronizer'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_encoder'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_visualizer'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_visualizer'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_encoder'
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' ...
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' DONE
2019-12-18 21:05:47.248 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'sim_camera'
2019-12-18 21:05:47.248 INFO    engine/alice/components/TcpSubscriber.cpp@85: Stopping TCP receiver for (localhost:5003)...
2019-12-18 21:05:47.249 INFO    engine/alice/components/TcpSubscriber.cpp@102: Stopping TCP receiver for (localhost:5003)... DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_interactive_markers_bridge'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_pose_tree_bridge'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_statistics'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_statistics/NodeStatistics' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_statistics/NodeStatistics' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node '_config_bridge'
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' ...
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' DONE
2019-12-18 21:05:47.249 DEBUG   engine/alice/backend/node_backend.cpp@342: Stopping node 'websight'
2019-12-18 21:05:47.249 ERROR   engine/alice/components/TcpSubscriber.cpp@188: Socket failed to receive data. Will try to reconnect.
2019-12-18 21:05:47.330 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/isaac.sight.AliceSight' ...
2019-12-18 21:05:47.418 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/isaac.sight.AliceSight' DONE
2019-12-18 21:05:47.428 DEBUG   engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/WebsightServer' ...
2019-12-18 21:05:47.630 INFO    packages/sight/WebsightServer.cpp@117: Disconnected
2019-12-18 21:05:48.133 WARNING engine/gems/scheduler/execution_groups.cpp@219: waitForJobDestruction: Job: Name 'NodeQueue' | Handle 'lu' is not stopping...
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@395: =====================================================================================
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@396: |                             Websight Statistics Report                            |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@397: =====================================================================================
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@398: | Name                                            | Bandwidth |    Total |    Count |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@399: -------------------------------------------------------------------------------------
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..ualizer/isaac.viewers.ColorCameraViewer/Color |      0.00 |    60.84 |     3660 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..saac.viewers.SegmentationCameraViewer/Classes |      0.00 |    58.30 |     3271 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..ac.viewers.SegmentationCameraViewer/Instances |      0.00 |    30.42 |     3271 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..aining_samples/isaac.ml.SampleAccumulator/sps |      0.00 |     0.20 |     3351 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@403: | ..amples/isaac.ml.SampleAccumulator/buffer_size |      0.00 |     0.15 |     3351 |
2019-12-18 21:05:48.244 INFO    packages/sight/WebsightServer.cpp@418: =====================================================================================
2019-12-18 21:05:48.244 DEBUG   engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/WebsightServer' DONE
2019-12-18 21:05:48.254 INFO    engine/alice/backend/asio_backend.cpp@49: Stopping ASIO service
2019-12-18 21:05:48.265 DEBUG   engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __BlockerGroup__...
2019-12-18 21:05:48.265 DEBUG   engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@547: |================================================================================================|
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@548: |                                 Job Statistics Report (blocking)                               |
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@549: |================================================================================================|
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@550: | Name                                               | Job Mode          |   #Run |     Avg Time |
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@551: |------------------------------------------------------------------------------------------------|
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@564: | connectToRemote                                    | Blocking One Shot |      1 |      1.13 ms |
2019-12-18 21:05:48.357 INFO    engine/gems/scheduler/execution_groups.cpp@564: | websight/WebsightServer                            | Blocking          |  91186 |      5.31 ms |
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@564: | ASIO Service                                       | Blocking          |      1 | 484771.98 ms |
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@564: | NodeQueue                                          | Blocking One Shot |      1 | 484761.78 ms |
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@566: |================================================================================================|
2019-12-18 21:05:48.358 DEBUG   engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __WorkerGroup__...
2019-12-18 21:05:48.358 DEBUG   engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.358 INFO    engine/gems/scheduler/execution_groups.cpp@628: 
|=========================================================================================================================================================|
|                                                             Job Statistics Report (regular)                                                             |
|=========================================================================================================================================================|
| Name                                               |   Job Mode |   Count | Time (Median - 90% - Max) [ms] | Rl Load | Overrun |   Overrun | Exec Delay |
|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| ..isualizer/isaac.viewers.SegmentationCameraViewer |      Event |    4719 |     0.02 |     4.58 |  1822.52 |  32.8 % |   0.0 % |   0.00 ms |  139.8 mus |
| ..amera_visualizer/isaac.viewers.ColorCameraViewer |      Event |    4718 |     1.69 |     2.03 |  1911.34 |  18.3 % |   0.0 % |   0.00 ms |   52.2 mus |
| ..or_camera_encoder/isaac.ml.ColorCameraEncoderCpu |      Event |    4718 |     1.52 |     2.11 |   378.97 |  16.4 % |   0.0 % |   0.00 ms | 27077.9 mus |
|        camera_teleportation/isaac.ml.Teleportation |   Periodic |   14400 |     0.18 |     0.26 |   449.10 |   8.8 % |   0.0 % | 166.04 ms | 104675.0 mus |
| tensor_synchronizer/isaac.ml.TensorSynchronization |      Event |    4718 |     0.73 |     0.90 |   469.22 |   8.4 % |   0.0 % |   0.00 ms | 247844.5 mus |
|  segmentation_encoder/isaac.ml.SegmentationEncoder |      Event |    4718 |     0.45 |     0.54 |   307.89 |   5.8 % |   0.0 % |   0.00 ms | 378595.3 mus |
| ..tion_training_samples/isaac.ml.SampleAccumulator |      Event |    3526 |     0.05 |     0.06 |  1452.20 |   4.5 % |   0.0 % |   0.00 ms | 3586.8 mus |
|          ball_teleportation/isaac.ml.Teleportation |      Event |   14399 |     0.07 |     0.17 |    16.62 |   3.0 % |   0.0 % |   0.00 ms | 94960.5 mus |
|                         _statistics/NodeStatistics |   Periodic |    1930 |     0.24 |     0.33 |     5.12 |   0.9 % |   0.0 % |   0.00 ms | 138636.3 mus |
|               _pose_tree_bridge/PoseTreeJsonBridge |   Periodic |    9617 |     0.04 |     0.05 |     8.22 |   0.7 % |   0.0 % |   0.00 ms | 105652.8 mus |
| ..eractive_markers_bridge/InteractiveMarkersBridge |   Periodic |    9617 |     0.01 |     0.02 |     0.56 |   0.2 % |   0.0 % |   0.00 ms | 105674.6 mus |
|                                  FailsafeHeartBeat |   Periodic |   48139 |     0.00 |     0.00 |     0.37 |   0.1 % |   0.0 % |   0.00 ms | 37960.8 mus |
|            _config_bridge/isaac.alice.ConfigBridge |      Event |       1 |     0.19 |     0.19 |     0.19 |   0.0 % |   0.0 % |   0.00 ms |   45.0 mus |
|=========================================================================================================================================================|
2019-12-18 21:05:48.359 INFO    engine/alice/backend/backend.cpp@127: Backend is shutting down... DONE
2019-12-18 21:05:48,893 DEBUG Python Codelets All stopped...
===========================|
| Name                                               |   Job Mode |   Count | Time (Median - 90% - Max) [ms] | Rl Load | Overrun |   Overrun | Exec Delay |
|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| ..isualizer/isaac.viewers.SegmentationCameraViewer |      Event |    4719 |     0.02 |     4.58 |  1822.52 |  32.8 % |   0.0 % |   0.00 ms |  139.8 mus |
| ..amera_visualizer/isaac.viewers.ColorCameraViewer |      Event |    4718 |     1.69 |     2.03 |  1911.34 |  18.3 % |   0.0 % |   0.00 ms |   52.2 mus |
| ..or_camera_encoder/isaac.ml.ColorCameraEncoderCpu |      Event |    4718 |     1.52 |     2.11 |   378.97 |  16.4 % |   0.0 % |   0.00 ms | 27077.9 mus |
|        camera_teleportation/isaac.ml.Teleportation |   Periodic |   14400 |     0.18 |     0.26 |   449.10 |   8.8 % |   0.0 % | 166.04 ms | 104675.0 mus |
| tensor_synchronizer/isaac.ml.TensorSynchronization |      Event |    4718 |     0.73 |     0.90 |   469.22 |   8.4 % |   0.0 % |   0.00 ms | 247844.5 mus |
|  segmentation_encoder/isaac.ml.SegmentationEncoder |      Event |    4718 |     0.45 |     0.54 |   307.89 |   5.8 % |   0.0 % |   0.00 ms | 378595.3 mus |
| ..tion_training_samples/isaac.ml.SampleAccumulator |      Event |    3526 |     0.05 |     0.06 |  1452.20 |   4.5 % |   0.0 % |   0.00 ms | 3586.8 mus |
|          ball_teleportation/isaac.ml.Teleportation |      Event |   14399 |     0.07 |     0.17 |    16.62 |   3.0 % |   0.0 % |   0.00 ms | 94960.5 mus |
|                         _statistics/NodeStatistics |   Periodic |    1930 |     0.24 |     0.33 |     5.12 |   0.9 % |   0.0 % |   0.00 ms | 138636.3 mus |
|               _pose_tree_bridge/PoseTreeJsonBridge |   Periodic |    9617 |     0.04 |     0.05 |     8.22 |   0.7 % |   0.0 % |   0.00 ms | 105652.8 mus |
| ..eractive_markers_bridge/InteractiveMarkersBridge |   Periodic |    9617 |     0.01 |     0.02 |     0.56 |   0.2 % |   0.0 % |   0.00 ms | 105674.6 mus |
|                                  FailsafeHeartBeat |   Periodic |   48139 |     0.00 |     0.00 |     0.37 |   0.1 % |   0.0 % |   0.00 ms | 37960.8 mus |
|            _config_bridge/isaac.alice.ConfigBridge |      Event |       1 |     0.19 |     0.19 |     0.19 |   0.0 % |   0.0 % |   0.00 ms |   45.0 mus |
|=========================================================================================================================================================|
2019-12-18 21:05:48.359 INFO    engine/alice/backend/backend.cpp@127: Backend is shutting down... DONE
2019-12-18 21:05:48,893 DEBUG Python Codelets All stopped...

Hi, n_lona,

Reading the error message, it seems that the training data is not being fed fast enough thus the training pipline is hitting tensorflow.errors.OutOfRangeError exception:

https://github.com/tensorflow/tensorflow/issues/24399

As we read allocation failure before OutOfRangeError, I am wondering if your system is under heavy load. Have you tried rebooting or reducing system load?

Another app that might be useful to experiment with segmentation is the example provided in packages/freespace_dnn/apps. The training app includes support for IsaacSim Unity3D, which is officially supported by the SDK.