ball segmentation on sdk2 crashes even after recent gpu upgrade to rtx 2080 super, nvidia-xorg updated
waiting for samples samples: 111
waiting for samples samples: 128
waiting for samples samples: 142
waiting for samples samples: 158
waiting for samples samples: 178
waiting for samples samples: 192
waiting for samples samples: 208
waiting for samples samples: 222
waiting for samples samples: 238
waiting for samples samples: 254
waiting for samples samples: 270
waiting for samples samples: 287
waiting for samples samples: 303
waiting for samples samples: 319
waiting for samples samples: 335
waiting for samples samples: 351
waiting for samples samples: 365
waiting for samples samples: 379
waiting for samples samples: 396
waiting for samples samples: 412
waiting for samples samples: 427
waiting for samples samples: 441
waiting for samples samples: 456
waiting for samples samples: 470
waiting for samples samples: 483
Starting training
2019-12-18 20:58:14.297092: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2019-12-18 20:58:15.339279: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-18 20:58:15.344088: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6a12c00 executing computations on platform CUDA. Devices:
2019-12-18 20:58:15.344142: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2080 SUPER, Compute Capability 7.5
2019-12-18 20:58:15.541784: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3194400000 Hz
2019-12-18 20:58:15.542694: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6ac7150 executing computations on platform Host. Devices:
2019-12-18 20:58:15.542854: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-12-18 20:58:15.549201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.83
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 6.77GiB
2019-12-18 20:58:15.549689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 20:58:20.780846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 20:58:20.780912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-12-18 20:58:20.780947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-12-18 20:58:20.800837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
image: -1.0 1.0
label: 0.0 1.0
Start training from step: 0
2019-12-18 21:03:51.522104: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.437769: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613812: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613882: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630239: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630316: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.727920: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728028: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728272: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728451: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2019-12-18 21:04:25,488 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:25,635 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:35.703217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 21:04:36.817058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 21:04:36.817121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-12-18 21:04:36.817137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-12-18 21:04:36.897029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
2019-12-18 21:04:47,746 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2019-12-18 21:04:47,747 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
Saved frozen model at /tmp/ball_segmentation/ckpts/model-0-frozen.pb
Exiting due to training data stall
2019-12-18 21:05:46.934 INFO engine/alice/application.cpp@219: Stopping application 'ball_segmentation_training' (instance UUID: 'a7619038-21d0-11ea-8e00-437676dba1c8') ...
2019-12-18 21:05:47.108 INFO engine/alice/backend/backend.cpp@115: Backend is shutting down...
2019-12-18 21:05:47.141 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'actor_group_rigid_command_publisher'
2019-12-18 21:05:47.141 INFO engine/alice/components/TcpPublisher.cpp@108: Stopping TCP publisher on port 5004
2019-12-18 21:05:47.141 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_teleportation'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'camera_teleportation'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_navigation_training_samples'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'tensor_synchronizer'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_encoder'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_visualizer'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_visualizer'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_encoder'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'sim_camera'
2019-12-18 21:05:47.248 INFO engine/alice/components/TcpSubscriber.cpp@85: Stopping TCP receiver for (localhost:5003)...
2019-12-18 21:05:47.249 INFO engine/alice/components/TcpSubscriber.cpp@102: Stopping TCP receiver for (localhost:5003)... DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_interactive_markers_bridge'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_pose_tree_bridge'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_statistics'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_statistics/NodeStatistics' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_statistics/NodeStatistics' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_config_bridge'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'websight'
2019-12-18 21:05:47.249 ERROR engine/alice/components/TcpSubscriber.cpp@188: Socket failed to receive data. Will try to reconnect.
2019-12-18 21:05:47.330 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/isaac.sight.AliceSight' ...
2019-12-18 21:05:47.418 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/isaac.sight.AliceSight' DONE
2019-12-18 21:05:47.428 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/WebsightServer' ...
2019-12-18 21:05:47.630 INFO packages/sight/WebsightServer.cpp@117: Disconnected
2019-12-18 21:05:48.133 WARNING engine/gems/scheduler/execution_groups.cpp@219: waitForJobDestruction: Job: Name 'NodeQueue' | Handle 'lu' is not stopping...
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@395: =====================================================================================
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@396: | Websight Statistics Report |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@397: =====================================================================================
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@398: | Name | Bandwidth | Total | Count |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@399: -------------------------------------------------------------------------------------
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..ualizer/isaac.viewers.ColorCameraViewer/Color | 0.00 | 60.84 | 3660 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..saac.viewers.SegmentationCameraViewer/Classes | 0.00 | 58.30 | 3271 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..ac.viewers.SegmentationCameraViewer/Instances | 0.00 | 30.42 | 3271 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..aining_samples/isaac.ml.SampleAccumulator/sps | 0.00 | 0.20 | 3351 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..amples/isaac.ml.SampleAccumulator/buffer_size | 0.00 | 0.15 | 3351 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@418: =====================================================================================
2019-12-18 21:05:48.244 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/WebsightServer' DONE
2019-12-18 21:05:48.254 INFO engine/alice/backend/asio_backend.cpp@49: Stopping ASIO service
2019-12-18 21:05:48.265 DEBUG engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __BlockerGroup__...
2019-12-18 21:05:48.265 DEBUG engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@547: |================================================================================================|
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@548: | Job Statistics Report (blocking) |
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@549: |================================================================================================|
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@550: | Name | Job Mode | #Run | Avg Time |
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@551: |------------------------------------------------------------------------------------------------|
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@564: | connectToRemote | Blocking One Shot | 1 | 1.13 ms |
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@564: | websight/WebsightServer | Blocking | 91186 | 5.31 ms |
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@564: | ASIO Service | Blocking | 1 | 484771.98 ms |
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@564: | NodeQueue | Blocking One Shot | 1 | 484761.78 ms |
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@566: |================================================================================================|
2019-12-18 21:05:48.358 DEBUG engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __WorkerGroup__...
2019-12-18 21:05:48.358 DEBUG engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@628:
|=========================================================================================================================================================|
| Job Statistics Report (regular) |
|==============================================================================================================================waiting for samples samples: 111
waiting for samples samples: 128
waiting for samples samples: 142
waiting for samples samples: 158
waiting for samples samples: 178
waiting for samples samples: 192
waiting for samples samples: 208
waiting for samples samples: 222
waiting for samples samples: 238
waiting for samples samples: 254
waiting for samples samples: 270
waiting for samples samples: 287
waiting for samples samples: 303
waiting for samples samples: 319
waiting for samples samples: 335
waiting for samples samples: 351
waiting for samples samples: 365
waiting for samples samples: 379
waiting for samples samples: 396
waiting for samples samples: 412
waiting for samples samples: 427
waiting for samples samples: 441
waiting for samples samples: 456
waiting for samples samples: 470
waiting for samples samples: 483
Starting training
2019-12-18 20:58:14.297092: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: FMA
2019-12-18 20:58:15.339279: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-12-18 20:58:15.344088: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6a12c00 executing computations on platform CUDA. Devices:
2019-12-18 20:58:15.344142: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2080 SUPER, Compute Capability 7.5
2019-12-18 20:58:15.541784: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3194400000 Hz
2019-12-18 20:58:15.542694: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x6ac7150 executing computations on platform Host. Devices:
2019-12-18 20:58:15.542854: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-12-18 20:58:15.549201: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2080 SUPER major: 7 minor: 5 memoryClockRate(GHz): 1.83
pciBusID: 0000:01:00.0
totalMemory: 7.79GiB freeMemory: 6.77GiB
2019-12-18 20:58:15.549689: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 20:58:20.780846: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 20:58:20.780912: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-12-18 20:58:20.780947: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-12-18 20:58:20.800837: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
image: -1.0 1.0
label: 0.0 1.0
Start training from step: 0
2019-12-18 21:03:51.522104: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.437769: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.12GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613812: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.613882: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.41GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630239: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.630316: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.25GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.727920: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728028: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728272: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2019-12-18 21:03:52.728451: W tensorflow/core/common_runtime/bfc_allocator.cc:211] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.69GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
2019-12-18 21:04:25,488 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:249: FastGFile.__init__ (from tensorflow.python.platform.gfile) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.gfile.GFile.
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:25,635 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:127: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
2019-12-18 21:04:35.703217: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-12-18 21:04:36.817058: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-12-18 21:04:36.817121: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-12-18 21:04:36.817137: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-12-18 21:04:36.897029: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 2633 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 SUPER, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
2019-12-18 21:04:47,746 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/tools/freeze_graph.py:232: convert_variables_to_constants (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.convert_variables_to_constants
WARNING:tensorflow:From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
2019-12-18 21:04:47,747 WARNING From /home/hp/.local/lib/python3.6/site-packages/tensorflow/python/framework/graph_util_impl.py:245: extract_sub_graph (from tensorflow.python.framework.graph_util_impl) is deprecated and will be removed in a future version.
Instructions for updating:
Use tf.compat.v1.graph_util.extract_sub_graph
Saved frozen model at /tmp/ball_segmentation/ckpts/model-0-frozen.pb
Exiting due to training data stall
2019-12-18 21:05:46.934 INFO engine/alice/application.cpp@219: Stopping application 'ball_segmentation_training' (instance UUID: 'a7619038-21d0-11ea-8e00-437676dba1c8') ...
2019-12-18 21:05:47.108 INFO engine/alice/backend/backend.cpp@115: Backend is shutting down...
2019-12-18 21:05:47.141 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'actor_group_rigid_command_publisher'
2019-12-18 21:05:47.141 INFO engine/alice/components/TcpPublisher.cpp@108: Stopping TCP publisher on port 5004
2019-12-18 21:05:47.141 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_teleportation'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'camera_teleportation'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'camera_teleportation/isaac.ml.Teleportation' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'ball_navigation_training_samples'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'ball_navigation_training_samples/isaac.ml.SampleAccumulator' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'tensor_synchronizer'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'tensor_synchronizer/isaac.ml.TensorSynchronization' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_encoder'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_encoder/isaac.ml.SegmentationEncoder' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'segmentation_visualizer'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'segmentation_visualizer/isaac.viewers.SegmentationCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_visualizer'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_visualizer/isaac.viewers.ColorCameraViewer' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'color_camera_encoder'
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' ...
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' DONE
2019-12-18 21:05:47.248 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'sim_camera'
2019-12-18 21:05:47.248 INFO engine/alice/components/TcpSubscriber.cpp@85: Stopping TCP receiver for (localhost:5003)...
2019-12-18 21:05:47.249 INFO engine/alice/components/TcpSubscriber.cpp@102: Stopping TCP receiver for (localhost:5003)... DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_interactive_markers_bridge'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_interactive_markers_bridge/InteractiveMarkersBridge' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_pose_tree_bridge'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_pose_tree_bridge/PoseTreeJsonBridge' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_statistics'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_statistics/NodeStatistics' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_statistics/NodeStatistics' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node '_config_bridge'
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' ...
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet '_config_bridge/isaac.alice.ConfigBridge' DONE
2019-12-18 21:05:47.249 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'websight'
2019-12-18 21:05:47.249 ERROR engine/alice/components/TcpSubscriber.cpp@188: Socket failed to receive data. Will try to reconnect.
2019-12-18 21:05:47.330 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/isaac.sight.AliceSight' ...
2019-12-18 21:05:47.418 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/isaac.sight.AliceSight' DONE
2019-12-18 21:05:47.428 DEBUG engine/alice/backend/codelet_backend.cpp@97: Stopping codelet 'websight/WebsightServer' ...
2019-12-18 21:05:47.630 INFO packages/sight/WebsightServer.cpp@117: Disconnected
2019-12-18 21:05:48.133 WARNING engine/gems/scheduler/execution_groups.cpp@219: waitForJobDestruction: Job: Name 'NodeQueue' | Handle 'lu' is not stopping...
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@395: =====================================================================================
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@396: | Websight Statistics Report |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@397: =====================================================================================
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@398: | Name | Bandwidth | Total | Count |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@399: -------------------------------------------------------------------------------------
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..ualizer/isaac.viewers.ColorCameraViewer/Color | 0.00 | 60.84 | 3660 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..saac.viewers.SegmentationCameraViewer/Classes | 0.00 | 58.30 | 3271 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..ac.viewers.SegmentationCameraViewer/Instances | 0.00 | 30.42 | 3271 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..aining_samples/isaac.ml.SampleAccumulator/sps | 0.00 | 0.20 | 3351 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@403: | ..amples/isaac.ml.SampleAccumulator/buffer_size | 0.00 | 0.15 | 3351 |
2019-12-18 21:05:48.244 INFO packages/sight/WebsightServer.cpp@418: =====================================================================================
2019-12-18 21:05:48.244 DEBUG engine/alice/backend/codelet_backend.cpp@107: Stopping codelet 'websight/WebsightServer' DONE
2019-12-18 21:05:48.254 INFO engine/alice/backend/asio_backend.cpp@49: Stopping ASIO service
2019-12-18 21:05:48.265 DEBUG engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __BlockerGroup__...
2019-12-18 21:05:48.265 DEBUG engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@547: |================================================================================================|
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@548: | Job Statistics Report (blocking) |
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@549: |================================================================================================|
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@550: | Name | Job Mode | #Run | Avg Time |
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@551: |------------------------------------------------------------------------------------------------|
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@564: | connectToRemote | Blocking One Shot | 1 | 1.13 ms |
2019-12-18 21:05:48.357 INFO engine/gems/scheduler/execution_groups.cpp@564: | websight/WebsightServer | Blocking | 91186 | 5.31 ms |
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@564: | ASIO Service | Blocking | 1 | 484771.98 ms |
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@564: | NodeQueue | Blocking One Shot | 1 | 484761.78 ms |
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@566: |================================================================================================|
2019-12-18 21:05:48.358 DEBUG engine/gems/scheduler/execution_groups.cpp@513: Stopping all threads for execution group __WorkerGroup__...
2019-12-18 21:05:48.358 DEBUG engine/gems/scheduler/execution_groups.cpp@517: Stopping all threads DONE
2019-12-18 21:05:48.358 INFO engine/gems/scheduler/execution_groups.cpp@628:
|=========================================================================================================================================================|
| Job Statistics Report (regular) |
|=========================================================================================================================================================|
| Name | Job Mode | Count | Time (Median - 90% - Max) [ms] | Rl Load | Overrun | Overrun | Exec Delay |
|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| ..isualizer/isaac.viewers.SegmentationCameraViewer | Event | 4719 | 0.02 | 4.58 | 1822.52 | 32.8 % | 0.0 % | 0.00 ms | 139.8 mus |
| ..amera_visualizer/isaac.viewers.ColorCameraViewer | Event | 4718 | 1.69 | 2.03 | 1911.34 | 18.3 % | 0.0 % | 0.00 ms | 52.2 mus |
| ..or_camera_encoder/isaac.ml.ColorCameraEncoderCpu | Event | 4718 | 1.52 | 2.11 | 378.97 | 16.4 % | 0.0 % | 0.00 ms | 27077.9 mus |
| camera_teleportation/isaac.ml.Teleportation | Periodic | 14400 | 0.18 | 0.26 | 449.10 | 8.8 % | 0.0 % | 166.04 ms | 104675.0 mus |
| tensor_synchronizer/isaac.ml.TensorSynchronization | Event | 4718 | 0.73 | 0.90 | 469.22 | 8.4 % | 0.0 % | 0.00 ms | 247844.5 mus |
| segmentation_encoder/isaac.ml.SegmentationEncoder | Event | 4718 | 0.45 | 0.54 | 307.89 | 5.8 % | 0.0 % | 0.00 ms | 378595.3 mus |
| ..tion_training_samples/isaac.ml.SampleAccumulator | Event | 3526 | 0.05 | 0.06 | 1452.20 | 4.5 % | 0.0 % | 0.00 ms | 3586.8 mus |
| ball_teleportation/isaac.ml.Teleportation | Event | 14399 | 0.07 | 0.17 | 16.62 | 3.0 % | 0.0 % | 0.00 ms | 94960.5 mus |
| _statistics/NodeStatistics | Periodic | 1930 | 0.24 | 0.33 | 5.12 | 0.9 % | 0.0 % | 0.00 ms | 138636.3 mus |
| _pose_tree_bridge/PoseTreeJsonBridge | Periodic | 9617 | 0.04 | 0.05 | 8.22 | 0.7 % | 0.0 % | 0.00 ms | 105652.8 mus |
| ..eractive_markers_bridge/InteractiveMarkersBridge | Periodic | 9617 | 0.01 | 0.02 | 0.56 | 0.2 % | 0.0 % | 0.00 ms | 105674.6 mus |
| FailsafeHeartBeat | Periodic | 48139 | 0.00 | 0.00 | 0.37 | 0.1 % | 0.0 % | 0.00 ms | 37960.8 mus |
| _config_bridge/isaac.alice.ConfigBridge | Event | 1 | 0.19 | 0.19 | 0.19 | 0.0 % | 0.0 % | 0.00 ms | 45.0 mus |
|=========================================================================================================================================================|
2019-12-18 21:05:48.359 INFO engine/alice/backend/backend.cpp@127: Backend is shutting down... DONE
2019-12-18 21:05:48,893 DEBUG Python Codelets All stopped...
===========================|
| Name | Job Mode | Count | Time (Median - 90% - Max) [ms] | Rl Load | Overrun | Overrun | Exec Delay |
|---------------------------------------------------------------------------------------------------------------------------------------------------------|
| ..isualizer/isaac.viewers.SegmentationCameraViewer | Event | 4719 | 0.02 | 4.58 | 1822.52 | 32.8 % | 0.0 % | 0.00 ms | 139.8 mus |
| ..amera_visualizer/isaac.viewers.ColorCameraViewer | Event | 4718 | 1.69 | 2.03 | 1911.34 | 18.3 % | 0.0 % | 0.00 ms | 52.2 mus |
| ..or_camera_encoder/isaac.ml.ColorCameraEncoderCpu | Event | 4718 | 1.52 | 2.11 | 378.97 | 16.4 % | 0.0 % | 0.00 ms | 27077.9 mus |
| camera_teleportation/isaac.ml.Teleportation | Periodic | 14400 | 0.18 | 0.26 | 449.10 | 8.8 % | 0.0 % | 166.04 ms | 104675.0 mus |
| tensor_synchronizer/isaac.ml.TensorSynchronization | Event | 4718 | 0.73 | 0.90 | 469.22 | 8.4 % | 0.0 % | 0.00 ms | 247844.5 mus |
| segmentation_encoder/isaac.ml.SegmentationEncoder | Event | 4718 | 0.45 | 0.54 | 307.89 | 5.8 % | 0.0 % | 0.00 ms | 378595.3 mus |
| ..tion_training_samples/isaac.ml.SampleAccumulator | Event | 3526 | 0.05 | 0.06 | 1452.20 | 4.5 % | 0.0 % | 0.00 ms | 3586.8 mus |
| ball_teleportation/isaac.ml.Teleportation | Event | 14399 | 0.07 | 0.17 | 16.62 | 3.0 % | 0.0 % | 0.00 ms | 94960.5 mus |
| _statistics/NodeStatistics | Periodic | 1930 | 0.24 | 0.33 | 5.12 | 0.9 % | 0.0 % | 0.00 ms | 138636.3 mus |
| _pose_tree_bridge/PoseTreeJsonBridge | Periodic | 9617 | 0.04 | 0.05 | 8.22 | 0.7 % | 0.0 % | 0.00 ms | 105652.8 mus |
| ..eractive_markers_bridge/InteractiveMarkersBridge | Periodic | 9617 | 0.01 | 0.02 | 0.56 | 0.2 % | 0.0 % | 0.00 ms | 105674.6 mus |
| FailsafeHeartBeat | Periodic | 48139 | 0.00 | 0.00 | 0.37 | 0.1 % | 0.0 % | 0.00 ms | 37960.8 mus |
| _config_bridge/isaac.alice.ConfigBridge | Event | 1 | 0.19 | 0.19 | 0.19 | 0.0 % | 0.0 % | 0.00 ms | 45.0 mus |
|=========================================================================================================================================================|
2019-12-18 21:05:48.359 INFO engine/alice/backend/backend.cpp@127: Backend is shutting down... DONE
2019-12-18 21:05:48,893 DEBUG Python Codelets All stopped...