Hello, I am trying to train the yolo model with custom objects to recognize different things (like cones, etc.) But before I do that, I am just trying to run the basic yolo_training app with the sim and it is giving me some vague errors.
I start the sim before I launch the app, as well as the carter_rgb_detection.json is updated with the absolute path to it’s corresponding config and graph files.
shatterhand@shatterhand-PRIME-Z390-A:~$ bazel run //apps/samples/yolo:yolo_training
WARNING: ignoring LD_PRELOAD in environment.
ERROR: The 'run' command is only supported from within a workspace.
WARNING: --batch mode is deprecated. Please instead explicitly shut down your Bazel server using the command "bazel shutdown".
shatterhand@shatterhand-PRIME-Z390-A:~$ cd Projects/Navon/isaac/
shatterhand@shatterhand-PRIME-Z390-A:~/Projects/Navon/isaac$ bazel run //apps/samples/yolo:yolo_training
WARNING: /home/shatterhand/Projects/Navon/isaac/packages/viewers/BUILD:14:12: in srcs attribute of cc_binary rule //packages/viewers:libviewers_module.so: please do not import '//packages_x86_64/viewers:libviewers_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
WARNING: /home/shatterhand/Projects/Navon/isaac/packages/perception/BUILD:34:12: in srcs attribute of cc_binary rule //packages/perception:libperception_module.so: please do not import '//packages_x86_64/perception:libperception_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
WARNING: /home/shatterhand/Projects/Navon/isaac/packages/ml/BUILD:15:12: in srcs attribute of cc_binary rule //packages/ml:libml_module.so: please do not import '//packages_x86_64/ml:libml_module.so' directly. You should either move the file to this package or depend on an appropriate rule there
INFO: Analysed target //apps/samples/yolo:yolo_training (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //apps/samples/yolo:yolo_training up-to-date:
bazel-bin/apps/samples/yolo/yolo_training
INFO: Elapsed time: 0.217s, Critical Path: 0.00s, Remote (0.00% of the time): [queue: 0.00%, setup: 0.00%, process: 0.00%]
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Using TensorFlow backend.
/home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_quint16 = np.dtype([("quint16", np.uint16, 1)])
/home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
_np_qint32 = np.dtype([("qint32", np.int32, 1)])
/home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
np_resource = np.dtype([("resource", np.ubyte, 1)])
2019-11-26 15:04:16.078 INFO engine/alice/tools/websight.cpp@166: Loading websight...
2019-11-26 15:04:16.078 WARNING engine/alice/backend/application_json_loader.cpp@98: This application does not have an explicit scheduler configuration. One will be autogenerated to the best of the system's abilities if possible.
2019-11-26 15:04:16.078 WARNING engine/alice/backend/application_json_loader.cpp@241: Loaded a subgraph with no modules section. Did you forget to add one?
2019-11-26 15:04:16.078 INFO engine/alice/backend/application_json_loader.cpp@237: Modules requested by subgraph 'yolo_training': ml, perception, viewers,
2019-11-26 15:04:16.078 WARNING engine/alice/backend/backend.cpp@168: This application does not have an execution group configuration. One will be autogenerated to the best of the systems abilities if possible.
2019-11-26 15:04:16.078 WARNING engine/gems/scheduler/scheduler.cpp@337: No default execution groups specified. Attempting to create scheduler configuration for 12 remaining cores. This may be non optimal for the system and application.
2019-11-26 15:04:16.078 INFO engine/gems/scheduler/scheduler.cpp@290: Scheduler execution groups are:
2019-11-26 15:04:16.078 INFO engine/gems/scheduler/scheduler.cpp@299: __BlockerGroup__: Cores = [9, 10, 11], Workers = No
2019-11-26 15:04:16.078 INFO engine/gems/scheduler/scheduler.cpp@299: __WorkerGroup__: Cores = [0, 1, 2, 3, 4, 5, 6, 7, 8], Workers = Yes
2019-11-26 15:04:16.088 INFO engine/alice/backend/modules.cpp@283: Loaded module 'packages/ml/libml_module.so': Now has 56 components total
2019-11-26 15:04:16.093 INFO engine/alice/backend/modules.cpp@283: Loaded module 'packages/perception/libperception_module.so': Now has 66 components total
2019-11-26 15:04:16.097 INFO engine/alice/backend/modules.cpp@283: Loaded module 'packages/sight/libsight_module.so': Now has 67 components total
2019-11-26 15:04:16.103 INFO engine/alice/backend/modules.cpp@283: Loaded module 'packages/viewers/libviewers_module.so': Now has 92 components total
2019-11-26 15:04:16.103 INFO engine/alice/application.cpp@265: Loaded 92 components: isaac::alice::Behavior, isaac::alice::ChannelMonitor, isaac::alice::Config, isaac::alice::ConfigBridge, isaac::alice::Failsafe, isaac::alice::FailsafeHeartbeat, isaac::alice::InteractiveMarkersBridge, isaac::alice::MessageLedger, isaac::alice::NodeStatistics, isaac::alice::Pose, isaac::alice::PoseInitializer, isaac::alice::PoseMessageInjector, isaac::alice::PoseTreeJsonBridge, isaac::alice::PyCodelet, isaac::alice::Recorder, isaac::alice::RecorderBridge, isaac::alice::Replay, isaac::alice::ReplayBridge, isaac::alice::Scheduling, isaac::alice::Sight, isaac::alice::SightChannelStatus, isaac::alice::Subgraph, isaac::alice::Subprocess, isaac::alice::TcpPublisher, isaac::alice::TcpSubscriber, isaac::alice::Throttle, isaac::alice::TimeOffset, isaac::alice::TimeSynchronizer, isaac::alice::UdpPublisher, isaac::alice::UdpSubscriber, isaac::alice::behaviors::Behavior, isaac::alice::behaviors::MemorySelectorBehavior, isaac::alice::behaviors::MemorySequenceBehavior, isaac::alice::behaviors::NodeGroup, isaac::alice::behaviors::ParallelBehavior, isaac::alice::behaviors::SwitchBehavior, isaac::alice::behaviors::TimerBehavior, isaac::map::Map, isaac::map::OccupancyGridMapLayer, isaac::map::PolygonMapLayer, isaac::map::WaypointMapLayer, isaac::ml::BinarizePredictions, isaac::ml::ColorCameraEncoderCpu, isaac::ml::ColorCameraEncoderCuda, isaac::ml::DetectionDecoder, isaac::ml::DetectionEncoder, isaac::ml::EvaluateSegmentation, isaac::ml::HeatmapDecoder, isaac::ml::HeatmapEncoder, isaac::ml::LabelToBoundingBox, isaac::ml::SampleAccumulator, isaac::ml::SegmentationDecoder, isaac::ml::SegmentationEncoder, isaac::ml::Teleportation, isaac::ml::TensorRTInference, isaac::ml::TensorReshape, isaac::ml::TensorSynchronization, isaac::navigation::DistanceMap, isaac::navigation::GroupSelectorBehavior, isaac::navigation::NavigationMap, isaac::navigation::ObstacleWorld, isaac::navigation::RangeScanModelClassic, isaac::navigation::RangeScanModelFlatloc, isaac::navigation::SelectorBehavior, isaac::perception::CropAndDownsampleCpu, isaac::perception::CropAndDownsampleCuda, isaac::perception::EstimateGroundPlane, isaac::perception::EstimateObstacleDistances, isaac::perception::FiducialAsGoal, isaac::perception::ImageUndistortion, isaac::perception::PointCloudAccumulator, isaac::perception::RangeScanFlattening, isaac::perception::RangeToPointCloud, isaac::perception::ScanAccumulator, isaac::planner::DifferentialBaseModel, isaac::sight::AliceSight, isaac::sight::WebsightServer, isaac::viewers::BinaryMapViewer, isaac::viewers::ColorCameraViewer, isaac::viewers::DepthCameraViewer, isaac::viewers::Detections3Viewer, isaac::viewers::DetectionsViewer, isaac::viewers::FiducialsViewer, isaac::viewers::FlatscanViewer, isaac::viewers::GoalViewer, isaac::viewers::LidarViewer, isaac::viewers::MosaicViewer, isaac::viewers::OccupancyMapViewer, isaac::viewers::PointCloudViewer, isaac::viewers::SegmentationCameraViewer, isaac::viewers::SegmentationViewer, isaac::viewers::TensorListViewer,
2019-11-26 15:04:16.103 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'websight'
2019-11-26 15:04:16.103 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='websight', component='WebsightServer', key='tick_period'
2019-11-26 15:04:16.103 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='websight', component='isaac.sight.AliceSight', key='tick_period'
2019-11-26 15:04:16.103 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node '_config_bridge'
2019-11-26 15:04:16.103 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='_config_bridge', component='isaac.alice.ConfigBridge', key='tick_period'
2019-11-26 15:04:16.103 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node '_statistics'
2019-11-26 15:04:16.103 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node '_pose_tree_bridge'
2019-11-26 15:04:16.103 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node '_interactive_markers_bridge'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'simulation.interface'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.interface'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.color_camera_encoder'
2019-11-26 15:04:16.104 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='yolo_training.color_camera_encoder', component='isaac.ml.ColorCameraEncoderCpu', key='tick_period'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.detection_encoder'
2019-11-26 15:04:16.104 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='yolo_training.detection_encoder', component='isaac.ml.DetectionEncoder', key='tick_period'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.color_camera_visualizer'
2019-11-26 15:04:16.104 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='yolo_training.color_camera_visualizer', component='isaac.viewers.ColorCameraViewer', key='tick_period'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.bounding_box_visualizer'
2019-11-26 15:04:16.104 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='yolo_training.bounding_box_visualizer', component='isaac.viewers.DetectionsViewer', key='tick_period'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.tensor_synchronizer'
2019-11-26 15:04:16.104 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='yolo_training.tensor_synchronizer', component='isaac.ml.TensorSynchronization', key='tick_period'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.object_detection_training_samples'
2019-11-26 15:04:16.104 WARNING engine/alice/hooks/config_hook.cpp@59: Configuration does not contain a value and no default provided: node='yolo_training.object_detection_training_samples', component='isaac.ml.SampleAccumulator', key='tick_period'
2019-11-26 15:04:16.104 DEBUG engine/alice/backend/node_backend.cpp@49: Creating node 'yolo_training.camera_teleportation'
2019-11-26 15:04:16.104 WARNING engine/alice/application.cpp@173: The function Application::findComponentByName is deprecated. Please use `getNodeComponentOrNull` instead. Note that the new method requires a node name instead of a component name. (argument: 'websight/isaac.sight.AliceSight')
2019-11-26 15:04:16,104 DEBUG Launching isaac core
2019-11-26 15:04:16.104 INFO engine/alice/application.cpp@213: Starting application 'yolo_training_ue4' (instance UUID: 'ec635dce-1087-11ea-8c00-010d7058e921') ...
2019-11-26 15:04:16.104 DEBUG engine/gems/scheduler/execution_groups.cpp@484: Launching 0 pre-start job(s)
2019-11-26 15:04:16.104 DEBUG engine/gems/scheduler/execution_groups.cpp@493: Replaying 0 pre-start event(s)
2019-11-26 15:04:16.104 DEBUG engine/gems/scheduler/execution_groups.cpp@484: Launching 0 pre-start job(s)
2019-11-26 15:04:16.104 DEBUG engine/gems/scheduler/execution_groups.cpp@493: Replaying 0 pre-start event(s)
2019-11-26 15:04:16.104 INFO engine/alice/backend/asio_backend.cpp@33: Starting ASIO service
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'websight'
2019-11-26 15:04:16,105 DEBUG Launching pycodelet threads
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'websight/WebsightServer' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'websight/WebsightServer' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'websight/WebsightServer'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'websight/isaac.sight.AliceSight' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'websight/isaac.sight.AliceSight' DONE
2019-11-26 15:04:16.105 WARNING engine/alice/backend/codelet_backend.cpp@281: Codelet 'websight/isaac.sight.AliceSight' was not added to scheduler because no tick method is specified.
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node '_config_bridge'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet '_config_bridge/isaac.alice.ConfigBridge' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet '_config_bridge/isaac.alice.ConfigBridge' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet '_config_bridge/isaac.alice.ConfigBridge'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node '_statistics'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet '_statistics/NodeStatistics' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet '_statistics/NodeStatistics' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet '_statistics/NodeStatistics'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node '_pose_tree_bridge'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet '_pose_tree_bridge/PoseTreeJsonBridge' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet '_pose_tree_bridge/PoseTreeJsonBridge' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet '_pose_tree_bridge/PoseTreeJsonBridge'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node '_interactive_markers_bridge'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet '_interactive_markers_bridge/InteractiveMarkersBridge' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet '_interactive_markers_bridge/InteractiveMarkersBridge' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet '_interactive_markers_bridge/InteractiveMarkersBridge'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'simulation.interface'
2019-11-26 15:04:16.105 INFO engine/alice/components/TcpPublisher.cpp@75: Starting TCP publisher on port 5004
2019-11-26 15:04:16.105 ERROR engine/alice/component.cpp@61: Component 'simulation.interface/input' reported an error: Failed to bind TCP endpoint at port 5004. Is the port already in use? Will not publish any data.
2019-11-26 15:04:16.105 ERROR engine/alice/backend/event_manager.cpp@40: Stopping node 'simulation.interface' because it reached status 'FAILURE'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.interface'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.color_camera_encoder'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.color_camera_encoder/isaac.ml.ColorCameraEncoderCpu' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.color_camera_encoder/isaac.ml.ColorCameraEncoderCpu'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.detection_encoder'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.detection_encoder/isaac.ml.DetectionEncoder' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.detection_encoder/isaac.ml.DetectionEncoder' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.detection_encoder/isaac.ml.DetectionEncoder'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.color_camera_visualizer'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.color_camera_visualizer/isaac.viewers.ColorCameraViewer' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.color_camera_visualizer/isaac.viewers.ColorCameraViewer' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.color_camera_visualizer/isaac.viewers.ColorCameraViewer'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.bounding_box_visualizer'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.bounding_box_visualizer/isaac.viewers.DetectionsViewer' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.bounding_box_visualizer/isaac.viewers.DetectionsViewer' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.bounding_box_visualizer/isaac.viewers.DetectionsViewer'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.tensor_synchronizer'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.tensor_synchronizer/isaac.ml.TensorSynchronization' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.tensor_synchronizer/isaac.ml.TensorSynchronization' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.tensor_synchronizer/isaac.ml.TensorSynchronization'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.object_detection_training_samples'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.object_detection_training_samples/isaac.ml.SampleAccumulator' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.object_detection_training_samples/isaac.ml.SampleAccumulator' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.object_detection_training_samples/isaac.ml.SampleAccumulator'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/node_backend.cpp@317: Starting node 'yolo_training.camera_teleportation'
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@61: Starting codelet 'yolo_training.camera_teleportation/isaac.ml.Teleportation' ...
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@73: Starting codelet 'yolo_training.camera_teleportation/isaac.ml.Teleportation' DONE
2019-11-26 15:04:16.105 DEBUG engine/alice/backend/codelet_backend.cpp@291: Starting job for codelet 'yolo_training.camera_teleportation/isaac.ml.Teleportation'
2019-11-26 15:04:16.105 INFO engine/alice/components/TcpSubscriber.cpp@137: TCP receiver trying to connect to remote (localhost:5003)...
2019-11-26 15:04:16.106 INFO engine/alice/components/TcpSubscriber.cpp@167: Successfully connected to remote (localhost:5003). Will start receiving.
2019-11-26 15:04:16.114910: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2019-11-26 15:04:16.115 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'simulation.interface'
2019-11-26 15:04:16.115 INFO engine/alice/components/TcpSubscriber.cpp@85: Stopping TCP receiver for (localhost:5003)...
2019-11-26 15:04:16.115 ERROR engine/alice/components/TcpSubscriber.cpp@188: Socket failed to receive data. Will try to reconnect.
2019-11-26 15:04:16.115 INFO engine/alice/components/TcpSubscriber.cpp@102: Stopping TCP receiver for (localhost:5003)... DONE
2019-11-26 15:04:16.115 INFO engine/alice/components/TcpPublisher.cpp@108: Stopping TCP publisher on port 5004
2019-11-26 15:04:16.281270: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:998] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-11-26 15:04:16.281999: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x19fe800 executing computations on platform CUDA. Devices:
2019-11-26 15:04:16.282017: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): GeForce RTX 2080 Ti, Compute Capability 7.5
2019-11-26 15:04:16.301299: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3696000000 Hz
2019-11-26 15:04:16.301836: I tensorflow/compiler/xla/service/service.cc:150] XLA service 0x19ef230 executing computations on platform Host. Devices:
2019-11-26 15:04:16.301853: I tensorflow/compiler/xla/service/service.cc:158] StreamExecutor device (0): <undefined>, <undefined>
2019-11-26 15:04:16.301992: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1433] Found device 0 with properties:
name: GeForce RTX 2080 Ti major: 7 minor: 5 memoryClockRate(GHz): 1.635
pciBusID: 0000:01:00.0
totalMemory: 10.73GiB freeMemory: 8.80GiB
2019-11-26 15:04:16.302006: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-11-26 15:04:16.504417: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-26 15:04:16.504445: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-11-26 15:04:16.504451: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-11-26 15:04:16.504545: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5493 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
WARNING:tensorflow:From /home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-11-26 15:04:16,516 WARNING From /home/shatterhand/.local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py:263: colocate_with (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Colocations handled automatically by placer.
2019-11-26 15:04:16.530357: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1512] Adding visible gpu devices: 0
2019-11-26 15:04:16.530400: I tensorflow/core/common_runtime/gpu/gpu_device.cc:984] Device interconnect StreamExecutor with strength 1 edge matrix:
2019-11-26 15:04:16.530407: I tensorflow/core/common_runtime/gpu/gpu_device.cc:990] 0
2019-11-26 15:04:16.530411: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1003] 0: N
2019-11-26 15:04:16.530529: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 5493 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:01:00.0, compute capability: 7.5)
Create YOLOv3 model with 9 anchors and 4 classes.
/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/saving.py:1281: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((1, 1, 1024, 27) vs (255, 1024, 1, 1)).
weight_values[i].shape))
/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/saving.py:1281: UserWarning: Skipping loading of weights for layer conv2d_59 due to mismatch in shape ((27,) vs (255,)).
weight_values[i].shape))
/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/saving.py:1281: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((1, 1, 512, 27) vs (255, 512, 1, 1)).
weight_values[i].shape))
/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/saving.py:1281: UserWarning: Skipping loading of weights for layer conv2d_67 due to mismatch in shape ((27,) vs (255,)).
weight_values[i].shape))
/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/saving.py:1281: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((1, 1, 256, 27) vs (255, 256, 1, 1)).
weight_values[i].shape))
/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/saving.py:1281: UserWarning: Skipping loading of weights for layer conv2d_75 due to mismatch in shape ((27,) vs (255,)).
weight_values[i].shape))
Load weights ../yolo_pretrained_models/yolo.h5.
Freeze the first 249 layers of total 252 layers.
Traceback (most recent call last):
File "/home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/apps/samples/yolo/keras-yolo3/yolo_training.py", line 348, in <module>
_main()
File "/home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/apps/samples/yolo/keras-yolo3/yolo_training.py", line 83, in _main
weights_path='../yolo_pretrained_models/yolo.h5')
File "/home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/apps/samples/yolo/keras-yolo3/yolo_training.py", line 252, in create_model
})(model_body.output + y_true)
File "/home/shatterhand/.local/lib/python3.6/site-packages/keras/engine/base_layer.py", line 451, in __call__
output = self.call(inputs, **kwargs)
File "/home/shatterhand/.local/lib/python3.6/site-packages/keras/layers/core.py", line 716, in call
return self.function(inputs, **arguments)
File "/home/shatterhand/Projects/Navon/isaac/apps/samples/yolo/keras-yolo3/yolo3/model.py", line 404, in yolo_loss
_, ignore_mask = K.control_flow_ops.while_loop(lambda b,*args: b<m, loop_body, [0, ignore_mask])
AttributeError: module 'keras.backend' has no attribute 'control_flow_ops'
2019-11-26 15:04:26.490 INFO engine/alice/backend/backend.cpp@115: Backend is shutting down...
2019-11-26 15:04:26.513 DEBUG engine/alice/backend/node_backend.cpp@342: Stopping node 'yolo_training.camera_teleportation'
====================================================================================================
| Isaac application terminated unexpectedly |
====================================================================================================
#01 /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0x16abda) [0x7ff3af020bda]
#02 google_breakpad::ExceptionHandler::GenerateDump(google_breakpad::ExceptionHandler::CrashContext*) /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN15google_breakpad16ExceptionHandler12GenerateDumpEPNS0_12CrashContextE+0x3f0) [0x7ff3af0e3e50]
#03 google_breakpad::ExceptionHandler::SignalHandler(int, siginfo_t*, void*) /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN15google_breakpad16ExceptionHandler13SignalHandlerEiP9siginfo_tPv+0xc0) [0x7ff3af0e41c0]
#04 /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7ff434cddf20]
#05 isaac::alice::NodeBackend::stop() /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice11NodeBackend4stopEv+0x117) [0x7ff3af0300f7]
#06 isaac::alice::Backend::stop() /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice7Backend4stopEv+0x49) [0x7ff3af00f4a9]
#07 isaac::alice::Backend::~Backend() /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice7BackendD1Ev+0x19) [0x7ff3af010759]
#08 isaac::alice::Application::~Application() /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice11ApplicationD1Ev+0x2e) [0x7ff3aefe288e]
#09 isaac::alice::PybindApplication::~PybindApplication() /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(_ZN5isaac5alice17PybindApplicationD2Ev+0x19) [0x7ff3aefb0719]
#10 /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0xfa779) [0x7ff3aefb0779]
#11 /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0xf84c0) [0x7ff3aefae4c0]
#12 /home/shatterhand/.cache/bazel/_bazel_shatterhand/0af7fe4f32a7d778187d9ad7125d767d/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/samples/yolo/yolo_training.runfiles/com_nvidia_isaac/engine/pyalice/bindings.so(+0xf96ef) [0x7ff3aefaf6ef]
#13 /usr/bin/python3() [0x573520]
#14 /usr/bin/python3() [0x54f1a2]
#15 /usr/bin/python3() [0x5891e8]
#16 /usr/bin/python3() [0x5add18]
#17 /usr/bin/python3() [0x5add2e]
#18 /usr/bin/python3() [0x56b906]
#19 /usr/bin/python3(PyDict_SetItemString+0x153) [0x571573]
#20 /usr/bin/python3(PyImport_Cleanup+0x76) [0x4f3066]
#21 /usr/bin/python3(Py_FinalizeEx+0x5e) [0x63822e]
#22 /usr/bin/python3(Py_Main+0x395) [0x639295]
#23 /usr/bin/python3(main+0xe0) [0x4b0f60]
#24 /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7ff434cc0b97]
#25 /usr/bin/python3(_start+0x2a) [0x5b2eaa]
====================================================================================================
Minidump written to: /tmp/03f9308f-a2ab-4d52-b4fdd9bc-18263cd8.dmp
Segmentation fault (core dumped)