Isaac application terminated unexpectedly - docker + isaac sdk + isaac sim

Hi,

https://docs.nvidia.com/isaac/isaac_sim/quickstart.html

I’m following above link to try the carter warehouse simulation. I build isaac sdk with dockerfile and mount compiled isaac sim in container.

when I run
./Engine/Binaries/Linux/UE4Editor IsaacSimProject CarterWarehouse_P -vulkan -isaac_sim_config_json="<FULL_PATH>/apps/carter/carter_sim/bridge_config/carter_full.json"

changed the <Full_PATH> to my working directory
no errors and UE4 opens.

then open another terminal and run
bazel run apps/carter/carter_sim:carter_sim – --config=“apps/assets/maps/carter_warehouse_p.config.json” --graph=“apps/assets/maps/carter_warehouse_p.graph.json”

no errors but show WARN engine/alice/components/TcpSubscriber.cpp@162: Failed to connect to remote. Will try again in 0.500000 seconds.

then after I click Play in UE4, the robot spawns. Isaac sdk terminal terminated unexpectedly.

isaac sim terminal output after click Play:

2020-03-19 16:01:23.270 INFO TcpPublisher.cpp@82: Starting TCP transmitter on port 5001
2020-03-19 16:01:23.271 INFO TcpPublisher.cpp@82: Starting TCP transmitter on port 5003
2020-03-19 16:01:23.271 INFO TcpSubscriber.cpp@137: TCP receiver trying to connect to remote (localhost:5002)…
2020-03-19 16:01:23.271 INFO TcpPublisher.cpp@82: Starting TCP transmitter on port 5000
2020-03-19 16:01:23.271 INFO TcpPublisher.cpp@82: Starting TCP transmitter on port 5004
2020-03-19 16:01:23.271 INFO TcpSubscriber.cpp@167: Successfully connected to remote (localhost:5002). Will start receiving.
2020-03-19 16:01:23.289 INFO Interface.cpp@158: All nodes started
[2020.03.19-08.01.23:289][282]LogBlueprintUserMessages: [CharacterSpawner]
Random Seed for Character Simulation: 28883289
[2020.03.19-08.01.23:294][282]PIE: Play in editor start time for /Game/Maps/Warehouse/UEDPIE_0_CarterWarehouse_P 0.67592
[2020.03.19-08.01.23:294][282]LogBlueprintUserMessages: Late PlayInEditor Detection: Level ‘/Game/Maps/Warehouse/CarterWarehouse_P.CarterWarehouse_P:PersistentLevel’ has LevelScriptBlueprint ‘/Game/Maps/Warehouse/CarterWarehouse_P.CarterWarehouse_P:PersistentLevel.CarterWarehouse_P’ with GeneratedClass ‘/Game/Maps/Warehouse/CarterWarehouse_P.CarterWarehouse_P_C’ with ClassGeneratedBy ‘/Game/Maps/Warehouse/CarterWarehouse_P.CarterWarehouse_P:PersistentLevel.CarterWarehouse_P’
[2020.03.19-08.01.23:295][282]LogBlueprintUserMessages: Late PlayInEditor Detection: Level ‘/Game/Maps/Warehouse/CarterWarehouse_Geo.CarterWarehouse_Geo:PersistentLevel’ has LevelScriptBlueprint ‘/Game/Maps/Warehouse/CarterWarehouse_Geo.CarterWarehouse_Geo:PersistentLevel.CarterWarehouse_Geo’ with GeneratedClass ‘/Game/Maps/Warehouse/CarterWarehouse_Geo.CarterWarehouse_Geo_C’ with ClassGeneratedBy ‘/Game/Maps/Warehouse/CarterWarehouse_Geo.CarterWarehouse_Geo:PersistentLevel.CarterWarehouse_Geo’
[2020.03.19-08.01.23:299][282]LogClass: Display: Spawned Pawn_Carter_FirstPerson_C_0
[2020.03.19-08.01.23:329][282]Cmd: ShowFlag.Rendering 0
[2020.03.19-08.01.23:329][282]ShowFlag.Rendering = “0”
[2020.03.19-08.01.23:329][282]LogClass: Display: Spawned IsaacSimController_BP_C_0
[2020.03.19-08.01.23:392][282]LogClass: Display: Spawned IsaacSimCameraOverlayBP_C_0
[2020.03.19-08.01.23:392][282]IsaacSim: Display: Create a Lidar
[2020.03.19-08.01.23:393][282]LogActorComponent: RegisterComponentWithWorld: (/Game/Maps/Warehouse/UEDPIE_0_CarterWarehouse_P.CarterWarehouse_P:PersistentLevel.Pawn_Carter_FirstPerson_C_0.d344f2ec-69b7-11ea-af04-0242c0a87f80) Already registered. Aborting.
[2020.03.19-08.01.23:409][282]LogActorComponent: RegisterComponentWithWorld: (/Game/Maps/Warehouse/UEDPIE_0_CarterWarehouse_P.CarterWarehouse_P:PersistentLevel.Pawn_Carter_FirstPerson_C_0.d344f72e-69b7-11ea-af04-0242c0a87f80) Already registered. Aborting.
[2020.03.19-08.01.23:409][282]LogActorComponent: RegisterComponentWithWorld: (/Game/Maps/Warehouse/UEDPIE_0_CarterWarehouse_P.CarterWarehouse_P:PersistentLevel.Pawn_Carter_FirstPerson_C_0.d344fb84-69b7-11ea-af04-0242c0a87f80) Already registered. Aborting.
2020-03-19 16:01:23.514 INFO TcpPublisher.cpp@119: TCP transmitter successfully accepted conntection
2020-03-19 16:01:23.514 INFO TcpPublisher.cpp@119: TCP transmitter successfully accepted conntection
2020-03-19 16:01:23.514 INFO TcpPublisher.cpp@119: TCP transmitter successfully accepted conntection
2020-03-19 16:01:23.514 INFO TcpPublisher.cpp@119: TCP transmitter successfully accepted conntection
[2020.03.19-08.01.23:632][283]LogStaticMesh: Allocated 457x87x88 distance field atlas = 6.7Mb, with 145 objects containing 2.3Mb backing data
2020-03-19 16:01:24.079 WARNING TcpSubscriber.cpp@190: EOF encountered which likely indicates that the connection was closed by the remote. Will try to reconnect.
2020-03-19 16:01:24.079 WARNING TcpPublisher.cpp@150: Did not sent all bytes (1675177 instead of 2073600)
2020-03-19 16:01:24.079 WARNING TcpPublisher.cpp@161: Closed 1 invalid connection(s)
2020-03-19 16:01:24.080 INFO TcpSubscriber.cpp@137: TCP receiver trying to connect to remote (localhost:5002)…
2020-03-19 16:01:24.080 WARNING TcpSubscriber.cpp@162: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2020-03-19 16:01:24.580 WARNING TcpSubscriber.cpp@162: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2020-03-19 16:01:25.080 WARNING TcpSubscriber.cpp@162: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2020-03-19 16:01:25.580 WARNING TcpSubscriber.cpp@162: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.

isaac sdk terminal output after click Play:

2020-03-19 16:01:23.271 INFO engine/alice/components/TcpPublisher.cpp@141: TCP publisher successfully accepted conntection
2020-03-19 16:01:23.514 INFO engine/alice/components/TcpSubscriber.cpp@167: Successfully connected to remote (localhost:5001). Will start receiving.
2020-03-19 16:01:23.514 INFO engine/alice/components/TcpSubscriber.cpp@167: Successfully connected to remote (localhost:5000). Will start receiving.
2020-03-19 16:01:23.514 INFO engine/alice/components/TcpSubscriber.cpp@167: Successfully connected to remote (localhost:5004). Will start receiving.
2020-03-19 16:01:23.514 INFO engine/alice/components/TcpSubscriber.cpp@167: Successfully connected to remote (localhost:5003). Will start receiving.
terminate called after throwing an instance of ‘kj::ExceptionImpl’
what(): capnp/layout.c++:2169: failed: expected ref->kind() == WirePointer::STRUCT; Message contains non-struct pointer where struct pointer was expected.
stack: 7f6734fef382 7f6734ff103f 7f6734feb544 7f673507fb26 7f6735080343 7f6734feb7fc 55b493278a75 55b4932e24f5 55b493214a0b 7f6737b4f6ee 7f6737e226da 7f67375aa88e

| Isaac application terminated unexpectedly |

#01 engine/alice/tools/main(+0x880ca) [0x55b4932800ca]
#02 engine/alice/tools/main(+0xf97f0) [0x55b4932f17f0]
#03 engine/alice/tools/main(+0xf9b60) [0x55b4932f1b60]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f6737e2d890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f67374c7e97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f67374c9801]
#07 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x8c957) [0x7f6737b1e957]
#08 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92ae6) [0x7f6737b24ae6]
#09 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92b21) [0x7f6737b24b21]
#10 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0x92d54) [0x7f6737b24d54]
#11 kj::ExceptionCallback::RootExceptionCallback::onRecoverableException(kj::Exception&&) /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(ZN2kj17ExceptionCallback21RootExceptionCallback22onRecoverableExceptionEONS_9ExceptionE+0x249) [0x7f67352646e9]
#12 kj::
::Debug::Fault::~Fault() /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(ZN2kj1_5Debug5FaultD1Ev+0x1bc) [0x7f6735253fec]
#13 capnp::
::PointerReader::getStruct(capnp::word const*) const /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(_ZNK5capnp1_13PointerReader9getStructEPKNS_4wordE+0x439) [0x7f673521b319]
#14 isaac::perception::ScanAccumulator::processScan(RangeScanProto::Reader, std::vector<isaac::SharedBuffer, std::allocatorisaac::SharedBuffer > const&, long, long) /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(_ZN5isaac10perception15ScanAccumulator11processScanEN14RangeScanProto6ReaderERKSt6vectorINS_12SharedBufferESaIS5_EEll+0x193) [0x7f6734fef383]
#15 /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(+0x1d5040) [0x7f6734ff1040]
#16 std::_Function_handler<void (std::shared_ptr<isaac::alice::MessageBase const> const&), isaac::alice::ProtoRx::processAllNewMessagesBuffered(std::function<void (RangeScanProto::Reader, std::vector<isaac::SharedBuffer, std::allocatorisaac::SharedBuffer > const&, long, long)>) const::{lambda(std::shared_ptr<isaac::alice::MessageBase const> const&)#1}>::_M_invoke(std::_Any_data const&, std::shared_ptr<isaac::alice::MessageBase const> const&) /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(ZNSt17_Function_handlerIFvRKSt10shared_ptrIKN5isaac5alice11MessageBaseEEEZNKS2_7ProtoRxI14RangeScanProtoE29processAllNewMessagesBufferedESt8functionIFvNSA_6ReaderERKSt6vectorINS1_12SharedBufferESaISF_EEllEEEUlS7_E_E9_M_invokeERKSt9_Any_dataS7+0x105) [0x7f6734feb545]
#17 isaac::alice::MessageLedger::Channel::readAllNew(std::function<void (std::shared_ptr<isaac::alice::MessageBase const> const&)>) /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(_ZN5isaac5alice13MessageLedger7Channel10readAllNewESt8functionIFvRKSt10shared_ptrIKNS0_11MessageBaseEEEE+0x77) [0x7f673507fb27]
#18 isaac::alice::MessageLedger::readAllNew(isaac::alice::MessageLedger::Endpoint const&, std::function<void (std::shared_ptr<isaac::alice::MessageBase const> const&)>) const /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(_ZNK5isaac5alice13MessageLedger10readAllNewERKNS1_8EndpointESt8functionIFvRKSt10shared_ptrIKNS0_11MessageBaseEEEE+0x74) [0x7f6735080344]
#19 isaac::perception::ScanAccumulator::tick() /home/developer/.cache/bazel/_bazel_root/0a4466b5ddf1bb82f913f8d41c50582b/execroot/com_nvidia_isaac/bazel-out/k8-opt/bin/apps/carter/carter_sim/carter_sim.runfiles/com_nvidia_isaac//packages_x86_64/perception/libperception_module.so(_ZN5isaac10perception15ScanAccumulator4tickEv+0x11d) [0x7f6734feb7fd]
#20 engine/alice/tools/main(+0x80a76) [0x55b493278a76]
#21 engine/alice/tools/main(+0xea4f6) [0x55b4932e24f6]
#22 engine/alice/tools/main(+0x1ca0c) [0x55b493214a0c]
#23 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd6ef) [0x7f6737b4f6ef]
#24 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f6737e226db]
#25 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f67375aa88f]

Minidump written to: /tmp/8a640431-fb09-4cc8-a1e3ff85-dde7899b.dmp
Aborted (core dumped)

Any help will be appreciated!

DGX station
ubuntu 18.04
cudnn 7.6.1.34-1+cuda10.1
nvidia driver 418.67
vulkan sdk 1.1.101
isaac sdk 2019.3

Hi dlan,
With Isaac SDK 2019.3 we don’t support Isaac SIM based on UE4 anymore. Can you try using Unity3D: https://docs.nvidia.com/isaac/isaac/doc/simulation/unity3d.html?