kaya_sim failure - tcp connection issue

I’m having an issue with kaya_sim (and other sim apps) where there seems to be a TCP communication failure. At this point, I’m running apps as they are without changes. For kaya_sim, I’m seeing the following output:

2019-05-07 03:17:44.661 INFO    engine/alice/components/TcpPublisher.cpp@73: Starting TCP transmitter on port 5002
2019-05-07 03:17:44.661 DEBUG   engine/alice/backend/node_backend.cpp@318: Starting node 'RigidBodySubscriber'
2019-05-07 03:17:44.661 INFO    engine/alice/components/TcpSubscriber.cpp@131: TCP receiver trying to connect to remote (localhost:5001)...
2019-05-07 03:17:44.661 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2019-05-07 03:17:45.161 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2019-05-07 03:17:45.661 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2019-05-07 03:17:46.113 INFO    packages/sight/WebsightServer.cpp@106: Server connected / 1
2019-05-07 03:17:46.161 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2019-05-07 03:17:46.661 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2019-05-07 03:17:47.162 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
2019-05-07 03:17:47.662 WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused. Will try again in 0.500000 seconds.
^C2019-05-07 03:17:48.065 INFO    engine/alice/application.cpp@262: Stopping application 'joystick' (instance UUID: '5acc540e-70b1-11e9-9313-ebe4ef9ce87e') ...

Hello,
you might find useful to check this link: https://docs.nvidia.com/isaac/isaac_sim/quickstart.html#kaya, these warnings are telling you that the Issac sim app couldn’t connect to the Isaac SDK application, you also need to start the app on the SDK side, by executing the following command at the SDK root directory:

bazel run apps/kaya_sim:joystick

Hi

Make sure both the sim and the IsaacSDK Kaya app are running at the same time

I have the same issue, but with the carter example:

the app is trying to connect to localhost:5000 in engine/alice/components/TcpSubscriber.cpp@156, but the sim appears to be sending on:

2019-05-16 20:19:52.427 DEBUG node_backend.cpp@285: Starting node ‘LidarPublisher’
2019-05-16 20:19:52.427 INFO TcpPublisher.cpp@73: Starting TCP transmitter on port 5000

Is this a misconfiguration somewhere?

I should also note, I’m running:

bazel run apps/carter_sim:carter_sim – --config=“apps/assets/maps/carter_warehouse_p.config.json” --graph=“apps/assets/maps/carter_warehouse_p.graph.json”

Ok, if I switch localhost for 0.0.0.0 in the configs, it connects, then segfaults:

2019-05-16 20:32:22.990 INFO    engine/alice/components/TcpSubscriber.cpp@161: Successfully connected to remote (0.0.0.0:5003). Will start receiving.
2019-05-16 20:32:22.990 INFO    engine/alice/components/TcpSubscriber.cpp@161: Successfully connected to remote (0.0.0.0:5001). Will start receiving.
2019-05-16 20:32:22.990 INFO    engine/alice/components/TcpSubscriber.cpp@161: Successfully connected to remote (0.0.0.0:5004). Will start receiving.
2019-05-16 20:32:22.990 INFO    engine/alice/components/TcpSubscriber.cpp@161: Successfully connected to remote (0.0.0.0:5000). Will start receiving.
2019-05-16 20:32:23.057 PANIC   engine/core/buffers/algorithm_cuda.cpp@55: Could not copy memory. Error: 35
====================================================================================================
|                            Isaac application terminated unexpectedly                             |
====================================================================================================
#01 engine/alice/tools/main(+0x7ca57) [0x560a8de1da57]
#02 engine/alice/tools/main(+0xc6850) [0x560a8de67850]
#03 engine/alice/tools/main(+0xc6bc0) [0x560a8de67bc0]
#04 /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890) [0x7f41a8424890]
#05 /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7) [0x7f41a7abee97]
#06 /lib/x86_64-linux-gnu/libc.so.6(abort+0x141) [0x7f41a7ac0801]
#07 packages_x86_64/navigation/libnavigation_module.so(+0x2fee36) [0x7f41a6002e36]
#08 isaac::navigation::Transform(isaac::partitions::GridMap<isaac::navigation::ObservedOccupiedMapCell, float>&, isaac::Pose2<float> const&) packages_x86_64/navigation/libnavigation_module.so(_ZN5isaac10navigation9TransformERNS_10partitions7GridMapINS0_23ObservedOccupiedMapCellEfEERKNS_5Pose2IfEE+0x9e) [0x7f41a5ee8cee]
#09 isaac::navigation::LocalMap::tick() packages_x86_64/navigation/libnavigation_module.so(_ZN5isaac10navigation8LocalMap4tickEv+0x7c0) [0x7f41a5e469d0]
#10 engine/alice/tools/main(+0x5af94) [0x560a8ddfbf94]
#11 engine/alice/tools/main(+0xbbec6) [0x560a8de5cec6]
#12 engine/alice/tools/main(+0x17ae9) [0x560a8ddb8ae9]
#13 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xbd57f) [0x7f41a814657f]
#14 /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db) [0x7f41a84196db]
#15 /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f41a7ba188f]
====================================================================================================
Minidump written to: /tmp/dff6f7da-bfbe-493f-13c07193-afbc7af1.dmp

Annnd, with current drivers + cuda 10.1 (install via runfile), it runs!

(drivers via)

sudo add-apt-repository ppa:graphics-drivers
sudo apt-get update
sudo ubuntu-drivers autoinstall

Hi

@tslator1,

Did you finally get you Kaya app running?

@natester

I am glad you figured it out.

Thanks,
Liila

@tslator1 I ran into the same problem,
WARNING engine/alice/components/TcpSubscriber.cpp@156: Failed to connect to remote: Connection refused.

My mistake was that I didn’t use the full path at the Setup Paths step for the Carter and Kaya Sim.

This was the incorrect full path. I thought I could type it out by heart but boy was I wrong XD

{
 "graphs": ["/home/isaac/apps/carter_sim/bridge_config/carter_full_graph.json"],
 "configs": ["/home/isaac/apps/carter_sim/bridge_config/carter_full_config.json"]
}

cd to the actual location of apps/carter_sim/bridge_config/carter_full_graph.json and use pwd to find the actual full path to carter_full_graph.json and carter_full_config.json.

jackdaw@jackdaw-ubuntu-18:~/isaac/apps/carter_sim/bridge_config$ pwd
=> /home/jackdaw/isaac/apps/carter_sim/bridge_config

So the correct full path for me is:

{
 "graphs": ["/home/jackdaw/isaac/apps/carter_sim/bridge_config/carter_full_graph.json"],
 "configs": ["/home/jackdaw/isaac/apps/carter_sim/bridge_config/carter_full_config.json"]
}

This solves the “TCP Failed to connect to remote: Connection refused.” problem and I now have Carter Sim and Kaya Sim running.

Make sure the full path is correct.