UCX for GPU tensors chosing TCP over CUDA for transporting GPU tensors

utkarsh02t · May 26, 2026, 7:26pm

I’m creating a holoscan distributed application.

Currently each fragment is running as a different process in the same container.

The container is a TensorRT-LLM container with holoscan–cu13 installed a pip module
Started with the following command:

 docker run --rm -it --network host --name devut_holo_container --ipc host --shm-size=32g --gpus all --ulimit memlock=-1 --cap-add=CAP_SYS_PTRACE --runtime=nvidia -v $(pwd):/workspace -v ~/.cache/huggingface:/root/.cache/huggingface -w /workspace --ulimit stack=67108864 holotensorrt_llm:latest

Few of the fragments send GPU tensors on gpu specific ports. (I have segregated CPU-CPU tensor ports)

The transport method begin used is tcp/lo instead of cuda_copy, or even cuda_ipc

2026-05-26 09:53:27,871 - DEBUG - embed_node_variant_collector emit_gpu is not None! it is {'embedding': <holoscan.core._core.Tensor object at 0x7fa30b429bb0>}
      | tcp/lo |
[1779789016.224876] [server:14442:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1779789016.228024] [server:14442:2]   +--------------------------------+-------------------------------------------------------------------------------------+
[1779789016.228026] [server:14442:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag(fast-completion) from host memory    |
[1779789016.228027] [server:14442:2]   +--------------------------------+----------------------------------------------------------------------------+--------+
[1779789016.228028] [server:14442:2]   |                      0..262143 | short                                                                      | tcp/lo |
[1779789016.228029] [server:14442:2]   |                   256K..447669 | (?) rendezvous zero-copy                                                   | tcp/lo |
[1779789016.228030] [server:14442:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote                            | tcp/lo |
[1779789016.228031] [server:14442:2]   +--------------------------------+----------------------------------------------------------------------------+--------+
[1779789016.231461] [server:14442:2]   +--------------------------------+---------------------------------------------------------------------------+
[1779789016.231463] [server:14442:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag(multi) from host memory    |
[1779789016.231464] [server:14442:2]   +--------------------------------+------------------------------------------------------------------+--------+
[1779789016.231465] [server:14442:2]   |                         0..626 | short                                                            | tcp/lo |
[1779789016.231466] [server:14442:2]   |                      627..1862 | multi-frag copy-in                                               | tcp/lo |
[1779789016.231467] [server:14442:2]   |                      1863..inf | (?) rendezvous fragmented copy-in copy-out                       | tcp/lo |
[1779789016.231467] [server:14442:2]   +--------------------------------+------------------------------------------------------------------+--------+
[1779789207.748129] [server:14442:3]   +--------------------------------+---------------------------------------------------+
[1779789207.748137] [server:14442:3]   | client_server inter-node cfg#1 | rendezvous data fetch into cuda/GPU0 from cuda    |
[1779789207.748139] [server:14442:3]   +--------------------------------+------------------------------------------+--------+
[1779789207.748141] [server:14442:3]   |                              0 | no data fetch                            |        |
[1779789207.748143] [server:14442:3]   |                         1..inf | (?) fragmented copy-in copy-out          | tcp/lo |
[1779789207.748145] [server:14442:3]   +--------------------------------+------------------------------------------+--------+
[1779789207.875243] [server:14442:3]   +--------------------------------+--------------------------------------------------------+
[1779789207.875246] [server:14442:3]   | client_server inter-node cfg#3 | active message by ucp_am_send*(rndv) from cuda/GPU0    |
[1779789207.875248] [server:14442:3]   +--------------------------------+-----------------------------------------------+--------+
[1779789207.875250] [server:14442:3]   |                              0 | (?) rendezvous no data fetch                  |        |
[1779789207.875251] [server:14442:3]   |                         1..inf | (?) rendezvous fragmented copy-in copy-out    | tcp/lo |
[1779789207.875252] [server:14442:3]   +--------------------------------+-----------------------------------------------+--------+
[1779789208.148496] [server:14442:3]   +--------------------------------+------------------------------------------------+

I have tried with UCX_TLS=all and different combinations.

ucx_info -v :

ucx_info -v
# Library version: 1.20.0
# Library path: /usr/local/ucx//lib/libucs.so.0
# API headers version: 1.20.0
# Git branch '', revision f656dbd
# Configured with: --disable-logging --disable-debug --disable-assertions --disable-params-check --prefix=/usr/local/ucx/ --enable-shared --disable-static --disable-doxygen-doc --enable-optimizations --enable-cma --enable-devel-headers --with-cuda=/usr/local/cuda --with-verbs --with-dm --enable-mt

OS version + CPU Arch:
cat /etc/issue:

root@server:/workspace# cat /etc/issue
Ubuntu 24.04.3 LTS \n \l

uname -a:

root@server:/workspace# uname -a
Linux server 6.8.12+ #13 SMP PREEMPT_DYNAMIC Wed Feb 11 15:34:54 IST 2026 x86_64 x86_64 x86_64 GNU/Linux

GPU related information

GPU Type:

GPU 0 : NVIDIA RTX PRO 4500 Blackwell
GPU 1: NVIDIA RTX A5000

Driver version from nvidia-smi:

| NVIDIA-SMI 595.71.05              Driver Version: 595.71.05      CUDA Version: 13.2     |

GPU Topology from nvidia-smi topo -m:

        GPU0    GPU1    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      SYS     0-15,32-47      0               N/A
GPU1    SYS      X      16-31,48-63     1               N/A

Here is my ucx_info -d: https://gist.github.com/DevUt/dcb2947b86385c9d3d77cba3a1f2845a
My UCX_* variables:

export UCX_PROTO_INFO=y
export UCX_TLS=cuda_copy,sm,shm,tcp
export UCX_LOG_LEVEL=info
export UCX_ADDRESS_DEBUG_INFO=y
export UCX_CUDA_COPY_FAILURE=TRACE
export UCX_CUDA_IPC_FAILURE=TRACE
export UCX_TCP_TX_SEG_SIZE=2M
export UCX_TCP_RX_SEG_SIZE=2M

My suspicions are on it being identified as inter-node instead of intra-node.

I also tried with ucx_perftestwith cuda tag, and it worked with cuda_copy and cuda_ipc.

I launch my fragments as such

python frag_tensorrt_rag_new.py --driver --worker --fragments=api_tier --worker-address=127.0.0.1:50000 2>&1 | grep --line-buffered -v 'Received null entity in transmitter' > logs/api_tier.log &

sleep 10

echo "Starting Worker for embed_node_router_fragment on port 50001..."
python frag_tensorrt_rag_new.py --worker --fragments=embed_node_router_fragment --worker-address=127.0.0.1:50001 2>&1 | grep --line-buffered -v 'Received null entity in transmitter' > logs/embed_node_router_fragment.log &
sleep 0.1
echo "Starting Worker for embed_node_collector_fragment on port 50002..."
python frag_tensorrt_rag_new.py --worker --fragments=embed_node_collector_fragment --worker-address=127.0.0.1:50002 2>&1 | grep --line-buffered -v 'Received null entity in transmitter' > logs/embed_node_collector_fragment.log &
sleep 0.1

and so on…

utkarsh02t · June 1, 2026, 9:43am

Any help will be highly appreciated :D

utkarsh02t · June 4, 2026, 6:46am

I did some more experiments.

This is NOT just a GPU tensor issue. I see that the CPU-tensors are also being passed along the TCP/lo route.

Banning tcp from TLS doesn’t work, because the fragments are not able to form a connection.

At this point I have tried quite a few UCX flags, which all have failed me.

I ran UCX perf tests with debug and and proto info, and was able to see that CMA was being used.

So its not a container or capabilities issue.
This is only happening with holoscan, and I highly suspect it is because of the inter-node classification.
Is there an hidden assumption that fragments on different processes MUST be on different nodes? But UCX should be able to detect that the nodes are same!

We are currently doing different operator compute control through MPS which requires different fragments.

grelee · June 5, 2026, 12:11pm

Hi,

Thank you for the detailed diagnostic information. Unfortunately, I don’t immediately see the issue, but below are a few things you might try to debug further.

One possible issue is that I see that this is a multi-GPU setup. Can you confirm whether all fragments of the distributed app are using the same GPU for the tensors being sent/received? Distributed applications have been tested on systems where each node typically only has one discrete GPU.

In Holoscan distributed execution, the UcxContext is created automatically by the driver/worker, and the executor later generates GPUDevice resources for UCX network entities. Those generated GPUDevice resources default to CUDA device 0 unless HOLOSCAN_UCX_DEVICE_ID is set.

To simplify things, it would be good to first confirm if things are working when only 1 GPU is visible in the system by setting

export CUDA_VISIBLE_DEVICES=0
export HOLOSCAN_UCX_DEVICE_ID=0

Also note that Holoscan currently disables cuda_ipc by default if UCX_TLS is not explicitly set (I think this was due to a limitation with lack of CUDA IPC support on the Orin iGPU)

You mentioned having tried different settings for UCX_TLS. If you did not already, I would try with UCX_TLS=cuda_ipc,cuda_copy,tcp
You mentioned ucx_perftest is working. Can you also verify it works with settings such as the following which will use active message APIs to more closely match Holoscan’s UCX usage:

export UCX_TLS=cuda_copy,cuda_ipc,tcp
export UCX_PROTO_INFO=y
export UCX_LOG_LEVEL=info

where you can vary UCX_TLS values (and/or whether UCX_RNDV_THRESH=0 is set) to narrow the transports allowed to confirm which can work in the scenario

Receiver/server process:

CUDA_VISIBLE_DEVICES=0 ucx_perftest -t ucp_am_bw -m cuda -s 1048576 -H 256 -n 10000 -w 1000 -p 50000

Sender/client process:

CUDA_VISIBLE_DEVICES=0 ucx_perftest IP_ADDRESS -t ucp_am_bw -m cuda -s 1048576 -H 256 -n 10000 -w 1000 -p 50000

Reasoning for specific options:

-t ucp_am_bw: uses UCP active message API, i.e. ucp_am_send_nbx() / ucp_am_recv_data_nbx().
-m cuda: CUDA memory on both send and receive sides.
Can try varying CUDA_VISIBLE_DEVICES=0 vs. CUDA_VISIBLE_DEVICES=1 for the two processes (it seems the -a argument to ucx_perftest is not supported for the ucp_am_bw test so the environment can be used to select the device)
UCX_RNDV_THRESH=0: force rendezvous selection.
-H 256: adds an AM header

When I tried this locally, if cuda_ipc was within UCX_TLS, I had to substitute the actual IP address of the network card in my system (as reported by ifconfig) for IP_ADDRESS in the command above and not 127.0.0.1

utkarsh02t · June 5, 2026, 1:17pm

I did the steps

I tried,
export CUDA_VISIBLE_DEVICES=0
export HOLOSCAN_UCX_DEVICE_ID=0

with UCX_TLS=cuda_ipc,cuda_copy,tcp

[1780663142.550435] [server:15089:1]   +--------------------------------+---------------------------------------------------------------------------+
[1780663142.550438] [server:15089:1]   | client_server inter-node cfg#3 | active message by ucp_am_send* with reply flag(multi) from host memory    |
[1780663142.550440] [server:15089:1]   +--------------------------------+------------------------------------------------------------------+--------+
[1780663142.550441] [server:15089:1]   |                         0..626 | short                                                            | tcp/lo |
[1780663142.550443] [server:15089:1]   |                      627..1862 | multi-frag copy-in                                               | tcp/lo |
[1780663142.550444] [server:15089:1]   |                      1863..inf | (?) rendezvous fragmented copy-in copy-out                       | tcp/lo |
[1780663142.550446] [server:15089:1]   +--------------------------------+------------------------------------------------------------------+--------+
[1780663142.551247] [server:15089:1]     ucp_context.c:2463 UCX  INFO  Version 1.20.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
[1780663182.216834] [server:15089:3]   +--------------------------------+---------------------------------------------------+
[1780663182.216839] [server:15089:3]   | client_server inter-node cfg#1 | rendezvous data fetch into cuda/GPU0 from cuda    |
[1780663182.216841] [server:15089:3]   +--------------------------------+------------------------------------------+--------+
[1780663182.216843] [server:15089:3]   |                              0 | no data fetch                            |        |
[1780663182.216844] [server:15089:3]   |                         1..inf | (?) fragmented copy-in copy-out          | tcp/lo |
[1780663182.216846] [server:15089:3]   +--------------------------------+------------------------------------------+--------+
[1780663182.233334] [server:15089:4]   +--------------------------------+--------------------------------------------------------+
[1780663182.233340] [server:15089:4]   | client_server inter-node cfg#3 | active message by ucp_am_send*(rndv) from cuda/GPU0    |
[1780663182.233343] [server:15089:4]   +--------------------------------+-----------------------------------------------+--------+
[1780663182.233345] [server:15089:4]   |                              0 | (?) rendezvous no data fetch                  |        |
[1780663182.233348] [server:15089:4]   |                         1..inf | (?) rendezvous fragmented copy-in copy-out    | tcp/lo |
[1780663182.233350] [server:15089:4]   +--------------------------------+-----------------------------------------------+--------+
[1780663182.247294] [server:15089:4]   +--------------------------------+------------------------------------------------+
[1780663182.247298] [server:15089:4]   | client_server inter-node cfg#3 | rendezvous data send from cuda/GPU0 to cuda    |
[1780663182.247300] [server:15089:4]   +--------------------------------+---------------------------------------+--------+
[1780663182.247302] [server:15089:4]   |                         0..inf | fragmented copy-in copy-out           | tcp/lo |
[1780663182.247304] [server:15089:4]   +--------------------------------+---------------------------------------+--------+
[1780663182.248954] [server:15089:4]   +--------------------------------+--------------------------------------------------------------+
[1780663182.248957] [server:15089:4]   | client_server inter-node cfg#3 | active message by ucp_am_send*(rndv) from iov host memory    |

This didn’t work :(. I am only still seeing TCP/lo being used.

With ucx_perftest
Sender/client:

root@server:/workspace# CUDA_VISIBLE_DEVICES=0 ucx_perftest 127.0.0.1 -t ucp_am_bw -m cuda -s 1048576 -H 256 -n 10000 -w 1000 -p 50000
[1780663508.261263] [server:266  :0]        perftest.c:825  UCX  WARN  CPU affinity is not set (bound to 64 cpus). Performance may be impacted.
+--------------+--------------+------------------------------+---------------------+-----------------------+
|              |              |       overhead (usec)        |   bandwidth (MB/s)  |  message rate (msg/s) |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
|    Stage     | # iterations | 50.0%ile | average | overall |  average |  overall |  average  |  overall  |
+--------------+--------------+----------+---------+---------+----------+----------+-----------+-----------+
[1780663508.689658] [server:266  :0]     ucp_context.c:2463 UCX  INFO  Version 1.20.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
[1780663508.793063] [server:266  :0]   +---------------------+-------------------------------------------------------------+
[1780663508.793073] [server:266  :0]   | perftest self cfg#0 | remote memory write by ucp_put* from host memory to host    |
[1780663508.793076] [server:266  :0]   +---------------------+--------------------------------------------+----------------+
[1780663508.793078] [server:266  :0]   |       0..4294967295 | short                                      | cuda_copy/cuda |
[1780663508.793080] [server:266  :0]   +---------------------+--------------------------------------------+----------------+
[1780663508.793111] [server:266  :0]   +---------------------+------------------------------------------------------------------------------+
[1780663508.793114] [server:266  :0]   | perftest self cfg#0 | remote memory write by ucp_put*(fast-completion) from host memory to host    |
[1780663508.793116] [server:266  :0]   +---------------------+-------------------------------------------------------------+----------------+
[1780663508.793118] [server:266  :0]   |       0..4294967295 | short                                                       | cuda_copy/cuda |
[1780663508.793120] [server:266  :0]   +---------------------+-------------------------------------------------------------+----------------+
[1780663508.793146] [server:266  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.793150] [server:266  :0]   | perftest self cfg#0 | remote memory write by ucp_put*(multi) from host memory to host    |
[1780663508.793152] [server:266  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.793154] [server:266  :0]   |       0..4294967295 | short                                             | cuda_copy/cuda |
[1780663508.793158] [server:266  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.793193] [server:266  :0]   +---------------------+----------------------------------------------------+
[1780663508.793197] [server:266  :0]   | perftest self cfg#0 | active message by ucp_am_send* from host memory    |
[1780663508.793199] [server:266  :0]   +---------------------+-------------------------------------------------+--+
[1780663508.793200] [server:266  :0]   +---------------------+-------------------------------------------------+--+
[1780663508.793213] [server:266  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.793217] [server:266  :0]   | perftest self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1780663508.793219] [server:266  :0]   +---------------------+-----------------------------------------------------------------+--+
[1780663508.793221] [server:266  :0]   +---------------------+-----------------------------------------------------------------+--+
[1780663508.793402] [server:266  :0]          parser.c:2364 UCX  INFO  UCX_* env variables: UCX_PROTO_INFO=y UCX_TLS=cuda_copy,cuda_ipc,tcp UCX_LOG_LEVEL=info
[1780663508.795279] [server:266  :0]      ucp_worker.c:1912 UCX  INFO    perftest intra-node cfg#1 rma_am(tcp/enp28s0f0np0)  am(tcp/enp28s0f0np0 cuda_ipc/cuda)
[1780663508.798818] [server:266  :0]   +---------------------------+-------------------------------------------------------------+
[1780663508.798824] [server:266  :0]   | perftest intra-node cfg#1 | remote memory write by ucp_put* from host memory to host    |
[1780663508.798826] [server:266  :0]   +---------------------------+------------------------------------------+------------------+
[1780663508.798828] [server:266  :0]   |                    0..inf | zero-copy                                | tcp/enp28s0f0np0 |
[1780663508.798830] [server:266  :0]   +---------------------------+------------------------------------------+------------------+
[1780663508.800833] [server:266  :0]   +---------------------------+--------------------------------------------------------------------+
[1780663508.800840] [server:266  :0]   | perftest intra-node cfg#1 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1780663508.800842] [server:266  :0]   +---------------------------+-------------------------------------------------+------------------+
[1780663508.800844] [server:266  :0]   |                    0..inf | software emulation                              | tcp/enp28s0f0np0 |
[1780663508.800846] [server:266  :0]   +---------------------------+-------------------------------------------------+------------------+
[1780663508.805536] [server:266  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.805543] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* from host memory                                                                |
[1780663508.805545] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.805547] [server:266  :0]   |                         0 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.805549] [server:266  :0]   |                  1..16515 | zero-copy                                                                                   | tcp/enp28s0f0np0 |
[1780663508.805551] [server:266  :0]   |              16516..27303 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.805553] [server:266  :0]   |                 27304..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.805555] [server:266  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.805557] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.808228] [server:266  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.808233] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send*(fast-completion) from host memory                                               |
[1780663508.808236] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.808238] [server:266  :0]   |                   0..8184 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.808240] [server:266  :0]   |              8185..262143 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.808241] [server:266  :0]   |                  256K..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.808243] [server:266  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.808245] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.813373] [server:266  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.813379] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send*(multi) from host memory                                                         |
[1780663508.813381] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.813383] [server:266  :0]   |                   0..6031 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.813385] [server:266  :0]   |                      6032 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.813387] [server:266  :0]   |                6033..8184 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.813389] [server:266  :0]   |               8185..17790 | zero-copy                                                                                   | tcp/enp28s0f0np0 |
[1780663508.813391] [server:266  :0]   |              17791..18792 | (?) rendezvous fragmented copy-in copy-out                                                  | tcp/enp28s0f0np0 |
[1780663508.813394] [server:266  :0]   |                 18793..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.813396] [server:266  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.813398] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.816165] [server:266  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.816171] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* with reply flag from host memory                                                |
[1780663508.816173] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.816175] [server:266  :0]   |                         0 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.816177] [server:266  :0]   |                  1..16515 | zero-copy                                                                                   | tcp/enp28s0f0np0 |
[1780663508.816179] [server:266  :0]   |              16516..27303 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.816181] [server:266  :0]   |                 27304..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.816183] [server:266  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.816185] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.818825] [server:266  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.818830] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                               |
[1780663508.818832] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.818834] [server:266  :0]   |                   0..8176 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.818836] [server:266  :0]   |              8177..262143 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.818838] [server:266  :0]   |                  256K..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.818840] [server:266  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.818842] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.821613] [server:266  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.821619] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* with reply flag(multi) from host memory                                         |
[1780663508.821622] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.821624] [server:266  :0]   |                   0..6031 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.821626] [server:266  :0]   |               6032..17790 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.821628] [server:266  :0]   |              17791..18792 | (?) rendezvous fragmented copy-in copy-out                                                  | tcp/enp28s0f0np0 |
[1780663508.821630] [server:266  :0]   |                 18793..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.821632] [server:266  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.821634] [server:266  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.822477] [server:266  :0]      ucp_worker.c:1912 UCX  INFO    perftest self cfg#2 rma_am(tcp/enp28s0f0np0)  am(tcp/enp28s0f0np0 cuda_copy/cuda)
[1780663508.824100] [server:266  :0]   +---------------------+-------------------------------------------------------------+
[1780663508.824105] [server:266  :0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to host    |
[1780663508.824107] [server:266  :0]   +---------------------+------------------------------------------+------------------+
[1780663508.824110] [server:266  :0]   |              0..inf | software emulation                       | tcp/enp28s0f0np0 |
[1780663508.824112] [server:266  :0]   +---------------------+------------------------------------------+------------------+
[1780663508.825580] [server:266  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.825584] [server:266  :0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1780663508.825586] [server:266  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.825588] [server:266  :0]   |              0..inf | zero-copy                                         | cuda_copy/cuda |
[1780663508.825590] [server:266  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.831539] [server:266  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.831544] [server:266  :0]   | perftest self cfg#2 | active message by ucp_am_send* from host memory                                   |
[1780663508.831546] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.831548] [server:266  :0]   |                   0 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.831551] [server:266  :0]   |            1..13884 | zero-copy                                                      | tcp/enp28s0f0np0 |
[1780663508.831552] [server:266  :0]   |           13885..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.831554] [server:266  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.831556] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.834737] [server:266  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.834743] [server:266  :0]   | perftest self cfg#2 | active message by ucp_am_send*(fast-completion) from host memory                  |
[1780663508.834745] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.834747] [server:266  :0]   |             0..8184 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.834749] [server:266  :0]   |        8185..262143 | multi-frag copy-in                                             | tcp/enp28s0f0np0 |
[1780663508.834751] [server:266  :0]   |            256K..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.834753] [server:266  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.834755] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.840539] [server:266  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.840545] [server:266  :0]   | perftest self cfg#2 | active message by ucp_am_send*(multi) from host memory                            |
[1780663508.840547] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.840549] [server:266  :0]   |             0..6031 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.840551] [server:266  :0]   |                6032 | multi-frag copy-in                                             | tcp/enp28s0f0np0 |
[1780663508.840553] [server:266  :0]   |          6033..8184 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.840555] [server:266  :0]   |         8185..12725 | zero-copy                                                      | tcp/enp28s0f0np0 |
[1780663508.840557] [server:266  :0]   |           12726..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.840559] [server:266  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.840561] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.843554] [server:266  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.843559] [server:266  :0]   | perftest self cfg#2 | active message by ucp_am_send* with reply flag from host memory                   |
[1780663508.843561] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.843563] [server:266  :0]   |                   0 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.843565] [server:266  :0]   |            1..13884 | zero-copy                                                      | tcp/enp28s0f0np0 |
[1780663508.843567] [server:266  :0]   |           13885..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.843569] [server:266  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.843571] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.846605] [server:266  :0]   +---------------------+-------------------------------------------------------------------------------------+
[1780663508.846610] [server:266  :0]   | perftest self cfg#2 | active message by ucp_am_send* with reply flag(fast-completion) from host memory    |
[1780663508.846612] [server:266  :0]   +---------------------+------------------------------------------------------------------+------------------+
[1780663508.846615] [server:266  :0]   |             0..8176 | short                                                            | tcp/enp28s0f0np0 |
[1780663508.846617] [server:266  :0]   |        8177..262143 | multi-frag copy-in                                               | tcp/enp28s0f0np0 |
[1780663508.846618] [server:266  :0]   |            256K..4M | (?) rendezvous read from remote, cuda_copy, frag cuda            | cuda_copy/cuda   |
[1780663508.846620] [server:266  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda   | cuda_copy/cuda   |
[1780663508.846622] [server:266  :0]   +---------------------+------------------------------------------------------------------+------------------+
[1780663508.849953] [server:266  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.849960] [server:266  :0]   | perftest self cfg#2 | active message by ucp_am_send* with reply flag(multi) from host memory            |
[1780663508.849962] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.849964] [server:266  :0]   |             0..6031 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.849965] [server:266  :0]   |         6032..12725 | multi-frag copy-in                                             | tcp/enp28s0f0np0 |
[1780663508.849967] [server:266  :0]   |           12726..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.849969] [server:266  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.849971] [server:266  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.850959] [server:266  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.850965] [server:266  :0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1780663508.850967] [server:266  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.850969] [server:266  :0]   |              0..inf | zero-copy                                         | cuda_copy/cuda |
[1780663508.850972] [server:266  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.881306] [server:266  :0]   +---------------------------+--------------------------------------------------------------+
[1780663508.881317] [server:266  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send*(multi) from cuda/GPU0         |
[1780663508.881322] [server:266  :0]   +---------------------------+-------------------------------------------+------------------+
[1780663508.881326] [server:266  :0]   |                         0 | short                                     | tcp/enp28s0f0np0 |
[1780663508.881330] [server:266  :0]   |                   1..6737 | copy-in                                   | tcp/enp28s0f0np0 |
[1780663508.881334] [server:266  :0]   |                      6738 | multi-frag copy-in                        | tcp/enp28s0f0np0 |
[1780663508.881338] [server:266  :0]   |                6739..8184 | copy-in                                   | tcp/enp28s0f0np0 |
[1780663508.881342] [server:266  :0]   |               8185..12255 | multi-frag copy-in                        | tcp/enp28s0f0np0 |
[1780663508.881346] [server:266  :0]   |                12256..inf | (?) rendezvous zero-copy read from remote | cuda_ipc/cuda    |
[1780663508.881350] [server:266  :0]   +---------------------------+-------------------------------------------+------------------+
Final:                 10000      7.937    12.408    12.408    80595.68   80595.68       80596       80596

Server:

root@server:/workspace# CUDA_VISIBLE_DEVICES=0 ucx_perftest -t ucp_am_bw -m cuda -s 1048576 -H 256 -n 10000 -w 1000 -p 50000
[1780663469.071324] [server:263  :0]        perftest.c:825  UCX  WARN  CPU affinity is not set (bound to 64 cpus). Performance may be impacted.
Waiting for connection...
Accepted connection from 127.0.0.1:53562
+----------------------------------------------------------------------------------------------------------+
| API:          protocol layer                                                                             |
| Test:         am bandwidth / message rate                                                                |
| Data layout:  (automatic)                                                                                |
| Send memory:  cuda                                                                                       |
| Recv memory:  cuda                                                                                       |
| Message size: 1048576                                                                                    |
| Window size:  32                                                                                         |
| AM header size: 256                                                                                      |
+----------------------------------------------------------------------------------------------------------+
[1780663508.692331] [server:263  :0]     ucp_context.c:2463 UCX  INFO  Version 1.20.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
[1780663508.794429] [server:263  :0]   +---------------------+-------------------------------------------------------------+
[1780663508.794437] [server:263  :0]   | perftest self cfg#0 | remote memory write by ucp_put* from host memory to host    |
[1780663508.794440] [server:263  :0]   +---------------------+--------------------------------------------+----------------+
[1780663508.794442] [server:263  :0]   |       0..4294967295 | short                                      | cuda_copy/cuda |
[1780663508.794444] [server:263  :0]   +---------------------+--------------------------------------------+----------------+
[1780663508.794477] [server:263  :0]   +---------------------+------------------------------------------------------------------------------+
[1780663508.794481] [server:263  :0]   | perftest self cfg#0 | remote memory write by ucp_put*(fast-completion) from host memory to host    |
[1780663508.794483] [server:263  :0]   +---------------------+-------------------------------------------------------------+----------------+
[1780663508.794485] [server:263  :0]   |       0..4294967295 | short                                                       | cuda_copy/cuda |
[1780663508.794488] [server:263  :0]   +---------------------+-------------------------------------------------------------+----------------+
[1780663508.794514] [server:263  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.794518] [server:263  :0]   | perftest self cfg#0 | remote memory write by ucp_put*(multi) from host memory to host    |
[1780663508.794521] [server:263  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.794523] [server:263  :0]   |       0..4294967295 | short                                             | cuda_copy/cuda |
[1780663508.794525] [server:263  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.794561] [server:263  :0]   +---------------------+----------------------------------------------------+
[1780663508.794565] [server:263  :0]   | perftest self cfg#0 | active message by ucp_am_send* from host memory    |
[1780663508.794567] [server:263  :0]   +---------------------+-------------------------------------------------+--+
[1780663508.794569] [server:263  :0]   +---------------------+-------------------------------------------------+--+
[1780663508.794580] [server:263  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.794584] [server:263  :0]   | perftest self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1780663508.794587] [server:263  :0]   +---------------------+-----------------------------------------------------------------+--+
[1780663508.794589] [server:263  :0]   +---------------------+-----------------------------------------------------------------+--+
[1780663508.794774] [server:263  :0]          parser.c:2364 UCX  INFO  UCX_* env variables: UCX_PROTO_INFO=y UCX_TLS=cuda_copy,cuda_ipc,tcp UCX_LOG_LEVEL=info
[1780663508.795276] [server:263  :0]      ucp_worker.c:1912 UCX  INFO    perftest intra-node cfg#1 rma_am(tcp/enp28s0f0np0)  am(tcp/enp28s0f0np0 cuda_ipc/cuda)
[1780663508.798819] [server:263  :0]   +---------------------------+-------------------------------------------------------------+
[1780663508.798824] [server:263  :0]   | perftest intra-node cfg#1 | remote memory write by ucp_put* from host memory to host    |
[1780663508.798826] [server:263  :0]   +---------------------------+------------------------------------------+------------------+
[1780663508.798828] [server:263  :0]   |                    0..inf | zero-copy                                | tcp/enp28s0f0np0 |
[1780663508.798830] [server:263  :0]   +---------------------------+------------------------------------------+------------------+
[1780663508.800863] [server:263  :0]   +---------------------------+--------------------------------------------------------------------+
[1780663508.800868] [server:263  :0]   | perftest intra-node cfg#1 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1780663508.800870] [server:263  :0]   +---------------------------+-------------------------------------------------+------------------+
[1780663508.800872] [server:263  :0]   |                    0..inf | software emulation                              | tcp/enp28s0f0np0 |
[1780663508.800874] [server:263  :0]   +---------------------------+-------------------------------------------------+------------------+
[1780663508.805620] [server:263  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.805626] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* from host memory                                                                |
[1780663508.805628] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.805630] [server:263  :0]   |                         0 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.805633] [server:263  :0]   |                  1..16515 | zero-copy                                                                                   | tcp/enp28s0f0np0 |
[1780663508.805636] [server:263  :0]   |              16516..27303 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.805640] [server:263  :0]   |                 27304..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.805642] [server:263  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.805644] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.808355] [server:263  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.808361] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send*(fast-completion) from host memory                                               |
[1780663508.808363] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.808365] [server:263  :0]   |                   0..8184 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.808368] [server:263  :0]   |              8185..262143 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.808371] [server:263  :0]   |                  256K..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.808375] [server:263  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.808377] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.813492] [server:263  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.813498] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send*(multi) from host memory                                                         |
[1780663508.813500] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.813502] [server:263  :0]   |                   0..6031 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.813505] [server:263  :0]   |                      6032 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.813508] [server:263  :0]   |                6033..8184 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.813512] [server:263  :0]   |               8185..17790 | zero-copy                                                                                   | tcp/enp28s0f0np0 |
[1780663508.813515] [server:263  :0]   |              17791..18792 | (?) rendezvous fragmented copy-in copy-out                                                  | tcp/enp28s0f0np0 |
[1780663508.813518] [server:263  :0]   |                 18793..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.813521] [server:263  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.813540] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.816287] [server:263  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.816292] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* with reply flag from host memory                                                |
[1780663508.816294] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.816296] [server:263  :0]   |                         0 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.816300] [server:263  :0]   |                  1..16515 | zero-copy                                                                                   | tcp/enp28s0f0np0 |
[1780663508.816302] [server:263  :0]   |              16516..27303 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.816305] [server:263  :0]   |                 27304..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.816309] [server:263  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.816311] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.818961] [server:263  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.818966] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* with reply flag(fast-completion) from host memory                               |
[1780663508.818968] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.818970] [server:263  :0]   |                   0..8176 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.818972] [server:263  :0]   |              8177..262143 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.818977] [server:263  :0]   |                  256K..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.818980] [server:263  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.818982] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.821803] [server:263  :0]   +---------------------------+----------------------------------------------------------------------------------------------------------------+
[1780663508.821809] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send* with reply flag(multi) from host memory                                         |
[1780663508.821811] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.821813] [server:263  :0]   |                   0..6031 | short                                                                                       | tcp/enp28s0f0np0 |
[1780663508.821817] [server:263  :0]   |               6032..17790 | multi-frag copy-in                                                                          | tcp/enp28s0f0np0 |
[1780663508.821819] [server:263  :0]   |              17791..18792 | (?) rendezvous fragmented copy-in copy-out                                                  | tcp/enp28s0f0np0 |
[1780663508.821822] [server:263  :0]   |                 18793..4M | (?) rendezvous cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda          | cuda_ipc/cuda    |
[1780663508.821826] [server:263  :0]   |              4194305..inf | (?) rendezvous pipeline cuda_copy, flushed write to remote, frag cuda, cuda_copy, frag cuda | cuda_ipc/cuda    |
[1780663508.821828] [server:263  :0]   +---------------------------+---------------------------------------------------------------------------------------------+------------------+
[1780663508.822477] [server:263  :0]      ucp_worker.c:1912 UCX  INFO    perftest self cfg#2 rma_am(tcp/enp28s0f0np0)  am(tcp/enp28s0f0np0 cuda_copy/cuda)
[1780663508.824102] [server:263  :0]   +---------------------+-------------------------------------------------------------+
[1780663508.824106] [server:263  :0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to host    |
[1780663508.824109] [server:263  :0]   +---------------------+------------------------------------------+------------------+
[1780663508.824111] [server:263  :0]   |              0..inf | software emulation                       | tcp/enp28s0f0np0 |
[1780663508.824113] [server:263  :0]   +---------------------+------------------------------------------+------------------+
[1780663508.825568] [server:263  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.825573] [server:263  :0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1780663508.825575] [server:263  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.825577] [server:263  :0]   |              0..inf | zero-copy                                         | cuda_copy/cuda |
[1780663508.825579] [server:263  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.831537] [server:263  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.831543] [server:263  :0]   | perftest self cfg#2 | active message by ucp_am_send* from host memory                                   |
[1780663508.831545] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.831547] [server:263  :0]   |                   0 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.831549] [server:263  :0]   |            1..13884 | zero-copy                                                      | tcp/enp28s0f0np0 |
[1780663508.831551] [server:263  :0]   |           13885..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.831553] [server:263  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.831555] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.834738] [server:263  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.834743] [server:263  :0]   | perftest self cfg#2 | active message by ucp_am_send*(fast-completion) from host memory                  |
[1780663508.834745] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.834748] [server:263  :0]   |             0..8184 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.834750] [server:263  :0]   |        8185..262143 | multi-frag copy-in                                             | tcp/enp28s0f0np0 |
[1780663508.834752] [server:263  :0]   |            256K..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.834754] [server:263  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.834756] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.840577] [server:263  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.840582] [server:263  :0]   | perftest self cfg#2 | active message by ucp_am_send*(multi) from host memory                            |
[1780663508.840584] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.840586] [server:263  :0]   |             0..6031 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.840588] [server:263  :0]   |                6032 | multi-frag copy-in                                             | tcp/enp28s0f0np0 |
[1780663508.840590] [server:263  :0]   |          6033..8184 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.840592] [server:263  :0]   |         8185..12725 | zero-copy                                                      | tcp/enp28s0f0np0 |
[1780663508.840594] [server:263  :0]   |           12726..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.840596] [server:263  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.840598] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.843564] [server:263  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.843569] [server:263  :0]   | perftest self cfg#2 | active message by ucp_am_send* with reply flag from host memory                   |
[1780663508.843571] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.843573] [server:263  :0]   |                   0 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.843574] [server:263  :0]   |            1..13884 | zero-copy                                                      | tcp/enp28s0f0np0 |
[1780663508.843592] [server:263  :0]   |           13885..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.843594] [server:263  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.843596] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.846601] [server:263  :0]   +---------------------+-------------------------------------------------------------------------------------+
[1780663508.846606] [server:263  :0]   | perftest self cfg#2 | active message by ucp_am_send* with reply flag(fast-completion) from host memory    |
[1780663508.846609] [server:263  :0]   +---------------------+------------------------------------------------------------------+------------------+
[1780663508.846611] [server:263  :0]   |             0..8176 | short                                                            | tcp/enp28s0f0np0 |
[1780663508.846613] [server:263  :0]   |        8177..262143 | multi-frag copy-in                                               | tcp/enp28s0f0np0 |
[1780663508.846615] [server:263  :0]   |            256K..4M | (?) rendezvous read from remote, cuda_copy, frag cuda            | cuda_copy/cuda   |
[1780663508.846616] [server:263  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda   | cuda_copy/cuda   |
[1780663508.846618] [server:263  :0]   +---------------------+------------------------------------------------------------------+------------------+
[1780663508.849969] [server:263  :0]   +---------------------+-----------------------------------------------------------------------------------+
[1780663508.849974] [server:263  :0]   | perftest self cfg#2 | active message by ucp_am_send* with reply flag(multi) from host memory            |
[1780663508.849976] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.849994] [server:263  :0]   |             0..6031 | short                                                          | tcp/enp28s0f0np0 |
[1780663508.849996] [server:263  :0]   |         6032..12725 | multi-frag copy-in                                             | tcp/enp28s0f0np0 |
[1780663508.849998] [server:263  :0]   |           12726..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda   |
[1780663508.850000] [server:263  :0]   |        4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda   |
[1780663508.850002] [server:263  :0]   +---------------------+----------------------------------------------------------------+------------------+
[1780663508.850972] [server:263  :0]   +---------------------+--------------------------------------------------------------------+
[1780663508.850977] [server:263  :0]   | perftest self cfg#2 | remote memory write by ucp_put* from host memory to cuda/dev[0]    |
[1780663508.850979] [server:263  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.850981] [server:263  :0]   |              0..inf | zero-copy                                         | cuda_copy/cuda |
[1780663508.850987] [server:263  :0]   +---------------------+---------------------------------------------------+----------------+
[1780663508.882238] [server:263  :0]   +---------------------------+-------------------------------------------------------------+
[1780663508.882250] [server:263  :0]   | perftest intra-node cfg#1 | remote memory write by ucp_put* from host memory to cuda    |
[1780663508.882254] [server:263  :0]   +---------------------------+------------------------------------------+------------------+
[1780663508.882258] [server:263  :0]   |                    0..inf | software emulation                       | tcp/enp28s0f0np0 |
[1780663508.882262] [server:263  :0]   +---------------------------+------------------------------------------+------------------+
[1780663508.889984] [server:263  :0]   +---------------------------+----------------------------------------------------------+
[1780663508.889991] [server:263  :0]   | perftest intra-node cfg#1 | rendezvous data fetch(multi) into cuda/GPU0 from cuda    |
[1780663508.889994] [server:263  :0]   +---------------------------+------------------------------------------+---------------+
[1780663508.889996] [server:263  :0]   |                         0 | no data fetch                            |               |
[1780663508.889999] [server:263  :0]   |                    1..inf | zero-copy read from remote               | cuda_ipc/cuda |
[1780663508.890001] [server:263  :0]   +---------------------------+------------------------------------------+---------------+
[1780663508.899661] [server:263  :0]   +---------------------------+--------------------------------------------------------------+
[1780663508.899674] [server:263  :0]   | perftest intra-node cfg#1 | active message by ucp_am_send*(multi) from cuda/GPU0         |
[1780663508.899676] [server:263  :0]   +---------------------------+-------------------------------------------+------------------+
[1780663508.899677] [server:263  :0]   |                         0 | short                                     | tcp/enp28s0f0np0 |
[1780663508.899679] [server:263  :0]   |                   1..6737 | copy-in                                   | tcp/enp28s0f0np0 |
[1780663508.899681] [server:263  :0]   |                      6738 | multi-frag copy-in                        | tcp/enp28s0f0np0 |
[1780663508.899683] [server:263  :0]   |                6739..8184 | copy-in                                   | tcp/enp28s0f0np0 |
[1780663508.899685] [server:263  :0]   |               8185..12255 | multi-frag copy-in                        | tcp/enp28s0f0np0 |
[1780663508.899687] [server:263  :0]   |                12256..inf | (?) rendezvous zero-copy read from remote | cuda_ipc/cuda    |
[1780663508.899688] [server:263  :0]   +---------------------------+-------------------------------------------+------------------+

It is working with UCX Perf test on 127.0.0.1.

Mind you I’m working with NVIDIA tensorrt llm container.
Should I try some different ips for inside the container?

All the fragments are on same node, same container, launched from same script

I also tried with UCX_RNDV_THRESH=0

How the cfg are getting setuped:

[1780664970.574887] [server:28428:0]          parser.c:2364 UCX  INFO  UCX_* env variables: UCX_RNDV_THRESH=0 UCX_ADDRESS_DEBUG_INFO=y UCX_CUDA_IPC_FAILURE=TRACE UCX_PROTO_INFO=y UCX_CUDA_COPY_FAILURE=TRACE UCX_TCP_TX_SEG_SIZE=2M UCX_TLS=cuda_copy,cuda_ipc,tcp UCX_TCP_RX_SEG_SIZE=2M UCX_LOG_LEVEL=info
[1780664970.584835] [server:28428:0]     ucp_context.c:2463 UCX  INFO  Version 1.20.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
[1780664970.592293] [server:28428:0]      ucp_worker.c:1912 UCX  INFO    ucp_context_0 self cfg#1 tag(tcp/enp28s0f0np0 cuda_copy/cuda)
[1780664970.633704] [server:28428:0]   +--------------------------+---------------------------------------------------------------------------------+
[1780664970.633707] [server:28428:0]   | ucp_context_0 self cfg#1 | tagged message by ucp_tag_send* from host memory                                |
[1780664970.633708] [server:28428:0]   +--------------------------+----------------------------------------------------------------+----------------+
[1780664970.633709] [server:28428:0]   |                        0 | (?) rendezvous no data fetch                                   |                |
[1780664970.633710] [server:28428:0]   |                    1..4M | (?) rendezvous read from remote, cuda_copy, frag cuda          | cuda_copy/cuda |
[1780664970.633711] [server:28428:0]   |             4194305..inf | (?) rendezvous pipeline read from remote, cuda_copy, frag cuda | cuda_copy/cuda |
[1780664970.633711] [server:28428:0]   +--------------------------+----------------------------------------------------------------+----------------+

[1780664995.898618] [server:28428:1]   +--------------------------+-----------------------------------------------------------------+--+
[1780664996.851670] [server:28428:2]      ucp_worker.c:1912 UCX  INFO    client_server inter-node cfg#1 am(tcp/lo)
[1780664996.862852] [server:28428:2]   +--------------------------------+----------------------------------------------------------+
[1780664996.862853] [server:28428:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* from host memory          |
[1780664996.862854] [server:28428:2]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.862855] [server:28428:2]   |                              0 | (?) rendezvous no data fetch                    |        |
[1780664996.862855] [server:28428:2]   |                      1..447669 | (?) rendezvous zero-copy                        | tcp/lo |
[1780664996.862856] [server:28428:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote | tcp/lo |
[1780664996.862856] [server:28428:2]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.864993] [server:28428:2]   +--------------------------------+--------------------------------------------------------------------+
[1780664996.864994] [server:28428:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1780664996.864995] [server:28428:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.864995] [server:28428:2]   |                              0 | (?) rendezvous no data fetch                              |        |
[1780664996.864996] [server:28428:2]   |                      1..447669 | (?) rendezvous zero-copy                                  | tcp/lo |
[1780664996.864997] [server:28428:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote           | tcp/lo |
[1780664996.864997] [server:28428:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.873358] [server:28428:2]      ucp_worker.c:1912 UCX  INFO    client_server inter-node cfg#1 am(tcp/lo)
[1780664996.878818] [server:28428:2]   +--------------------------------+----------------------------------------------------------+
[1780664996.878820] [server:28428:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* from host memory          |
[1780664996.878821] [server:28428:2]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.878821] [server:28428:2[info] [ucx_context.cpp:520] Connection request received to 0.0.0.0:10071 from client at address 127.0.0.1:58387
]   |                              0 | (?) rendezvous no data fetch                    |        |
[1780664996.878827] [server:28428:2]   |                      1..447669 | (?) rendezvous zero-copy                        | tcp/lo |
[1780664996.878827] [server:28428:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote | tcp/lo |
[1780664996.878828] [server:28428:2]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.880865] [server:28428:2]   +--------------------------------+--------------------------------------------------------------------+
[1780664996.880866] [server:28428:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1780664996.880867] [server:28428:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.880868] [server:28428:2]   |                              0 | (?) rendezvous no data fetch                              |        |
[1780664996.880868] [server:28428:2]   |                      1..447669 | (?) rendezvous zero-copy                                  | tcp/lo |
[1780664996.880869] [server:28428:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote           | tcp/lo |
[1780664996.880869] [server:28428:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.891182] [server:28428:2]      ucp_worker.c:1912 UCX  INFO    client_server inter-node cfg#1 am(tcp/lo)
[1780664996.896928] [server:28428:2]   +--------------------------------+----------------------------------------------------------+
[1780664996.896930] [server:28428:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* from host memory          |
[1780664996.896930] [server:28428:2]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.896931] [server:28428:2]   |                              0 | (?) rendezvous no data fetch                    |        |
[1780664996.896931] [server:28428:2]   |                      1..447669 | (?) rendezvous zero-copy                        | tcp/lo |
[1780664996.896932] [server:28428:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote | tcp/lo |
[1780664996.896932] [server:28428:2]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.898872] [server:28428:2]   +--------------------------------+--------------------------------------------------------------------+
[1780664996.898874] [server:28428:2]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1780664996.898874] [server:28428:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.898875] [server:28428:2]   |                              0 | (?) rendezvous no data fetch                              |        |
[1780664996.898875] [server:28428:2]   |                      1..447669 | (?) rendezvous zero-copy                                  | tcp/lo |
[1780664996.898876] [server:28428:2]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote           | tcp/lo |
[1780664996.898876] [server:28428:2]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.898902] [server:28428:1]   +--------------------------------+----------------------------------------------------+
[1780664996.898904] [server:28428:1]   | client_server inter-node cfg#1 | active message by ucp_am_send* from host memory    |
[1780664996.898904] [server:28428:1]   +--------------------------------+-------------------------------------------------+--+
[1780664996.898905] [server:28428:1]   +--------------------------------+-------------------------------------------------+--+
[1780664996.898914] [server:28428:1]   +--------------------------------+--------------------------------------------------------------------+
[1780664996.898920] [server:28428:1]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1780664996.898920] [server:28428:1]   +--------------------------------+-----------------------------------------------------------------+--+
[1780664996.898921] [server:28428:1]   +--------------------------------+-----------------------------------------------------------------+--+
[1780664996.899502] [server:28428:1]      ucp_worker.c:1912 UCX  INFO  client_server intra-node cfg#2 am(tcp/lo)
[1780664996.904261] [server:28428:1]   +--------------------------------+----------------------------------------------------------+
[1780664996.904262] [server:28428:1]   | client_server intra-node cfg#2 | active message by ucp_am_send* from host memory          |
[1780664996.904263] [server:28428:1]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.904263] [server:28428:1]   |                              0 | (?) rendezvous no data fetch                    |        |
[1780664996.904264] [server:28428:1]   |                      1..447669 | (?) rendezvous zero-copy                        | tcp/lo |
[1780664996.904264] [server:28428:1]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote | tcp/lo |
[1780664996.904265] [server:28428:1]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.906005] [server:28428:1]   +--------------------------------+--------------------------------------------------------------------+
[1780664996.906006] [server:28428:1]   | client_server intra-node cfg#2 | active message by ucp_am_send* with reply flag from host memory    |
[1780664996.906006] [server:28428:1]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.906007] [server:28428:1]   |                              0 | (?) rendezvous no data fetch                              |        |
[1780664996.906007] [server:28428:1]   |                      1..447669 | (?) rendezvous zero-copy                                  | tcp/lo |
[1780664996.906007] [server:28428:1]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote           | tcp/lo |
[1780664996.906008] [server:28428:1]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664996.915746] [server:28428:1]      ucp_worker.c:1912 UCX  INFO      client_server inter-node cfg#3 am(tcp/lo)
[1780664996.919286] [server:28428:1]   +--------------------------------+----------------------------------------------------------+
[1780664996.919287] [server:28428:1]   | client_server inter-node cfg#3 | active message by ucp_am_send* from host memory          |
[1780664996.919288] [server:28428:1]   +--------------------------------+-------------------------------------------------+--------+
[1780664996.919288] [server:28428:1]   |                              0 | (?) rendezvous no data fetch                    |        |
[1780664996.919289] [server:28428:1]   |                      1..447669 | (?) rendezvous zero-copy                        | tcp/lo |
[1780664996.919289] [server:28428:1]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote | tcp/lo |
[1780664996.919290] [server:28428:1]   +--------------------------------+-------------------------------------------------+--------+

And cfg3 is being used to send:

--+-----------------------------------------------------------+--------+
[1780664998.977414] [server:28428:1]   |                              0 | (?) rendezvous no data fetch                              |        |
[1780664998.977416] [server:28428:1]   |                      1..447669 | (?) rendezvous zero-copy                                  | tcp/lo |
[1780664998.977417] [server:28428:1]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote           | tcp/lo |
[1780664998.977419] [server:28428:1]   +--------------------------------+-----------------------------------------------------------+--------+
[1780664998.977952] [server:28428:1]     ucp_context.c:2463 UCX  INFO  Version 1.20.0 (loaded from /usr/local/ucx//lib/libucp.so.0)
[1780665169.844506] [server:28428:3]   +--------------------------------+---------------------------------------------------+
[1780665169.844514] [server:28428:3]   | client_server inter-node cfg#1 | rendezvous data fetch into cuda/GPU0 from cuda    |
[1780665169.844519] [server:28428:3]   +--------------------------------+------------------------------------------+--------+
[1780665169.844523] [server:28428:3]   |                              0 | no data fetch                            |        |
[1780665169.844527] [server:28428:3]   |                         1..inf | (?) fragmented copy-in copy-out          | tcp/lo |
[1780665169.844530] [server:28428:3]   +--------------------------------+------------------------------------------+--------+
[1780665169.878439] [server:28428:3]   +--------------------------------+--------------------------------------------------------+
[1780665169.878441] [server:28428:3]   | client_server inter-node cfg#3 | active message by ucp_am_send*(rndv) from cuda/GPU0    |
[1780665169.878442] [server:28428:3]   +--------------------------------+-----------------------------------------------+--------+
[1780665169.878443] [server:28428:3]   |                              0 | (?) rendezvous no data fetch                  |        |
[1780665169.878444] [server:28428:3]   |                         1..inf | (?) rendezvous fragmented copy-in copy-out    | tcp/lo |
[1780665169.878446] [server:28428:3]   +--------------------------------+-----------------------------------------------+--------+
[1780665170.088196] [server:28428:3]   +--------------------------------+------------------------------------------------+
[1780665170.088199] [server:28428:3]   | client_server inter-node cfg#3 | rendezvous data send from cuda/GPU0 to cuda    |
[1780665170.088200] [server:28428:3]   +--------------------------------+---------------------------------------+--------+
[1780665170.088201] [server:28428:3]   |                         0..inf | fragmented copy-in copy-out           | tcp/lo |
[1780665170.088202] [server:28428:3]   +--------------------------------+---------------------------------------+--------+
[1780665170.089481] [server:28428:3]   +--------------------------------+--------------------------------------------------------------+
[1780665170.089483] [server:28428:3]   | client_server inter-node cfg#3 | active message by ucp_am_send*(rndv) from iov host memory    |
[1780665170.089484] [server:28428:3]   +--------------------------------+-----------------------------------------------------+--------+
[1780665170.089485] [server:28428:3]   |                              0 | (?) rendezvous no data fetch                        |        |
[1780665170.089486] [server:28428:3]   |                      1..447669 | (?) rendezvous zero-copy                            | tcp/lo |
[1780665170.089487] [server:28428:3]   |                    447670..inf | (?) rendezvous zero-copy fenced write to remote     | tcp/lo |
[1780665170.089488] [server:28428:3]   +--------------------------------+-----------------------------------------------------+--------+
[1780665170.092008] [server:28428:3]   +--------------------------------+--------------------------------------------------------------+

My flags starting in the script:

export MALLOC_CHECK_=3
# export UCX_TLS=all

export UCX_PROTO_INFO=y
export CUDA_VISIBLE_DEVICES=0
export HOLOSCAN_UCX_DEVICE_ID=0

export UCX_TLS=cuda_copy,cuda_ipc,tcp
export UCX_LOG_LEVEL=info
export UCX_RNDV_THRESH=0

export UCX_ADDRESS_DEBUG_INFO=y
export UCX_CUDA_COPY_FAILURE=TRACE
export UCX_CUDA_IPC_FAILURE=TRACE
export UCX_TCP_TX_SEG_SIZE=2M
export UCX_TCP_RX_SEG_SIZE=2M
export HOLOSCAN_STOP_ON_DEADLOCK_TIMEOUT=300000
export HOLOSCAN_UCX_NETWORK_CONNECTION_TIMEOUT=300000

The driver is started with

echo "Starting API Tier (Driver)..."
python frag_tensorrt_rag_new.py --driver --worker --fragments=api_tier --worker-address=127.0.0.1:50000 2>&1 | grep --line-buffered -v 'Received null entity in transmitter' > logs/api_tier.log &

and every other fragment as well

echo "Starting Worker for embed_node_router_fragment on port 50001..."
python frag_tensorrt_rag_new.py --worker --fragments=embed_node_router_fragment --worker-address=127.0.0.1:50001 2>&1 | grep --line-buffered -v 'Received null entity in transmitter' > logs/embed_node_router_fragment.log &
sleep 0.1
echo "Starting Worker for embed_node_collector_fragment on port 50002..."
python frag_tensorrt_rag_new.py --worker --fragments=embed_node_collector_fragment --worker-address=127.0.0.1:50002 2>&1 | grep --line-buffered -v 'Received null entity in transmitter' > logs/embed_node_collector_fragment.log &
sleep 0.1

Appreciate the help a lot!!

utkarsh02t · June 11, 2026, 9:47am

@grelee

I was able to fix the inter-node problem, using the ip provided by ip a (as suggested by you).

Now, I am able to see the nodes come up as intra-node
Still though, I wasn’t able to transfer with cuda_copy or cuda_ipc

Entire log: preprocess_node__standard_i0.log · GitHub

export CUDA_VISIBLE_DEVICES=0
export HOLOSCAN_UCX_DEVICE_ID=0

export UCX_PROTO_INFO=y
export UCX_TLS=cuda_ipc,cuda_copy,tcp
export UCX_RNDV_THRESH=0
export UCX_LOG_LEVEL=debug
export UCX_TCP_TX_SEG_SIZE=2M
export UCX_TCP_RX_SEG_SIZE=2M
export HOLOSCAN_STOP_ON_DEADLOCK_TIMEOUT=3000000
export HOLOSCAN_UCX_NETWORK_CONNECTION_TIMEOUT=3000000

For the self cfg, I see the lanes are being setup with cuda_copy but for any one else its tcp

utkarsh02t · June 11, 2026, 3:32pm

A small reproducible example:

from holoscan.core import Application, Fragment, Operator, OperatorSpec
from holoscan.conditions import CountCondition 
import cupy as cp


class TensorGenOp(Operator):
    def __init__(self, fragment, *args, **kwargs):
        super().__init__(fragment, *args, **kwargs)

    def setup(self, spec: OperatorSpec):
        spec.output("out")

    def compute(self, op_input, op_output, *args):
        gpu_array = cp.ones((512, 512), dtype=cp.float32) * 42.0
        op_output.emit(gpu_array, "out")
        print("[TensorGenOp] Emitted GPU Tensor.")


class TensorProcessOp(Operator):
    def __init__(self, fragment, *args, **kwargs):
        super().__init__(fragment, *args, **kwargs)

    def setup(self, spec: OperatorSpec):
        spec.input("in")

    def compute(self, op_input, op_output, *args):
        gpu_tensor = op_input.receive("in")
        cupy_array = cp.asarray(gpu_tensor)
        result = cupy_array * 2.0
        print(f"[TensorProcessOp] Processed mean value: {result.mean().item()}")


class Fragment1(Fragment):
    def compose(self):
        gen_op = TensorGenOp(
            self, 
            CountCondition(self, count=5, name="count_cond"), 
            name="gen_op"
        )
        
        self.add_operator(gen_op)

class Fragment2(Fragment):
    def compose(self):
        process_op = TensorProcessOp(self, name="process_op")
        self.add_operator(process_op)

class MultiFragmentGPUApp(Application):
    def compose(self):
        fragment1 = Fragment1(self, name="fragment1")
        fragment2 = Fragment2(self, name="fragment2")
        
        self.add_fragment(fragment1)
        self.add_fragment(fragment2)
        
        self.add_flow(fragment1, fragment2, {("gen_op.out", "process_op.in")})

if __name__ == "__main__":
    app = MultiFragmentGPUApp()
    app.run()

Fragment1:

root@server:/workspace/experimental_dags/pure_holo# python3 multi_frag.py --driver --worker --fragment fragment1 --address 10.129.2.221:5000
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [gxf_executor.cpp:566] [fragment1] Creating context
[info] [gxf_executor.cpp:566] [fragment2] Creating context
[info] [app_driver.cpp:1375] Launching the driver/health checking service
[info] [app_driver.cpp:1394] Launching the worker service
[info] [server.cpp:88] AppDriverServer listening on 0.0.0.0:5000
[info] [gxf_executor.cpp:566] Creating context
[info] [server.cpp:195] AppWorkerServer listening on 0.0.0.0:10000
[info] [server.cpp:196] AppWorkerServer/Health checking server listening on 0.0.0.0:10000
[info] [service_impl.cpp:87] Connected from the client address: ipv4:10.129.2.221:52310 (worker address: 10.129.2.221:10000)
[info] [client.cpp:81] FragmentAllocation response (10.129.2.221:5000): Fragment allocation request is accepted.
[info] [app_driver.cpp:1232] 1/2 fragments scheduled. Awaiting remaining worker connections.
- fragment_name: fragment2, cpu: -1, cpu_limit: -1, gpu: -1, gpu_limit: -1, memory: 0, memory_limit: 0, shared_memory: 0, shared_memory_limit: 0, gpu_memory: 0, gpu_memory_limit: 0

[info] [service_impl.cpp:87] Connected from the client address: ipv4:10.129.2.221:52314 (worker address: 10.129.2.221:10001)
[info] [app_driver.cpp:1019] Fragment schedule is available:
[info] [app_driver.cpp:1021]   Fragment 'fragment2' => Worker '10.129.2.221:10000'
[info] [app_driver.cpp:1021]   Fragment 'fragment1' => Worker '10.129.2.221:10001'
[info] [client.cpp:103] GetFragmentInfo response (10.129.2.221:10001) received
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [client.cpp:103] GetFragmentInfo response (10.129.2.221:10000) received
[info] [service_impl.cpp:41] Number of ports requested: 1
[info] [client.cpp:248] FragmentExecution response (10.129.2.221:10001): Fragment Execution request processed successfully
[warning] [app_driver.cpp:106] UCX_TLS is set to 'cuda_ipc,cuda_copy,tcp' which does not disable CUDA IPC. You might need to disable the cuda_ipc transport (e.g. UCX_TLS=^cuda_ipc) if you run into issues running this distributed application.
[info] [client.cpp:248] FragmentExecution response (10.129.2.221:10000): Fragment Execution request processed successfully
[info] [app_worker.cpp:228] Launching fragment: fragment2
[info] [gxf_executor.cpp:2780] Activating Graph...
[info] [entity_pool.cpp:134] EntityPool::GetShared: created new shared pool for context 0x22c7c040 with capacity 256
[1781189945.898545] [server:2870 :0]   +--------------------------+----------------------------------------------------+
[1781189945.898556] [server:2870 :0]   | client_server self cfg#0 | active message by ucp_am_send* from host memory    |
[1781189945.898558] [server:2870 :0]   +--------------------------+-------------------------------------------------+--+
[1781189945.898561] [server:2870 :0]   +--------------------------+-------------------------------------------------+--+
[1781189945.898574] [server:2870 :0]   +--------------------------+--------------------------------------------------------------------+
[1781189945.898578] [server:2870 :0]   | client_server self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1781189945.898580] [server:2870 :0]   +--------------------------+-----------------------------------------------------------------+--+
[1781189945.898582] [server:2870 :0]   +--------------------------+-----------------------------------------------------------------+--+
[1781189945.901731] [server:2870 :0]   +--------------------------+----------------------------------------------------+
[1781189945.901738] [server:2870 :0]   | client_server self cfg#0 | active message by ucp_am_send* from host memory    |
[1781189945.901740] [server:2870 :0]   +--------------------------+-------------------------------------------------+--+
[1781189945.901742] [server:2870 :0]   +--------------------------+-------------------------------------------------+--+
[1781189945.901770] [server:2870 :0]   +--------------------------+--------------------------------------------------------------------+
[1781189945.901773] [server:2870 :0]   | client_server self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1781189945.901775] [server:2870 :0]   +--------------------------+-----------------------------------------------------------------+--+
[1781189945.901777] [server:2870 :0]   +--------------------------+-----------------------------------------------------------------+--+
[info] [ucx_context.cpp:700] UcxContext::register_am_recv_legacy_callback
[info] [ucx_context.cpp:619] Server is listening on IP 0.0.0.0 port 10002. Waiting for connection...
[info] [event_based_scheduler.cpp:221] Adding thread to default thread pool without pin cores
[info] [ucx_context.cpp:982] UcxContext::start_server
[info] [ucx_context.cpp:67] Ucx Context [cid: 25]: GPUDevice value found and cached. dev_id: 0.
[info] [ucx_receiver.cpp:174] Ucx Receiver [cid: 9]: GPUDevice value found and cached. dev_id: 0
[info] [gxf_executor.cpp:2921] [fragment2] Running Graph...
[info] [gxf_executor.cpp:2923] [fragment2] Waiting for completion...
[info] [event_based_scheduler.cpp:771] Event Based scheduler started worker thread [pool name: default_pool, thread uid: 0]
[info] [ucx_context.cpp:520] Connection request received to 0.0.0.0:10002 from client at address 10.129.2.221:55395
[1781189946.918274] [server:2870 :1]   +--------------------------------+---------------------------------------------------------------+
[1781189946.918281] [server:2870 :1]   | client_server intra-node cfg#1 | active message by ucp_am_send* from host memory               |
[1781189946.918283] [server:2870 :1]   +--------------------------------+--------------------------------------------+------------------+
[1781189946.918286] [server:2870 :1]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781189946.918288] [server:2870 :1]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781189946.918290] [server:2870 :1]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781189946.918292] [server:2870 :1]   +--------------------------------+--------------------------------------------+------------------+
[1781189946.920265] [server:2870 :1]   +--------------------------------+--------------------------------------------------------------------+
[1781189946.920272] [server:2870 :1]   | client_server intra-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1781189946.920274] [server:2870 :1]   +--------------------------------+-------------------------------------------------+------------------+
[1781189946.920276] [server:2870 :1]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781189946.920280] [server:2870 :1]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781189946.920284] [server:2870 :1]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781189946.920290] [server:2870 :1]   +--------------------------------+-------------------------------------------------+------------------+
[1781189947.269310] [server:2870 :2]   +--------------------------------+----------------------------------------------------+
[1781189947.269332] [server:2870 :2]   | client_server intra-node cfg#1 | rendezvous data fetch into cuda/GPU0 from cuda     |
[1781189947.269340] [server:2870 :2]   +--------------------------------+---------------------------------+------------------+
[1781189947.269348] [server:2870 :2]   |                              0 | no data fetch                   |                  |
[1781189947.269355] [server:2870 :2]   |                         1..inf | (?) fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781189947.269363] [server:2870 :2]   +--------------------------------+---------------------------------+------------------+
[TensorProcessOp] Processed mean value: 84.0
[TensorProcessOp] Processed mean value: 84.0
[TensorProcessOp] Processed mean value: 84.0
[TensorProcessOp] Processed mean value: 84.0
[TensorProcessOp] Processed mean value: 84.0
[info] [service_impl.cpp:283] Worker execution succeeded (worker id: 10.129.2.221:10001)
[info] [server.cpp:150] Closed connection to worker 10.129.2.221:10001
[info] [app_driver.cpp:1261] Worker 10.129.2.221:10001 has finished execution. Remaining: 1
[info] [event_based_scheduler.cpp:1575] Stopping all async jobs
[info] [event_based_scheduler.cpp:753] Dispatcher thread has stopped checking jobs
[info] [event_based_scheduler.cpp:1554] Waiting to join all async threads
[info] [event_based_scheduler.cpp:1116] Async event handler thread exiting.
[info] [event_based_scheduler.cpp:1561] Waiting to join max duration thread
[info] [event_based_scheduler.cpp:1569] All async worker threads joined, deactivating all entities
[info] [event_based_scheduler.cpp:1534] Event Based scheduler finished.
[info] [gxf_executor.cpp:2930] [fragment2] Deactivating Graph...
[info] [ucx_context.cpp:113] UcxContext: Initiating graceful shutdown
[info] [event_based_scheduler.cpp:1278] Total execution time of Event Based scheduler : 6509.692379 ms

[info] [gxf_executor.cpp:2939] [fragment2] Graph execution finished.
[info] [app_worker.cpp:308] Worker execution finished
[info] [service_impl.cpp:283] Worker execution succeeded (worker id: 10.129.2.221:10000)
[info] [client.cpp:122] WorkerExecutionFinished response (10.129.2.221:5000): Worker execution succeeded (worker id: 10.129.2.221:10000)
[info] [server.cpp:150] Closed connection to worker 10.129.2.221:10000
[info] [app_driver.cpp:1261] Worker 10.129.2.221:10000 has finished execution. Remaining: 0
[info] [app_driver.cpp:1264] All workers have finished execution

Fragment2:

root@server:/workspace/experimental_dags/pure_holo# python3 multi_frag.py --worker --fragment fragment2 --address 10.129.2.221:5000
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [gxf_executor.cpp:566] [fragment1] Creating context
[info] [gxf_executor.cpp:566] [fragment2] Creating context
[info] [app_driver.cpp:1394] Launching the worker service
[info] [server.cpp:195] AppWorkerServer listening on 0.0.0.0:10001
[info] [server.cpp:196] AppWorkerServer/Health checking server listening on 0.0.0.0:10001
[info] [client.cpp:81] FragmentAllocation response (10.129.2.221:5000): Fragment allocation request is accepted.
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [service_impl.cpp:41] Number of ports requested: 0
[warning] [app_driver.cpp:106] UCX_TLS is set to 'cuda_ipc,cuda_copy,tcp' which does not disable CUDA IPC. You might need to disable the cuda_ipc transport (e.g. UCX_TLS=^cuda_ipc) if you run into issues running this distributed application.
[info] [gxf_executor.cpp:566] Creating context
[info] [gxf_executor.cpp:566] Creating context
[info] [gxf_executor.cpp:601] Destroying context
[info] [app_worker.cpp:228] Launching fragment: fragment1
[info] [gxf_executor.cpp:2780] Activating Graph...
[info] [entity_pool.cpp:134] EntityPool::GetShared: created new shared pool for context 0x24271830 with capacity 256
[1781189945.904492] [server:2978 :0]   +--------------------------+----------------------------------------------------+
[1781189945.904503] [server:2978 :0]   | client_server self cfg#0 | active message by ucp_am_send* from host memory    |
[1781189945.904506] [server:2978 :0]   +--------------------------+-------------------------------------------------+--+
[1781189945.904508] [server:2978 :0]   +--------------------------+-------------------------------------------------+--+
[1781189945.904522] [server:2978 :0]   +--------------------------+--------------------------------------------------------------------+
[1781189945.904526] [server:2978 :0]   | client_server self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1781189945.904528] [server:2978 :0]   +--------------------------+-----------------------------------------------------------------+--+
[1781189945.904530] [server:2978 :0]   +--------------------------+-----------------------------------------------------------------+--+
[1781189946.904844] [server:2978 :0]   +--------------------------------+----------------------------------------------------+
[1781189946.904852] [server:2978 :0]   | client_server inter-node cfg#1 | active message by ucp_am_send* from host memory    |
[1781189946.904855] [server:2978 :0]   +--------------------------------+-------------------------------------------------+--+
[1781189946.904857] [server:2978 :0]   +--------------------------------+-------------------------------------------------+--+
[1781189946.904868] [server:2978 :0]   +--------------------------------+--------------------------------------------------------------------+
[1781189946.904887] [server:2978 :0]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1781189946.904889] [server:2978 :0]   +--------------------------------+-----------------------------------------------------------------+--+
[1781189946.904891] [server:2978 :0]   +--------------------------------+-----------------------------------------------------------------+--+
[1781189946.909643] [server:2978 :0]   +--------------------------------+---------------------------------------------------------------+
[1781189946.909649] [server:2978 :0]   | client_server intra-node cfg#2 | active message by ucp_am_send* from host memory               |
[1781189946.909651] [server:2978 :0]   +--------------------------------+--------------------------------------------+------------------+
[1781189946.909654] [server:2978 :0]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781189946.909656] [server:2978 :0]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781189946.909658] [server:2978 :0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781189946.909660] [server:2978 :0]   +--------------------------------+--------------------------------------------+------------------+
[1781189946.911323] [server:2978 :0]   +--------------------------------+--------------------------------------------------------------------+
[1781189946.911329] [server:2978 :0]   | client_server intra-node cfg#2 | active message by ucp_am_send* with reply flag from host memory    |
[1781189946.911331] [server:2978 :0]   +--------------------------------+-------------------------------------------------+------------------+
[1781189946.911333] [server:2978 :0]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781189946.911336] [server:2978 :0]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781189946.911340] [server:2978 :0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781189946.911343] [server:2978 :0]   +--------------------------------+-------------------------------------------------+------------------+
[1781189946.923936] [server:2978 :0]   +--------------------------------+---------------------------------------------------------------+
[1781189946.923942] [server:2978 :0]   | client_server intra-node cfg#3 | active message by ucp_am_send* from host memory               |
[1781189946.923945] [server:2978 :0]   +--------------------------------+--------------------------------------------+------------------+
[1781189946.923948] [server:2978 :0]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781189946.923951] [server:2978 :0]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781189946.923954] [server:2978 :0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781189946.923956] [server:2978 :0]   +--------------------------------+--------------------------------------------+------------------+
[1781189946.925636] [server:2978 :0]   +--------------------------------+--------------------------------------------------------------------+
[1781189946.925641] [server:2978 :0]   | client_server intra-node cfg#3 | active message by ucp_am_send* with reply flag from host memory    |
[1781189946.925643] [server:2978 :0]   +--------------------------------+-------------------------------------------------+------------------+
[1781189946.925646] [server:2978 :0]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781189946.925649] [server:2978 :0]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781189946.925651] [server:2978 :0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781189946.925656] [server:2978 :0]   +--------------------------------+-------------------------------------------------+------------------+
[info] [ucx_transmitter.cpp:172] Connection established
[info] [event_based_scheduler.cpp:221] Adding thread to default thread pool without pin cores
[info] [ucx_context.cpp:67] Ucx Context [cid: 16]: GPUDevice value found and cached. dev_id: 0.
[info] [ucx_transmitter.cpp:103] Ucx Transmitter [cid: 9]: GPUDevice value found and cached. dev_id: 0
[info] [gxf_executor.cpp:2921] [fragment1] Running Graph...
[info] [gxf_executor.cpp:2923] [fragment1] Waiting for completion...
[info] [event_based_scheduler.cpp:771] Event Based scheduler started worker thread [pool name: default_pool, thread uid: 0]
[TensorGenOp] Emitted GPU Tensor.
[1781189947.264255] [server:2978 :1]   +--------------------------------+---------------------------------------------------------------+
[1781189947.264263] [server:2978 :1]   | client_server intra-node cfg#3 | active message by ucp_am_send*(rndv) from cuda/GPU0           |
[1781189947.264265] [server:2978 :1]   +--------------------------------+--------------------------------------------+------------------+
[1781189947.264268] [server:2978 :1]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781189947.264270] [server:2978 :1]   |                         1..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781189947.264272] [server:2978 :1]   +--------------------------------+--------------------------------------------+------------------+
[1781189947.269954] [server:2978 :1]   +--------------------------------+------------------------------------------------+
[1781189947.269962] [server:2978 :1]   | client_server intra-node cfg#3 | rendezvous data send from cuda/GPU0 to cuda    |
[1781189947.269964] [server:2978 :1]   +--------------------------------+-----------------------------+------------------+
[1781189947.269966] [server:2978 :1]   |                         0..inf | fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781189947.269968] [server:2978 :1]   +--------------------------------+-----------------------------+------------------+
[TensorGenOp] Emitted GPU Tensor.
[TensorGenOp] Emitted GPU Tensor.
[TensorGenOp] Emitted GPU Tensor.
[TensorGenOp] Emitted GPU Tensor.
[info] [event_based_scheduler.cpp:1575] Stopping all async jobs
[info] [event_based_scheduler.cpp:753] Dispatcher thread has stopped checking jobs
[info] [event_based_scheduler.cpp:1554] Waiting to join all async threads
[info] [event_based_scheduler.cpp:1116] Async event handler thread exiting.
[info] [event_based_scheduler.cpp:1561] Waiting to join max duration thread
[info] [event_based_scheduler.cpp:1569] All async worker threads joined, deactivating all entities
[info] [event_based_scheduler.cpp:1534] Event Based scheduler finished.
[info] [gxf_executor.cpp:2930] [fragment1] Deactivating Graph...
[info] [ucx_context.cpp:113] UcxContext: Initiating graceful shutdown
[info] [event_based_scheduler.cpp:1278] Total execution time of Event Based scheduler : 1491.661634 ms

[info] [gxf_executor.cpp:2939] [fragment1] Graph execution finished.
[info] [app_worker.cpp:308] Worker execution finished
[info] [client.cpp:122] WorkerExecutionFinished response (10.129.2.221:5000): Worker execution succeeded (worker id: 10.129.2.221:10001)

ENV:



UCX_RNDV_THRESH=0
OPENUCX_VERSION=1.20.0
UCX_PROTO_INFO=y
UCX_TLS=cuda_ipc,cuda_copy,tcp
HOLOSCAN_UCX_DEVICE_ID=0

utkarsh02t · June 13, 2026, 2:56pm

I have an update I was able to make the single tensor work!

This is the flag I needed :export UCX_CM_USE_ALL_DEVICES=y

BUT, with my luck I am still catching hailstones:

When I try to send two tensors! with dict[str, Tensor]

from holoscan.core import Application, Fragment, Operator, OperatorSpec
from holoscan.conditions import CountCondition 
import cupy as cp


class TensorGenOp(Operator):
    def __init__(self, fragment, *args, **kwargs):
        super().__init__(fragment, *args, **kwargs)

    def setup(self, spec: OperatorSpec):
        spec.output("out")

    def compute(self, op_input, op_output, *args):
        gpu_array = cp.ones((5120, 5120), dtype=cp.float32) * 42.0
        gpu_array_2 = cp.ones((5120, 5120), dtype=cp.float32) * 23.0
        gpu_out = dict(gpu_one=gpu_array, gpu_two=gpu_array_2)
        # gpu_out = {"gpu_one" : gpu_array}
        op_output.emit(gpu_out, "out")
        print("[TensorGenOp] Emitted GPU Tensor.")


class TensorProcessOp(Operator):
    def __init__(self, fragment, *args, **kwargs):
        super().__init__(fragment, *args, **kwargs)

    def setup(self, spec: OperatorSpec):
        spec.input("in")

    def compute(self, op_input, op_output, *args):
        gpu_dict = op_input.receive("in")
        cupy_array = cp.asarray(gpu_dict["gpu_one"])
        cupy_array_two = cp.asarray(gpu_dict["gpu_one"])
        result = cupy_array * 2.0
        result2 = cupy_array_two * 3.0
        print(f"[TensorProcessOp] Processed mean value: {result.mean().item()} {result2.mean().item()}")
        # print(f"[TensorProcessOp] Processed mean value: {result.mean().item()} ")


class Fragment1(Fragment):
    def compose(self):
        gen_op = TensorGenOp(
            self, 
            CountCondition(self, count=5, name="count_cond"), 
            name="gen_op"
        )
        
        self.add_operator(gen_op)

class Fragment2(Fragment):
    def compose(self):
        process_op = TensorProcessOp(self, name="process_op")
        self.add_operator(process_op)

class MultiFragmentGPUApp(Application):
    def compose(self):
        fragment1 = Fragment1(self, name="fragment1")
        fragment2 = Fragment2(self, name="fragment2")
        
        self.add_fragment(fragment1)
        self.add_fragment(fragment2)
        
        self.add_flow(fragment1, fragment2, {("gen_op.out", "process_op.in")})

if __name__ == "__main__":
    app = MultiFragmentGPUApp()
    app.run()

Now this doesn’t work! This send it over TCP again :(

As from here @grelee Failing to send numpy array over fragment boundary - #3 by grelee

I know I can do dict[str, Tensor] and as I am using cupy they are on the GPU.

logs:

root@server:/workspace/experimental_dags/pure_holo# python3 multi_frag.py --worker --fragments fragment2 --address 10.129.2.221:5001
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [gxf_executor.cpp:566] [fragment1] Creating context
[info] [gxf_executor.cpp:566] [fragment2] Creating context
[info] [app_driver.cpp:1394] Launching the worker service
[info] [server.cpp:195] AppWorkerServer listening on 0.0.0.0:10001
[info] [server.cpp:196] AppWorkerServer/Health checking server listening on 0.0.0.0:10001
[info] [client.cpp:81] FragmentAllocation response (10.129.2.221:5001): Fragment allocation request is accepted.
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [gxf_executor.cpp:566] Creating context
[info] [service_impl.cpp:41] Number of ports requested: 1
[info] [gxf_executor.cpp:566] Creating context
[info] [gxf_executor.cpp:601] Destroying context
[info] [app_worker.cpp:228] Launching fragment: fragment2
[info] [gxf_executor.cpp:2780] Activating Graph...
[info] [entity_pool.cpp:134] EntityPool::GetShared: created new shared pool for context 0x19544e10 with capacity 256
[1781362359.311204] [server:95882:0]   +--------------------------+----------------------------------------------------+
[1781362359.311216] [server:95882:0]   | client_server self cfg#0 | active message by ucp_am_send* from host memory    |
[1781362359.311218] [server:95882:0]   +--------------------------+-------------------------------------------------+--+
[1781362359.311220] [server:95882:0]   +--------------------------+-------------------------------------------------+--+
[1781362359.311248] [server:95882:0]   +--------------------------+--------------------------------------------------------------------+
[1781362359.311251] [server:95882:0]   | client_server self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1781362359.311253] [server:95882:0]   +--------------------------+-----------------------------------------------------------------+--+
[1781362359.311255] [server:95882:0]   +--------------------------+-----------------------------------------------------------------+--+
[1781362359.318479] [server:95882:0]   +--------------------------+----------------------------------------------------+
[1781362359.318485] [server:95882:0]   | client_server self cfg#0 | active message by ucp_am_send* from host memory    |
[1781362359.318488] [server:95882:0]   +--------------------------+-------------------------------------------------+--+
[1781362359.318490] [server:95882:0]   +--------------------------+-------------------------------------------------+--+
[1781362359.318517] [server:95882:0]   +--------------------------+--------------------------------------------------------------------+
[1781362359.318520] [server:95882:0]   | client_server self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1781362359.318522] [server:95882:0]   +--------------------------+-----------------------------------------------------------------+--+
[1781362359.318524] [server:95882:0]   +--------------------------+-----------------------------------------------------------------+--+
[info] [ucx_context.cpp:700] UcxContext::register_am_recv_legacy_callback
[info] [ucx_context.cpp:619] Server is listening on IP 0.0.0.0 port 10002. Waiting for connection...
[info] [event_based_scheduler.cpp:221] Adding thread to default thread pool without pin cores
[info] [ucx_context.cpp:982] UcxContext::start_server
[info] [ucx_context.cpp:67] Ucx Context [cid: 25]: GPUDevice value found and cached. dev_id: 0.
[info] [ucx_receiver.cpp:174] Ucx Receiver [cid: 9]: GPUDevice value found and cached. dev_id: 0
[info] [gxf_executor.cpp:2921] [fragment2] Running Graph...
[info] [gxf_executor.cpp:2923] [fragment2] Waiting for completion...
[info] [event_based_scheduler.cpp:771] Event Based scheduler started worker thread [pool name: default_pool, thread uid: 0]
[info] [ucx_context.cpp:520] Connection request received to 0.0.0.0:10002 from client at address 10.129.2.221:55147
[1781362360.316563] [server:95882:1]   +--------------------------------+---------------------------------------------------------------+
[1781362360.316570] [server:95882:1]   | client_server intra-node cfg#1 | active message by ucp_am_send* from host memory               |
[1781362360.316573] [server:95882:1]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.316575] [server:95882:1]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781362360.316577] [server:95882:1]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781362360.316579] [server:95882:1]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.316582] [server:95882:1]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.318572] [server:95882:1]   +--------------------------------+--------------------------------------------------------------------+
[1781362360.318578] [server:95882:1]   | client_server intra-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1781362360.318580] [server:95882:1]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.318582] [server:95882:1]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781362360.318584] [server:95882:1]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781362360.318586] [server:95882:1]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781362360.318588] [server:95882:1]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.336445] [server:95882:1]   +--------------------------------+---------------------------------------------------------------+
[1781362360.336453] [server:95882:1]   | client_server intra-node cfg#2 | active message by ucp_am_send* from host memory               |
[1781362360.336456] [server:95882:1]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.336458] [server:95882:1]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781362360.336460] [server:95882:1]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781362360.336462] [server:95882:1]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.336464] [server:95882:1]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.338427] [server:95882:1]   +--------------------------------+--------------------------------------------------------------------+
[1781362360.338432] [server:95882:1]   | client_server intra-node cfg#2 | active message by ucp_am_send* with reply flag from host memory    |
[1781362360.338434] [server:95882:1]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.338436] [server:95882:1]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781362360.338438] [server:95882:1]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781362360.338440] [server:95882:1]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781362360.338442] [server:95882:1]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.684583] [server:95882:2]   +--------------------------------+----------------------------------------------------+
[1781362360.684605] [server:95882:2]   | client_server intra-node cfg#2 | rendezvous data fetch into iov[2] cuda/GPU0        |
[1781362360.684614] [server:95882:2]   +--------------------------------+---------------------------------+------------------+
[1781362360.684622] [server:95882:2]   |                              0 | no data fetch                   |                  |
[1781362360.684630] [server:95882:2]   |                         1..inf | (?) fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.684637] [server:95882:2]   +--------------------------------+---------------------------------+------------------+
[TensorProcessOp] Processed mean value: 84.0 126.0
[TensorProcessOp] Processed mean value: 84.0 126.0
[TensorProcessOp] Processed mean value: 84.0 126.0
[TensorProcessOp] Processed mean value: 84.0 126.0
[TensorProcessOp] Processed mean value: 84.0 126.0
[info] [event_based_scheduler.cpp:1575] Stopping all async jobs
[info] [event_based_scheduler.cpp:753] Dispatcher thread has stopped checking jobs
[info] [event_based_scheduler.cpp:1554] Waiting to join all async threads
[info] [event_based_scheduler.cpp:1116] Async event handler thread exiting.
[info] [event_based_scheduler.cpp:1561] Waiting to join max duration thread
[info] [event_based_scheduler.cpp:1569] All async worker threads joined, deactivating all entities
[info] [event_based_scheduler.cpp:1534] Event Based scheduler finished.
[info] [gxf_executor.cpp:2930] [fragment2] Deactivating Graph...
[info] [ucx_context.cpp:113] UcxContext: Initiating graceful shutdown
[info] [event_based_scheduler.cpp:1278] Total execution time of Event Based scheduler : 9549.578975 ms

[info] [gxf_executor.cpp:2939] [fragment2] Graph execution finished.
[info] [app_worker.cpp:308] Worker execution finished
[info] [client.cpp:122] WorkerExecutionFinished response (10.129.2.221:5001): Worker execution succeeded (worker id: 10.129.2.221:10001)

root@server:/workspace/experimental_dags/pure_holo# python3 multi_frag.py --driver --worker --fragments fragment1 --address 10.129.2.221:5001
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [gxf_executor.cpp:566] [fragment1] Creating context
[info] [gxf_executor.cpp:566] [fragment2] Creating context
[info] [app_driver.cpp:1375] Launching the driver/health checking service
[info] [app_driver.cpp:1394] Launching the worker service
[info] [server.cpp:88] AppDriverServer listening on 0.0.0.0:5001
[info] [gxf_executor.cpp:566] Creating context
[info] [server.cpp:195] AppWorkerServer listening on 0.0.0.0:10000
[info] [server.cpp:196] AppWorkerServer/Health checking server listening on 0.0.0.0:10000
[info] [service_impl.cpp:87] Connected from the client address: ipv4:10.129.2.221:41658 (worker address: 10.129.2.221:10000)
[info] [client.cpp:81] FragmentAllocation response (10.129.2.221:5001): Fragment allocation request is accepted.
[info] [app_driver.cpp:1232] 1/2 fragments scheduled. Awaiting remaining worker connections.

fragment_name: fragment2, cpu: -1, cpu_limit: -1, gpu: -1, gpu_limit: -1, memory: 0, memory_limit: 0, shared_memory: 0, shared_memory_limit: 0, gpu_memory: 0, gpu_memory_limit: 0

[info] [service_impl.cpp:87] Connected from the client address: ipv4:10.129.2.221:41150 (worker address: 10.129.2.221:10001)
[info] [app_driver.cpp:1019] Fragment schedule is available:
[info] [app_driver.cpp:1021]   Fragment 'fragment2' => Worker '10.129.2.221:10001'
[info] [app_driver.cpp:1021]   Fragment 'fragment1' => Worker '10.129.2.221:10000'
[info] [fragment.cpp:1142] Loading extensions from configs...
[info] [client.cpp:103] GetFragmentInfo response (10.129.2.221:10000) received
[info] [client.cpp:103] GetFragmentInfo response (10.129.2.221:10001) received
[info] [service_impl.cpp:41] Number of ports requested: 0
[info] [client.cpp:248] FragmentExecution response (10.129.2.221:10000): Fragment Execution request processed successfully
[info] [client.cpp:248] FragmentExecution response (10.129.2.221:10001): Fragment Execution request processed successfully
[info] [app_worker.cpp:228] Launching fragment: fragment1
[info] [gxf_executor.cpp:2780] Activating Graph...
[info] [entity_pool.cpp:134] EntityPool::GetShared: created new shared pool for context 0x1575ead0 with capacity 256
[1781362359.303177] [server:95774:0]   +--------------------------+----------------------------------------------------+
[1781362359.303189] [server:95774:0]   | client_server self cfg#0 | active message by ucp_am_send* from host memory    |
[1781362359.303191] [server:95774:0]   +--------------------------+-------------------------------------------------+--+
[1781362359.303194] [server:95774:0]   +--------------------------+-------------------------------------------------+--+
[1781362359.303206] [server:95774:0]   +--------------------------+--------------------------------------------------------------------+
[1781362359.303209] [server:95774:0]   | client_server self cfg#0 | active message by ucp_am_send* with reply flag from host memory    |
[1781362359.303211] [server:95774:0]   +--------------------------+-----------------------------------------------------------------+--+
[1781362359.303214] [server:95774:0]   +--------------------------+-----------------------------------------------------------------+--+
[1781362360.303549] [server:95774:0]   +--------------------------------+----------------------------------------------------+
[1781362360.303557] [server:95774:0]   | client_server inter-node cfg#1 | active message by ucp_am_send* from host memory    |
[1781362360.303559] [server:95774:0]   +--------------------------------+-------------------------------------------------+--+
[1781362360.303562] [server:95774:0]   +--------------------------------+-------------------------------------------------+--+
[1781362360.303589] [server:95774:0]   +--------------------------------+--------------------------------------------------------------------+
[1781362360.303591] [server:95774:0]   | client_server inter-node cfg#1 | active message by ucp_am_send* with reply flag from host memory    |
[1781362360.303593] [server:95774:0]   +--------------------------------+-----------------------------------------------------------------+--+
[1781362360.303596] [server:95774:0]   +--------------------------------+-----------------------------------------------------------------+--+
[1781362360.308441] [server:95774:0]   +--------------------------------+---------------------------------------------------------------+
[1781362360.308448] [server:95774:0]   | client_server intra-node cfg#2 | active message by ucp_am_send* from host memory               |
[1781362360.308450] [server:95774:0]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.308452] [server:95774:0]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781362360.308455] [server:95774:0]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781362360.308457] [server:95774:0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.308459] [server:95774:0]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.310122] [server:95774:0]   +--------------------------------+--------------------------------------------------------------------+
[1781362360.310127] [server:95774:0]   | client_server intra-node cfg#2 | active message by ucp_am_send* with reply flag from host memory    |
[1781362360.310130] [server:95774:0]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.310132] [server:95774:0]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781362360.310135] [server:95774:0]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781362360.310140] [server:95774:0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781362360.310142] [server:95774:0]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.322515] [server:95774:0]   +--------------------------------+---------------------------------------------------------------+
[1781362360.322521] [server:95774:0]   | client_server intra-node cfg#3 | active message by ucp_am_send* from host memory               |
[1781362360.322523] [server:95774:0]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.322526] [server:95774:0]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781362360.322528] [server:95774:0]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781362360.322530] [server:95774:0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.322532] [server:95774:0]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.324238] [server:95774:0]   +--------------------------------+--------------------------------------------------------------------+
[1781362360.324243] [server:95774:0]   | client_server intra-node cfg#3 | active message by ucp_am_send* with reply flag from host memory    |
[1781362360.324245] [server:95774:0]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.324248] [server:95774:0]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781362360.324251] [server:95774:0]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781362360.324253] [server:95774:0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781362360.324256] [server:95774:0]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.328024] [server:95774:0]   +--------------------------------+---------------------------------------------------------------+
[1781362360.328030] [server:95774:0]   | client_server intra-node cfg#4 | active message by ucp_am_send* from host memory               |
[1781362360.328033] [server:95774:0]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.328035] [server:95774:0]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781362360.328037] [server:95774:0]   |                       1..16352 | (?) rendezvous zero-copy                   | tcp/enp28s0f0np0 |
[1781362360.328039] [server:95774:0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.328041] [server:95774:0]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.329727] [server:95774:0]   +--------------------------------+--------------------------------------------------------------------+
[1781362360.329732] [server:95774:0]   | client_server intra-node cfg#4 | active message by ucp_am_send* with reply flag from host memory    |
[1781362360.329734] [server:95774:0]   +--------------------------------+-------------------------------------------------+------------------+
[1781362360.329737] [server:95774:0]   |                              0 | (?) rendezvous no data fetch                    |                  |
[1781362360.329740] [server:95774:0]   |                       1..16352 | (?) rendezvous zero-copy                        | tcp/enp28s0f0np0 |
[1781362360.329742] [server:95774:0]   |                     16353..inf | (?) rendezvous fragmented copy-in copy-out      | tcp/enp28s0f0np0 |
[1781362360.329744] [server:95774:0]   +--------------------------------+-------------------------------------------------+------------------+
[info] [ucx_transmitter.cpp:172] Connection established
[info] [event_based_scheduler.cpp:221] Adding thread to default thread pool without pin cores
[info] [ucx_context.cpp:67] Ucx Context [cid: 16]: GPUDevice value found and cached. dev_id: 0.
[info] [ucx_transmitter.cpp:103] Ucx Transmitter [cid: 9]: GPUDevice value found and cached. dev_id: 0
[info] [gxf_executor.cpp:2921] [fragment1] Running Graph...
[info] [gxf_executor.cpp:2923] [fragment1] Waiting for completion...
[info] [event_based_scheduler.cpp:771] Event Based scheduler started worker thread [pool name: default_pool, thread uid: 0]
[TensorGenOp] Emitted GPU Tensor.
[1781362360.681193] [server:95774:1]   +--------------------------------+---------------------------------------------------------------+
[1781362360.681201] [server:95774:1]   | client_server intra-node cfg#4 | active message by ucp_am_send*(rndv) from iov[2] cuda/GPU0    |
[1781362360.681203] [server:95774:1]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.681206] [server:95774:1]   |                              0 | (?) rendezvous no data fetch               |                  |
[1781362360.681208] [server:95774:1]   |                         1..inf | (?) rendezvous fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.681210] [server:95774:1]   +--------------------------------+--------------------------------------------+------------------+
[1781362360.685021] [server:95774:1]   +--------------------------------+------------------------------------------------+
[1781362360.685026] [server:95774:1]   | client_server intra-node cfg#4 | rendezvous data send from iov[2] cuda/GPU0     |
[1781362360.685029] [server:95774:1]   +--------------------------------+-----------------------------+------------------+
[1781362360.685031] [server:95774:1]   |                         0..inf | fragmented copy-in copy-out | tcp/enp28s0f0np0 |
[1781362360.685034] [server:95774:1]   +--------------------------------+-----------------------------+------------------+
[TensorGenOp] Emitted GPU Tensor.
[TensorGenOp] Emitted GPU Tensor.
[TensorGenOp] Emitted GPU Tensor.
[TensorGenOp] Emitted GPU Tensor.
[info] [event_based_scheduler.cpp:1575] Stopping all async jobs
[info] [event_based_scheduler.cpp:753] Dispatcher thread has stopped checking jobs
[info] [event_based_scheduler.cpp:1554] Waiting to join all async threads
[info] [event_based_scheduler.cpp:1116] Async event handler thread exiting.
[info] [event_based_scheduler.cpp:1561] Waiting to join max duration thread
[info] [event_based_scheduler.cpp:1569] All async worker threads joined, deactivating all entities
[info] [event_based_scheduler.cpp:1534] Event Based scheduler finished.
[info] [gxf_executor.cpp:2930] [fragment1] Deactivating Graph...
[info] [ucx_context.cpp:113] UcxContext: Initiating graceful shutdown
[info] [event_based_scheduler.cpp:1278] Total execution time of Event Based scheduler : 4536.289796 ms

[info] [gxf_executor.cpp:2939] [fragment1] Graph execution finished.
[info] [app_worker.cpp:308] Worker execution finished
[info] [service_impl.cpp:283] Worker execution succeeded (worker id: 10.129.2.221:10000)
[info] [client.cpp:122] WorkerExecutionFinished response (10.129.2.221:5001): Worker execution succeeded (worker id: 10.129.2.221:10000)
[info] [server.cpp:150] Closed connection to worker 10.129.2.221:10000
[info] [app_driver.cpp:1261] Worker 10.129.2.221:10000 has finished execution. Remaining: 1
[info] [service_impl.cpp:283] Worker execution succeeded (worker id: 10.129.2.221:10001)
[info] [server.cpp:150] Closed connection to worker 10.129.2.221:10001
[info] [app_driver.cpp:1261] Worker 10.129.2.221:10001 has finished execution. Remaining: 0
[info] [app_driver.cpp:1264] All workers have finished execution

I think I am on my last steps, any push from here would help!!

A bit more trace that it is being sent as TensorMap:


[2026-06-13 15:13:59.175] [96740] [holoscan] [debug] [entity_executor.cpp:921] [C00008] tick codelet gen_op in entity fragment1__gen_op
[2026-06-13 15:13:59.175] [96740] [holoscan] [trace] [gxf_wrapper.cpp:79] GXFWrapper::tick()
[2026-06-13 15:13:59.175] [96740] [holoscan] [trace] [gxf_wrapper.cpp:133] Calling operator: gen_op
[2026-06-13 15:13:59.175] [96738] [holoscan] [debug] [event_based_scheduler.cpp:741] Dispatcher waiting indefinitely for event (has_notifications=0)
[2026-06-13 15:13:59.176] [96740] [holoscan] [trace] [io_context.cpp:204] OutputContext::emit (TensorMap) for op: gen_op, name: out
[2026-06-13 15:13:59.176] [96740] [holoscan] [trace] [runtime.cpp:1094] [E00277] COMPONENT CREATE: C00290 (type=nvidia::gxf::Tensor) name: gpu_two
[2026-06-13 15:13:59.176] [96740] [holoscan] [trace] [runtime.cpp:1094] [E00277] COMPONENT CREATE: C00291 (type=nvidia::gxf::Tensor) name: gpu_one
[2026-06-13 15:13:59.176] [96740] [holoscan] [trace] [gxf_io_context.cpp:816] GXFOutputContext::emit_impl for op: gen_op, output_name: out, out_type: kGXFEntity
[2026-06-13 15:13:59.176] [96740] [holoscan] [trace] [gxf_io_context.cpp:907] in GXFOutputContext::emit_impl: out_type: 0
[TensorGenOp] Emitted GPU Tensor.
[2026-06-13 15:14:00.065] [96740] [holoscan] [trace] [entity_pool.cpp:327] EntityPool release: eid=277 returned to pool, available=252

Topic		Replies	Views
Failing to send numpy array over fragment boundary General	4	91	May 27, 2026
UCX ERROR failed to unpack address, invalid bandwidth 0.00 General	0	68	March 6, 2026
NVHPC Code with Multiple GPUs inside Singularity Container gives UCX Error GPU-Accelerated Libraries	0	273	May 6, 2024
Problem with IPC CUDA Programming and Performance	10	3715	May 27, 2020
Test the sample about "Connect Three DGX Spark in a Ring Topology" DGX Spark / GB10 cuda	15	1033	April 13, 2026
NCCL all-reduce deadlock on dual DGX Spark after successful channel establishment — affects both vLLM and TRT-LLM DGX Spark / GB10 nemotron	21	912	April 17, 2026
MultiGPU P2P Access Weird result. CUDA Programming and Performance	10	1402	June 13, 2016
NCCL 2.0 support inter-node communication using Sockets? GPU-Accelerated Libraries	3	4403	December 21, 2018
Inconsistent concurrent transfer speed CUDA Programming and Performance	20	1579	April 17, 2023
Jetson AGX Orin CUDA IPC Support Jetson AGX Orin cuda , docker , pytorch	6	2589	July 29, 2022

UCX for GPU tensors chosing TCP over CUDA for transporting GPU tensors

Related topics