BCM 9.2 node provision problem INSTALLER_UNREACHABLE

Infrastructure & Networking Nvidia Bright Cluster Manager
1

Recently my nodes booted up and are saying:

[root@bright92 ~]# cmsh -c 'device; foreach -n node001..node002 (status)'
node001 .................. [ INSTALLER_UNREACHABLE ] (switching to local root)
node002 .................. [ INSTALLER_UNREACHABLE ] (switching to local root)

I can SSH to them no problem and shares from the head node are mounted:

$ df -h
Filesystem         Size  Used Avail Use% Mounted on
devtmpfs            16G     0   16G   0% /dev
tmpfs               16G  632K   16G   1% /run
tmpfs               29G  8.3G   20G  30% /
tmpfs               16G     0   16G   0% /dev/shm
tmpfs               16G     0   16G   0% /sys/fs/cgroup
master:/home       222G   42G  180G  19% /home
master:/cm/shared  222G   42G  180G  19% /cm/shared

From the booted regular node I can SSH back to the head node fine. How do I troubleshoot? Thanks

My regular node:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 5c:ba:2c:20:38:3a brd ff:ff:ff:ff:ff:ff
    altname enp2s0f0
    inet 10.141.0.1/16 brd 10.141.255.255 scope global dynamic eno1
       valid_lft 84347sec preferred_lft 84347sec
    inet6 fe80::5eba:2cff:fe20:383a/64 scope link
       valid_lft forever preferred_lft forever
2

Hi,

Assuming that the node has booted up then the “INSTALLER_UNREACHABLE” in this case means that the cmd.service on the compute node can’t talk to the cmd.service on the headnode.

Perhaps you can check the following:

  1. if there is a firewall filtering the traffic on the internal management network, then make sure that port 2 is open:
    telnet 2 # here has to be replaced with the real hostname of the compute node

  2. check the /var/log/cmdaemon on the compute node to see if cmd.service is failing to start for any reason.

Kind regards,
adel

3

Thanks it turns out my default-image was messed up and was missing cmdaemon.