Node provisioning failing with BCM10 on Rocky9u4

Infrastructure & Networking Nvidia Bright Cluster Manager
1

Hi,

First of all, this is a test cluster (Easy8) so I’m posting here looking for direction.
The problem I’m seeing is that provisioning is broken with the default Rocky9U4 image.
Whenever I boot a node, I get the following:

[hn0->device]%
Mon Jan  6 09:54:20 2025 [notice] hn0: n001 [     INSTALLING      ] (node installer started)
[hn0->device]%
Mon Jan  6 09:55:09 2025 [notice] hn0: n001 [     INSTALLING      ] (provisioning error, will retry)
[hn0->device]%
Mon Jan  6 09:55:38 2025 [notice] hn0: n001 [  INSTALLER_FAILED   ] (An error occurred while provisioning. Rsync failed (Exit code: 23 Signal: 0).)

The /var/log/node-installer log just repeats the “Exit code: 23” error 29 times before finally giving up.
I’ve uploaded the relevant logs in case anyone wants to look.
logs.tar.gz (2.1 MB)

2

Hi Chad,

I would suggest that you take a look at the /var/spool/cmd/n001-.rsync on the headnode.

This file will have more details on why the rsync failed.

Kind regards,
adel

3

I looked in the log and the error(s) show that the xattr option in the rsync command is causing the provisioning to fail. A coworker opened a case last week and was giving the following work-around:

cmsh

% fspart use /cm/images/…

% set rsyncxattr no

% commit

4

Hi Chad,

Thank you for the heads up. Looks like the extended attributes on some files are causing issues with rsync?

I may suggest excluding the paths that cause rsync to fail instead of disabling the extended attributes all together.

In most of the cases, the files that cause issues with rsync are cache files grabbed from a live node into an image and they are not crucial for running the image.

The relevant exclude lists are excludelistsyncinstall and excludelistupdate and they can be set on the category level:

cmsh
category use mycategory
set excludelistsyncinstall # add the paths you’d like to exclude; save and exit
commit
set excludelistupdate # add the paths you’d like to exclude; save and exit
commit

Kind regards,
adel