GlusterFS: host, is not befriended at the moment
When attempting to expand the replication distrubted file system of our cluster nodes I struck the error message:
[root@index03 ~]# gluster volume replace-brick drvol01 index04:/srv/gluster/drbrk02 index05:/srv/gluster/drbrk01 start index05, is not befriended at the moment
Which was strange considering the previous ‘peer probe’ from another host to join the new server (index05) was successfully:
[root@index04 ~]# gluster peer probe index05 Probe successful
I think the only clue was (on the last line):
[root@index03 drbrk02]# gluster peer status Number of Peers: 4 Hostname: index02 Uuid: 31c0f246-0a8b-4c68-8ffa-42b2e6bb42ce State: Peer in Cluster (Connected) Hostname: index04 Uuid: ca61b983-b8fd-4fcc-848a-6c33f102cd5c State: Peer in Cluster (Connected) Hostname: index01 Uuid: b9bc474c-0017-4191-82b9-5760ba0d00fc State: Peer in Cluster (Connected) Hostname: index05 Uuid: c6a7cb83-4c1f-42ff-8421-bea9cc911d0f State: Accepted peer request (Connected)
Anyway I confirmed that there were no firewall rules blocking the requests Doco: Gluster Firewall Ports and restarted the glusterd service on index05, which seems to have resolved the problem.
[root@index03 ~]# gluster peer status Number of Peers: 4 Hostname: index02 Uuid: 31c0f246-0a8b-4c68-8ffa-42b2e6bb42ce State: Peer in Cluster (Connected) Hostname: index04 Uuid: ca61b983-b8fd-4fcc-848a-6c33f102cd5c State: Peer in Cluster (Connected) Hostname: index01 Uuid: b9bc474c-0017-4191-82b9-5760ba0d00fc State: Peer in Cluster (Connected) Hostname: index05 Uuid: c6a7cb83-4c1f-42ff-8421-bea9cc911d0f State: Peer in Cluster (Connected)
In summary I believe there were firewall rules in place, when the ‘gluster peer probe index05′ was run. The rules were removed, but the gluster daemon on index05 then required a restart to become active in the cluster.