Replication Feed not working through SCS

If your replication feed is not working through your SCS, here are some troubleshooting tips. Another symptom, if no replication feeds are present, is that health reports would not be sent correctly.

Ensure NAT is enabled on the source side SCS

• There was a bug in documentation for deploying an offline SCS where /etc/hosts has datacore.com configured to use the SCS's private IP address for resolution. This causes the SCS not to apply NAT on the proper interface (we presume the interface resolving datacore.com is the external interface), preventing the source side Swarm nodes from being able to reach past their local subnet.

• Run this command: firewall-cmd --zone=public --list-all on the source side SCS and ensure that masquerade: yes is shown. If that shows no, then the following remediation steps are necessary.

  • Edit /etc/hosts and change all of the manual DNS entries to point to the IP of the EXTERNAL SCS's interface. Typically that interface is ens192 if you are importing the SCS VM from one of our bundles. Please also run the following commands:

  • scsctl init config_update --external-interface <name of external interface>

  • scsctl init wizard --build-platform-pod

  • scsctl init config_update --finalize

  • firewall-cmd --zone=public --add-masquerade --permanent

  • firewall-cmd --reload

  • podman network reload -a

  • systemctl restart swarm-platform

• Verify that you now see masquerade: yes when you run firewall-cmd --zone=public --list-all

• You should be able to use the terminal console on one of the storage nodes to ping the destination now (presuming ICMP is allowed). The replication feed should start working within a few minutes on its own.

Ensure connectivity from the source side SCS to replication feed destination

• The first thing to check with your replication feed is that the replication feed destination is available from your source side SCS. To find which IP addresses are being used in your replication feed, you can look at the replication feed definition in either the Swarm UI or the Legacy console. Alternatively, you can use swarmctl from the support tools bundle to get a list of replication feeds like this: swarmctl -d <swarm node ip> -p <admin>:<password> --feeds

• Once you have the IP or FQDN and potentially port, you can either ping or use netcat or similar tool to verify connectivity. You can install netcat with yum -y install netcat and do something like this: nc -zv <IP from replication feed> <listening port> You should see something like this: Connection to 10.1.1.220 8090 port [tcp/*] succeeded!

• If that connection doesn't succeed, the source side SCS does not have connectivity to the destination. The source side SCS uses NAT to route traffic from the Swarm nodes by default. If the replication feed destination is a Content Gateway (which is common), then check the gateway's open ports (ss -tnl) to verify the port you intend to be listening on is actually listening. Also, verify that you have the correct allowSwarmAdminIP configuration in your gateway.cfg file. See https://perifery.atlassian.net/wiki/spaces/public/pages/2443810201/Gateway+Configuration#%5Bscsp%5D allowSwarmAdminIP parameter.

• If the configuration is correct, but the replication feed is still not able to connect, you can check firewalls along the path (use traceroute on the source side SCS to see where the packets stop) and also check the firewall on the destination side Content Gateway if that's the replication feed destination. This way, you can see if the destination is even receiving the packets. Like this: tcpdump -i <public side interface of Content Gateway> -nne port <listening port> and host <source side SCS public interface address>