Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 39 Next »

CentOS 8 / RHEL 8 / RockyLinux 8 : yum install haproxy

CentOS 7/RHEL 7 follow instructions provided here https://www.linux.org/threads/centos-announce-announcing-release-of-haproxy-1-8-on-centos-7-x86_64.18168/

Note: On CentOS7/RHEL7 the active haproxy.cfg will be in '/etc/opt/rh/rh-haproxy18/haproxy'

Haproxy is already pre-installed on the Swarm Cluster Installer provisioned - and VMware bundle provided Gateway VM

The following configuration steps are needed to configure HAProxy as an SSL offloader for Content Gateway.

Step-by-step guide

  • Verify Content Gateway is listening on port 8080 for SCSP, 8090 for S3 and 8091 for Service Proxy:

The example below are not the default gateway ports, this is done on purpose to avoid port conflicts.

/etc/caringo/cloudgateway/gateway.cfg

[scsp]
enabled = true
bindAddress = 0.0.0.0
bindPort = 8080
externalHTTPPort = 80
externalHTTPSPort = 443

[s3]
enabled = true
bindAddress = 0.0.0.0
bindPort = 8090

[cluster_admin]
enabled = true
bindAddress = 0.0.0.0
bindPort = 8091
externalHTTPSPort = 91

Note: In this example HTTP access to those personalities is provided. The bind address needs to be modified to 127.0.0.1 for all 3 personalities if security hardening is desired and HTTPS needs to be forced.

  • Setup and install HAProxy. This package is part of the EPEL repository.

HAproxy is pre-installed on the SwarmContentGateway VM and SCI deployed gateway VM’s.

  • Use the following example configuration for /etc/haproxy/haproxy.cfg or on CentOS7 with haproxy1.8 /etc/opt/rh/rh-haproxy18-haproxy/haproxy.cfg

global
    log 127.0.0.1 local2
    chroot /var/lib/haproxy
    stats socket /var/lib/haproxy/stats mode 660 level admin
    user haproxy
    group haproxy
    daemon

    ca-base /etc/pki/tls/certs
    crt-base /etc/pki/tls/private

    ssl-default-bind-ciphers ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!MD5:!DSS
    ssl-default-bind-options no-sslv3
    maxconn 2048
    tune.ssl.default-dh-param 2048

defaults
    log     global
    mode    http
    option  forwardfor
    # Do not use "option  http-server-close", it causes S3 PUT incompatibility with some clients including FileFly!
    option  httplog
    option  dontlognull
    timeout connect 5000
    timeout client  50000
    # This timeout should always be larger than gateway.cfg's [storage_cluster] indexerSocketTimeout.
    timeout server  130000

frontend www-http
    bind 0.0.0.0:80
    http-request set-header X-Forwarded-Proto http
    http-request set-header X-Forwarded-Port 80
    default_backend www-backend-scsp
    acl iss3 hdr_sub(Authorization) AWS
    acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)=
    use_backend www-backend-s3 if iss3

frontend www-https
    bind 0.0.0.0:443 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 443
    default_backend www-backend-scsp
    acl iss3 hdr_sub(Authorization) AWS
    acl iss3 url_reg [?&](AWSAccessKeyId|X-Amz-Credential)=
    use_backend www-backend-s3 if iss3

frontend www-https-svc
    bind 0.0.0.0:91 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem
    http-request set-header X-Forwarded-Proto https
    http-request set-header X-Forwarded-Port 91
    default_backend www-backend-svc

backend www-backend-scsp
    balance leastconn
    option httpchk
    http-check connect
    http-check send meth HEAD uri /_admin/manage/version
    http-check expect status 200
  
    #redirect scheme https if !{ ssl_fc }   <--- Uncomment this line if you want to force HTTPS
    server gw1 YOUR_GATEWAY1_IP:8080 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8080 check inter 10s fall 3 rise 2

backend www-backend-s3
    balance leastconn
    option httpchk
    http-check connect
    http-check send meth HEAD uri /_admin/manage/version
    http-check expect status 200
    
    #redirect scheme https if !{ ssl_fc }    <--- Uncomment this line if you want to force HTTPS
    server gw1 YOUR_GATEWAY1_IP:8090 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8090 check inter 10s fall 3 rise 2

backend www-backend-svc
    balance leastconn
    option httpchk
    http-check connect
    http-check send meth HEAD uri /_admin/manage/version
    http-check expect status 200
    
    # This rule rewrites CORS header to add the port number used on frontend
    http-request replace-value Access-Control-Allow-Origin (.*) \1:91
    #redirect scheme https if !{ ssl_fc } <--- Uncomment this line if you want to force HTTPS
    server gw1 YOUR_GATEWAY1_IP:8091 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8091 check inter 10s fall 3 rise 2

  • Start HAProxy:
    systemctl restart haproxy

If when restarting HAProxy this error is thrown “Starting frontend www-https-svc: cannot bind socket [0.0.0.0:91]”, either disable SELinux or run this command:

setsebool -P haproxy_connect_any=1

Loadbalancing healtcheck

If you have 2 or more gateways, it is recommended to use the http-check directives to verify the service is live. 

    option httpchk
    http-check connect
    http-check send meth HEAD uri /_admin/manage/version
    http-check expect status 200
    server gw1 YOUR_GATEWAY1_IP:8090 check inter 10s fall 3 rise 2
    server gw2 YOUR_GATEWAY2_IP:8090 check inter 10s fall 3 rise 2

In the following example I ran a while loop, to read an object every second… and while doing it shutdown 1 of the 2 gateways waited a few seconds then started it back up.

Mar 15 08:01:54 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45870 [15/Mar/2024:08:01:54.044] www-http swarm-scsp/gw1 0/0/0/20/29 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:01:55 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45876 [15/Mar/2024:08:01:55.095] www-http swarm-scsp/gw2 0/0/0/20/28 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:01:56 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45880 [15/Mar/2024:08:01:56.144] www-http swarm-scsp/gw1 0/0/0/16/22 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:01:57 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45890 [15/Mar/2024:08:01:57.187] www-http swarm-scsp/gw2 0/0/0/20/29 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"

**** Here I stop gw1 service...  which produces 503's until haproxy detects it

Mar 15 08:02:01 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45894 [15/Mar/2024:08:01:58.236] www-http swarm-scsp/gw1 0/0/-1/-1/3003 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:02 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45908 [15/Mar/2024:08:02:02.260] www-http swarm-scsp/gw2 0/0/0/20/29 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:06 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45914 [15/Mar/2024:08:02:03.310] www-http swarm-scsp/gw1 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:07 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45928 [15/Mar/2024:08:02:07.338] www-http swarm-scsp/gw2 0/0/0/19/27 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:11 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45932 [15/Mar/2024:08:02:08.387] www-http swarm-scsp/gw1 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:12 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45946 [15/Mar/2024:08:02:12.415] www-http swarm-scsp/gw2 0/0/0/14/21 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45954 [15/Mar/2024:08:02:13.456] www-http swarm-scsp/gw1 0/0/-1/-1/3006 503 216 - - SC-- 1/1/0/0/3 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {|} "GET /test14/test2 HTTP/1.1"

**** HAproxy figures out gw1 is down

Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: [WARNING]  (1411077) : Server swarm-scsp/gw1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Mar 15 08:02:16 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.

**** At this point all calls go to gw2 only

Mar 15 08:02:17 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45968 [15/Mar/2024:08:02:17.483] www-http swarm-scsp/gw2 0/0/0/19/28 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:18 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45974 [15/Mar/2024:08:02:18.532] www-http swarm-scsp/gw2 0/0/0/18/24 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:19 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45978 [15/Mar/2024:08:02:19.577] www-http swarm-scsp/gw2 0/0/0/17/24 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:20 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45984 [15/Mar/2024:08:02:20.623] www-http swarm-scsp/gw2 0/0/0/16/23 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:21 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:45988 [15/Mar/2024:08:02:21.667] www-http swarm-scsp/gw2 0/0/0/16/25 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"

*** Here I start gw1 back up, and Haproxy detects it.

Mar 15 08:02:36 cert-prod-haproxy haproxy[1411077]: [WARNING]  (1411077) : Server swarm-scsp/gw1 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mar 15 08:02:36 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
Mar 15 08:02:36 cert-prod-haproxy haproxy[1411077]: Server swarm-scsp/gw1 is UP, reason: Layer7 check passed, code: 200, check duration: 18ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.

*** Here we are back using both gateways

Mar 15 08:02:37 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46086 [15/Mar/2024:08:02:37.293] www-http swarm-scsp/gw2 0/0/0/13/21 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:39 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46090 [15/Mar/2024:08:02:38.336] www-http swarm-scsp/gw1 0/0/0/894/899 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:40 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46100 [15/Mar/2024:08:02:40.256] www-http swarm-scsp/gw2 0/0/0/14/21 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.a|} "GET /test14/test2 HTTP/1.1"
Mar 15 08:02:41 cert-prod-haproxy haproxy[1411077]: 172.30.4.246:46106 [15/Mar/2024:08:02:41.299] www-http swarm-scsp/gw1 0/0/0/18/23 200 1058993 - - ---- 1/1/0/0/0 0/0 {|swarm.certlab.datacore.com|curl/7.29.0|||Basic YWRtaW46ZGF0YWNvcmU=} {CAStor Cluster/16.0.0|} "GET /test14/test2 HTTP/1.1"

Create a Self-Signed SSL Certificate

First we need to make an openssl.conf file, in this example YOUR_DOMAIN = swarm.example.com

[ req ]
prompt = no
distinguished_name = server_distinguished_name
req_extensions = v3_req
default_md = sha256

[ server_distinguished_name ]
commonName = *.swarm.example.com
stateOrProvinceName = Texas
countryName = US
emailAddress = admin@example.com
organizationName = Example Inc.
localityName = Austin

[ v3_req ]
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

[ alt_names ]
DNS.1=swarm.example.com
DNS.2=*.swarm.example.com

Generate the self-signed CA private key

openssl genrsa  -out selfsignCA.key 4096

Generate the self-signed CA root certificate

openssl req -new -x509 -nodes -days 365 -sha256 -key selfsignCA.key -out selfsignCA.crt -subj "/C=US/O=_DEV CA/CN=SelfSigned certificates"

Generate wildcard custom domain private key

openssl genrsa -out YOUR_DOMAIN.key 4096

Generate certificate signing request for your domain

openssl req -new -nodes -key YOUR_DOMAIN.key -config openssl.conf -out YOUR_DOMAIN.csr

Generate the final domain certificate

openssl x509 -req -in YOUR_DOMAIN.csr -CA selfsignCA.crt -CAkey selfsignCA.key -CAcreateserial -out YOUR_DOMAIN.crt -days 365 -sha256 -extfile openssl.conf -extensions v3_req

Generate the certificate pem file.

cat YOUR_DOMAIN.crt YOUR_DOMAIN.key > YOUR_DOMAIN.pem

Place the YOUR_DOMAIN.pem file where you configured it in haproxy.cfg example here put it in /etc/pki/tls/certs

bind 0.0.0.0:443 ssl crt /etc/pki/tls/certs/YOUR_DOMAIN.pem

Copy the selfsignCA.crt to /etc/pki/ca-trust/source/anchors and run update-ca-trust to tell CentOS to trust your self-signed root certificate.

This is an important step for haproxy as by default it is configured to look for trusted certificates in the /etc/pki/tls/certs folder. ( see ca-base parameter )

Restart haproxy to activate the changes.

Now you can copy the selfsignCA.crt file to your clients and follow the browser specific procedures to install and accept it.

The host <contentgatewayIP>:8091 needs to be used in the login page to connect for Swarm UI in the configuration example above.

Replication Feed configuration

The following setting must appear and be set properly in the /etc/caringo/cloudgateway/gateway.cfg file if the content gateway is going to be used as the destination for a remote replication feed:

[scsp]
...
allowSwarmAdminIP=172.30

In the example above, replicate "172.30" with the IP addresses (or prefix) of clients sending administrative requests to the gateway.

The most common example is the IP addresses (or prefix) of the nodes in a cluster using a remote replication feed with the gateway as the destination.

Troubleshooting tips

If you occasionally see the error SEC_ERROR_UNKNOWN_ISSUER , verify you don't have lingering old haproxy processes , and kill them manually.

To check if your selfsigned CA certificate is trusted run the following command:

trust list --filter=ca-anchors | grep Self -i -A2 -B4

Example Output you should see:
pkcs11:id=%f4%a2%f6%c6%e4%db%bc%c8%a1%23%83%d3%67%14%7c%51%c1%8d%bd%ba;type=cert
    type: certificate
    label: SelfSigned certificates
    trust: anchor
    category: authority

for CentOS7 and haproxy 1.8 by default require additional log settings to see the output in a file, add the following line in /etc/rsyslog.conf after the boot.log 

local2.*    /var/log/haproxy.log

then run 

systemctl restart rsyslog

you should now see haproxy logging in /var/log/haproxy.log

Do not forget to enable the ports on your firewall

firewall-cmd --add-port=443/tcp --permanent
firewall-cmd --add-port=80/tcp --permanent
firewall-cmd --reload

Keep in mind that normally you have multiple zone’s , since gateway has 2 adapters, the above example only adds the ports to the Default firewall zone


  • No labels

© DataCore Software Corporation. · https://www.datacore.com · All rights reserved.