Managing Failovers with Keepalived & HAproxy

5 min readJul 17, 2019

“The keepalived runs on both the active and passive LVS routers. All routers running keepalived use the Virtual Redundancy Routing Protocol (VRRP). The active router sends VRRP advertisements at periodic intervals; if the backup/slave routers fail to receive these advertisements, a new active router(slave) is elected.Keepalived performs failover on layer 4, or the Transport layer.”

Above is a small description quoted from Redhat documentation about Keepalived [1]. We are going to maintain 2 Load Balancers with HAproxy installed and Keepalived connection to keep high availability.

“ HAProxy performs load-balancing management on layer 7, or the Application layer. In most cases, administrators deploy HAProxy for HTTP-based load balancing [1]”

HAproxy is similar to Nginx, where it's used for load balancing. First, create 2 Virtual Machines (VM) in GCP, AWS or suitable cloud service provider. Operating systems : Linux / Unix Distribution (Ubuntu / CentOS / Fedora etc. ). We are going to keep these 2 VM’s as 2 Load Balancers.

We need Virtual IP for(VIP) Keepalived Configurations. So make sure you have/create a VIP also. Now log into one VM.

As Pre-requisites, make sure you can ping from one instance to another after the creation of instances. If they are not under the same subnet, make sure to create an OpenVPN and connect other instances as clients to it. So Keepalived works fine.

Keepalived Configurations

Installing Keepalived on Master Node

yum install -y keepalivedvim /etc/keepalived/keepalived.conf

Edit the .conf file as below.

global_defs {
   notification_email {
  root@webserver-01.example.com
   }
   notification_email_from root@webserver-01.example.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
  script "killall -0 haproxy" # check the haproxy process
  interval 2 # every 2 seconds
  weight 2 # add 2 points if OK
}vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 101 # 101 for MASTER AND 100 for SLAVE
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    unicast_src_ip # PRIMARY PRIVATE IP
    unicast_peer {
        # SECONDARY PRIVATE IP
    }
    virtual_ipaddress {
        35.190.39.153 # VIRTUAL IP CREATED
    } 
    track_script {
    chk_haproxy
  }
}

Run the Keepalived.

systemctl enable keepalived
systemctl start keepalived

Type this command and see whether eth0 is assigned. If assigned Keepalived is successful on the master.

ip addr | grep "eth0" output :2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1460 qdisc pfifo_fast state UP group default qlen 1000
    inet X.X.X.X/ brd X.X.X.X scope global noprefixroute dynamic eth0
    inet X.X.X.X/X scope global eth0

Check Keepalived status using

systemctl status  keepalived● keepalived.service - LVS and VRRP High Availability Monitor
   Loaded: loaded (/usr/lib/systemd/system/keepalived.service; enabled; vendor preset: disabled)
   Active: active (running) since 2019-07-09 08:45:01 UTC; 26min ago
 Main PID: 17663 (keepalived)
   CGroup: /system.slice/keepalived.service
           ├─17663 /usr/sbin/keepalived -D
           ├─17664 /usr/sbin/keepalived -D
           └─17665 /usr/sbin/keepalived -D

Installing Keepalived on Slave Node

yum install -y keepalivedvim /etc/keepalived/keepalived.conf

Edit the .conf file as below.

global_defs {
   notification_email {
  root@webserver-01.example.com
   }
   notification_email_from root@webserver-01.example.com
   smtp_server 127.0.0.1
   smtp_connect_timeout 30
   router_id LVS_DEVEL
}
vrrp_script chk_haproxy {
  script "killall -0 haproxy" # check the haproxy process
  interval 2 # every 2 seconds
  weight 2 # add 2 points if OK
}vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 100 # PRIORITY SHOULD BE LOWER THAN MASTER VALUE
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1111
    }
    unicast_src_ip # SECONDARY PRIVATE IP
    unicast_peer {
        #  PRIMARY PRIVATE IP
   }
    virtual_ipaddress {
        35.190.39.153 # VIRTUAL IP CREATED
    }
track_script {
    chk_haproxy
  }
}

Check the status of Keepalived.

systemctl status  keepalived

HAproxy Configurations

Installing HAproxy in Both Nodes

yum install haproxy
vim /etc/haproxy/haproxy.cfg

This shows the HAproxy configuration.

global
        log 127.0.0.1   local0
        log 127.0.0.1   local1 debug
        maxconn   45000 # Total Max Connections.
        daemon
        nbproc      1 # Number of processing cores.
defaults
        timeout server 86400000
        timeout connect 86400000
        timeout client 86400000
        timeout queue   1000s

# [HTTP Site Configuration]
listen  http_web #VIRTUAL_IP:80
        mode http
        balance roundrobin  # Load Balancing algorithm
        option httpchk
        option forwardfor
        server server1 #MASTER:80 weight 1 maxconn 512 check
        server server2 #SLAVE:80 weight 1 maxconn 512 check

# [HTTPS Site Configuration]
listen  https_web #VIRTUAL_IP:443
        mode tcp
        balance source# Load Balancing algorithm
        reqadd X-Forwarded-Proto: http
        server server1 #MASTER:443 weight 1 maxconn 512 check
        server server2 #SLAVE:443 weight 1 maxconn 512 check

Start HAProxy Service.

service haproxy start
chkconfig haproxy on

Apache Configuration of Master & Slave

Install the Apache in both Master and Slave.

sudo yum install httpd
sudo systemctl enable httpd
sudo systemctl start httpd

Allow firewall for port 80 and 443

sudo firewall-cmd — permanent — zone=public — add-service=http
sudo firewall-cmd — permanent — zone=public — add-service=https
sudo firewall-cmd — reload

verify the installation

sudo systemctl status httpd
sudo httpd -vOutput : 
Server version: Apache/2.4.6 (CentOS)
Server built:   Oct 19 2017 20:39:16

Go to your primary IP address. http://X.X.X.X/. Then you will see the Apache default HTML page. The default index.html file is located here. Change the HTML and make it seems as the MASTER NODE.

/usr/share/httpd/noindex/index.html

Tip: You can simply select * and delete with the following command in vim.

Press ESC -> :1,$d

Add the following HTML.

<!DOCTYPE html>
<html>
<body>
<h1>MASTER NODE</h1>
</body>
</html>

Now you have to restart Apache.

sudo systemctl restart httpd

Now visit master and slave nodes from the browser and check the effect.

Enable Logs in CentOS

yum -y install rsyslog
vi /etc/rsyslog.conf

Edit as follows

# Provides UDP syslog reception
$ModLoad imudp
$UDPServerRun 514

# Provides TCP syslog reception
$ModLoad imtcp
$InputTCPServerRun 514

Restart rsyslog.service

systemctl restart rsyslog.service

Verify the syslog server listening

netstat -antup | grep 514

Now you can tail the logs

tailf /var/log/messages

Verify Keepalived — Automatic Virtual IP assignment

Go to master and check the assigned Virtual IP address to interface eth0.

Browse to Virtual IP from browser. You need to see the master HTML page.

Stop the master node.

Browse to Virtual IP from browser. You need to see the slave HTML page.

Check virtual IP assigned to the eth0 interface.

References

[1] Redhat/CentOS Documentation- Load Balancing

[2] HAproxy Documentation