Outgoing traffic with virtualIP using Keepalived and Haproxy

Motivation for writing this blog is the recent problem that I faced while working on Infrastructure where white-listing became a challenge. By this blog, I actually want to share the problem which I had and how I implemented a hack for that.

Let’s take a common scenario where applications need connectivity on read replicas of database. For this requirement, we can make highly available setup of read replicas where HaProxy will sit on top of those replicas as TCP load balancer and application will connect to HaProxy to access databases (as shown in the following image).

In the above image, Apps are connecting HaProxy over the Virtual IP (192.168.5.16) and HaProxy will further connect to databases with their respective Ips (192.168.5.17 and 192.168.5.18) to serve the requests by the application.

This setup works well. However the point here is, we need to take care of the white-listing of HaProxy Ips at DB level. Whenever, a new HaProxy box introduced, which can be a replacement of the existing one, we need to white-list the Ip of new HaProxy into Databases to receive the connections from it.

As you can see in the above image, I replaced 192.168.5.18 with new HaProxy(192.168.5.19). Here, we need to delete the old IP from whitelisted Ips and add new Ip in the list.

This will become an operation which engineer needs to take care every time when a new box comes into the picture. I tried to make it generic and saved this operation time by using the VirtaulIp (Keepalived one) for incoming/outgoing connections instead of Haproxy’s real Ip.

As shown below:

Now, Apps will connect to HaProxy over the Virtual IP (192.168.5.16) and Active HaProxy will further connect to databases with its VirtualIp i.e (192.168.5.16).

In this case, we do not need to white-list Ips (192.168.5.17 and 192.168.5.18). This can solve the problem of white-listing HaProxy Ips on Db every time when a new box comes as replacement of other.

This was the overview of the problem and the solution to it, now let’s have a look at the configuration part to get it implemented. I tried to implement this with virtual routes in the routing table of kernel.

To use the same virtualIp as outingIp, add virtual route in Keepalived configuration. This is because whenever the virtualIP gets shifted on the other machine, it will create the route automatically and you don’t need to worry about maintaining the routes on the machine.

Prerequisites:

• 2 nodes of HaProxy and 1 instance of DB (mysql) will be enough for this test.
• Keepalived should be installed on both HaProxy

Create/Edit a file /etc/keepalived/keepalived.conf and paste the following content in it after replacing the values with <>with actual values.

vrrp_script chk_http_port {
 script “/usr/bin/killall -0 haproxy”
 interval 2
}
vrrp_instance VI_1 {
 interface <interface-name>
 virtual_router_id <any-id>
 state BACKUP
 priority 101
 advert_int 1
 authentication {
 auth_type NONE
 }
 track_script {
 chk_http_port
 }
 virtual_ipaddress {
 <virtual-ip>/24 dev <interface-name>
 }
 virtual_routes {
 <subnet/ip of database> via <gateway-ip> src <virtual-ip>
 }
}

Example configuration for reference:

vrrp_script chk_http_port {
 script “/usr/bin/killall -0 haproxy”
 interval 2
}
vrrp_instance VI_1 {
 interface eth0
 virtual_router_id 114
 state BACKUP
 priority 101
 advert_int 1
 authentication {
 auth_type NONE
 }
 track_script {
 chk_http_port
 }
 virtual_ipaddress {
 192.168.5.16/24 dev eth0
 }
 virtual_routes {
 192.168.7.0/24 via 192.168.5.10 src 192.168.5.16
 }
}

Place the edited file on both HaProxy instances and restart keepalived service afterwards.

How to verify the above changes.

You need to first check on which machine VirtualIp is present. You can check is by following command:

ip addr

The machine on which VirtualIP is present, check the route on that machine by the following command:

ip route

This will show the added route that we tried to injected through keepalived.

Note: You may also need to reload HaProxy.

Verify the new connections:

Try to connect through LB endpoint to connect to the database. After that, you can verify the connections by the following command:

netstat -ton | grep ESTABLISHED

Output of the above command will show the connections are made from the virtualIp of HaProxy to DB hosts.

There are other ways to achieve the same as well. I shared the one which I implemented and worked well for me. If somebody has the same use case, this might help. I will try to post other solutions to achieve the same requirement in near future. If anyone has other ways, please share it here.