18 June 2023

vrrp and keepalived

Briefly keepalived is a utility for attaching virtual ip addresses to hosts that can fail over to another host when a check condition fails. keepalived uses the vrrp protocol, which sends status messages to a multicast address.

At the command line keepalived provides no utility that can tell whether nodes are exchanging their broadcast messages or which node is currently MASTER for a VIP.

keepalived can be made to dump some stats, but it requires sending a signal to the application. The stats will give an indication of packets sent and received, it doesn’t tell you directly which node currently has a VIP, but it can be used to deduce where master is – subtract the number of times the node released master from the number of times it became master, if the result is positive that node is master (if its greater than 1 there might be problem).

It takes two commands to see the stats, fortunately it is easy to wrap it in a shell script.

#!/usr/bin/bash

kill -s $(keepalived --signum=STATS) \
    $(cat /var/run/keepalived.pid)
cat /tmp/keepalived.stats

The keepalived stats dump has to be run against each host, and perhaps the most important item, which host is master for each of your virtual ips, needs to be inferred.

Since keepalived uses vrrp, it sends out vrrp broadcasts, at a frequency specified by the advert_int in each vrrp_instance block, most commonly this is 1. The vrrp broadcasts go to multicast 224.0.0.18, any host on the subnet (possibly further depending your router configuration) can hear all of these broadcasts.

The following command will capture 4 seconds of activity:

 tcpdump -v -i ${interface} host 224.0.0.18 -w /tmp/tcp.out -W 1 -G 4

To dump a readable version to both your screen and a text file:

tcpdump -r /tmp/tcp.out -e -n -v | tee /tmp/tcp.txt

You’ll get output that looks like:

1  11:56:12.537357 52:54:00:00:00:00 > 01:00:5e:00:00:12,
   >> ethertype IPv4 (0x0800), length 60:
   >> (tos 0xc0, ttl 255, id 22132, offset 0,
   >> flags [none], proto VRRP (112), length 40)
2  10.10.1.2 > 224.0.0.18: vrrp 10.10.1.2 > 224.0.0.18:
   >> VRRPv2, Advertisement, vrid 201, prio 106,
   >> authtype simple, intvl 1s, length 20,
   >> addrs: 10.10.1.200 auth "8 char key in clear text"

You can see that the auth strings the nodes use to authenticate each other aren’t secure at all, they’re broadcast in the clear, the insecure authentication has been removed from the vrrp specification, but keepalived still supports it. Given how insecure it is, and that other hosts would have no way of knowing the legitimate host if a rogue were advertising the same ip address, you may as well remove the authentication section from your configs.

I’ve written a perl script to capture tcpdump and output a table, I’ve published it as a snippet on bitbucket: https://bitbucket.org/brainbuz/workspace/snippets/eqaokz/vrrpinfo.

vrid host       hostname         vrrp_vip      prio mac
65   10.10.1.2  lb1.mydomain.    10.10.1.201   106  52:54:00:00:00:11
                                 10.10.1.25
66   10.10.1.3  lb2.mydomain.    10.10.1.202   106  52:54:00:00:00:12
                                 10.10.1.26
70   10.10.2.20 devlb0.mydomain. 10.20.2.2     202  52:54:00:00:00:13
71   10.10.2.21 devlb1.mydomain. 10.20.2.3     202  52:54:00:00:00:14