关于FastDFS蛋疼的集群和负载均衡(十八)之LVS+Keepalived双主模式-原创手记-慕课网

Interesting things

我们之前的Keepalived+LVS单主模式，是只有一台lvs工作，这会造成资源浪费，可以采用双主结构，让两台lvs都进行工作，采用dns轮询方式，当用户访问域名通过dns轮询每天lvs，双主结构需要2个vip，这2个vip需要绑定域名。

同样，在每台lvs上安装keepalived软件，当keepalived检测到其中一个lvs宕机则将宕机的vip漂移到活动lvs上，当lvs恢复则vip又重新漂移回来。

附上我画的拓扑图
初始状态

其中一个主机宕机

主机恢复

所需环境
vip1 192.168.12.101
vip2 192.168.12.102
lvs_master1 192.168.12.12
lvs_master2 192.168.12.13
nginx1 192.168.12.2
nginx2 192.168.12.3
tomcat1 192.168.12.6
tomcat2 192.168.12.7

What did you do today

双主模式相比主从环境，区别在于：
1.DNS轮询。
2.LVS负载均衡层需要2个vip。比如192.168.12.12和192.168.12.13
3.后端的realServer上要绑定这2个vip到lo本地回环设备上
4.keepalived.conf的配置相比于上面的主从模式有所不同。

在192.168.12.2和192.168.12.3机器要绑定2个vip到本地回环口lo上(分别绑定lo:0和lo:1),所以需要在/etc/init.d/下编写double_master_lvsdr0和double_master_lvsdr1脚本，具体如下：

#!/bin/sh
VIP=192.168.12.101
. /etc/rc.d/init.d/functions

case "$1" in

start)
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/sysctl -p >/dev/null 2>&1
    /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up  
    /sbin/route add -host $VIP dev lo:0
    echo "LVS-DR real server starts successfully.\n"
    ;;
stop)
    /sbin/ifconfig lo:0 down
    /sbin/route del $VIP >/dev/null 2>&1
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
    ;;
status)
    isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`
    isRoOn=`/bin/netstat -rn | grep "$VIP"`
    if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
        echo "LVS-DR real server has run yet."
    else
        echo "LVS-DR real server is running."
    fi
    exit 3
    ;;
*)
    echo "Usage: $0 {start|stop|status}"
    exit 1
esac
exit 0

#!/bin/sh
VIP=192.168.12.102
. /etc/rc.d/init.d/functions

case "$1" in

start)
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/sysctl -p >/dev/null 2>&1
    /sbin/ifconfig lo:1 $VIP netmask 255.255.255.255 up   
    /sbin/route add -host $VIP dev lo:1
    echo "LVS-DR real server starts successfully.\n"
    ;;
stop)
    /sbin/ifconfig lo:1 down
    /sbin/route del $VIP >/dev/null 2>&1
    echo "1" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "1" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "2" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
    ;;
status)
    isLoOn=`/sbin/ifconfig lo:1 | grep "$VIP"`
    isRoOn=`/bin/netstat -rn | grep "$VIP"`
    if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
        echo "LVS-DR real server has run yet."
    else
        echo "LVS-DR real server is running."
    fi
    exit 3
    ;;
*)
    echo "Usage: $0 {start|stop|status}"
    exit 1
esac
exit 0

将double_master_lvsdr0和double_master_lvsdr1设置开机启动

[root@localhost init.d]# chmod +x double_master_lvsdr0
[root@localhost init.d]# chmod +x double_master_lvsdr1
[root@localhost init.d]# echo "/etc/init.d/double_master_lvsdr0" >> /etc/rc.d/rc.local
[root@localhost init.d]# echo "/etc/init.d/double_master_lvsdr1" >> /etc/rc.d/rc.local
启动double_master_lvsdr0和double_master_lvsdr1脚本
查看192.168.12.2和192.168.12.3，发现vip已经成功绑定到本地回环口lo上了。
在lvs_master1和lvs_master2打开ip_forward路由转发功能

[root@localhost ~]# echo "1" > /proc/sys/net/ipv4/ip_forward
lvs_master1上的keepalived.conf配置如下：
```
! Configuration File for keepalived
```

global_defs {
router_id LVS_MASTER
}

vrrp_script check_lvs {
script "/etc/keepalived/lvs_check.sh"
interval 2
weight -20
}

vrrp_instance VI_1 {
state MASTER
interface eth0
virtual_router_id 51
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
    check_lvs
}

virtual_ipaddress {
    192.168.12.101
}

}

vrrp_instance VI_2 {
state BACKUP
interface eth0
virtual_router_id 52
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
    check_lvs
}

virtual_ipaddress {
    192.168.12.102
}

}

virtual_server 192.168.12.101 80 {
delay_loop 6
lb_algo wrr
lb_kind DR

nat_mask 255.255.255.0

persistence_timeout 50
protocol TCP

real_server 192.168.12.2 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}
real_server 192.168.12.3 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}

}

virtual_server 192.168.12.102 80 {
delay_loop 6
lb_algo wrr
lb_kind DR

nat_mask 255.255.255.0

persistence_timeout 50
protocol TCP

real_server 192.168.12.2 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}
real_server 192.168.12.3 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}

}

* lvs_master2中的keepalived配置如下:

! Configuration File for keepalived

global_defs {
router_id LVS_BACKUP
}

vrrp_script check_lvs {
script "/etc/keepalived/lvs_check.sh"
interval 2
weight -20
}

vrrp_instance VI_1 {
state BACKUP
interface eth0
virtual_router_id 51
priority 90
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
    check_lvs
}

virtual_ipaddress {
    192.168.12.101       
}

}

vrrp_instance VI_2 {
state MASTER
interface eth0
virtual_router_id 52
priority 100
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}

track_script {
    check_lvs
}

virtual_ipaddress {
    192.168.12.102   
}

}

virtual_server 192.168.12.101 80 {
delay_loop 6
lb_algo wrr
lb_kind DR

nat_mask 255.255.255.0

persistence_timeout 50   
protocol TCP            

real_server 192.168.12.2 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}
real_server 192.168.12.3 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}

}

virtual_server 192.168.12.102 80 {
delay_loop 6
lb_algo wrr
lb_kind DR

nat_mask 255.255.255.0

persistence_timeout 50   
protocol TCP            

real_server 192.168.12.2 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}
real_server 192.168.12.3 80 {
    weight 3
    TCP_CHECK {
        connect_timeout 3
        nb_get_retry 3
        delay_before_retry 3
        connect_port 80
    }
}

}

* 编写lvs_check.sh脚本。

a=ipvsadm -ln
str="Route"
bb=echo $aa|grep $str|wc -l
if [ $bb = 0 ];then
sleep 3
aa=ipvsadm -ln
bb=echo $aa|grep $str|wc -l
if [ $bb = 0 ];then
killall keepalived
fi
fi


* 启动192.168.12.2和192.16812.3的nginx、double_master_lvsdr0、double_master_lvsdr1服务。启动192.168.12.6和192.168.12.7的tomcat。

* 我们查看lvs_master1的eth0节点信息，发现绑定了vip1(192.168.12.101)
![image.png](http://upload-images.jianshu.io/upload_images/4636177-1ec80ce5b3c39990.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 查看lvs_master2的eth0节点信息，发现绑定了vip2(192.168.12.102)
![image.png](http://upload-images.jianshu.io/upload_images/4636177-9df30314c5601ee4.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 查看lvs_master1的lvs以及realserver的信息。
![image.png](http://upload-images.jianshu.io/upload_images/4636177-a90bea0a05196c38.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 查看lvs_master2的lvs以及realserver的信息
![image.png](http://upload-images.jianshu.io/upload_images/4636177-d10f3a86f051599b.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 修改hosts文件(C:\Windows\System32\drivers\etc\hosts)，指定cmaxiaoma.mayday.com对应的2个vip。(在hosts里这样设置，达不到负载均衡，只会优先访问192.168.12.101)
![image.png](http://upload-images.jianshu.io/upload_images/4636177-b60bd8fe60e0ce8d.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 访问cmazxiaoma.mayday.com
![image.png](http://upload-images.jianshu.io/upload_images/4636177-aa12a5d1cdf0f6c9.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 我们停止掉lvs_master1.
![image.png](http://upload-images.jianshu.io/upload_images/4636177-c9f3d54af26e9da0.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 当我们又恢复lvs_master1.vip1又回到了lvs_master1手里，而lvs_master2又失去了vip1。
![image.png](http://upload-images.jianshu.io/upload_images/4636177-f53b07003ba98555.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
![image.png](http://upload-images.jianshu.io/upload_images/4636177-ef8a6dda1aaa60e2.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

* 在lvs_master2的eth0以及lvs、RealServer信息。
![image.png](http://upload-images.jianshu.io/upload_images/4636177-ee8153e78025ef09.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)
![image.png](http://upload-images.jianshu.io/upload_images/4636177-292684945c1e31cd.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

###Summary
> 今天加班就到这里了，回去要照顾女朋友了！