Keepalived无法绑定VIP故障排查经历.docx
《Keepalived无法绑定VIP故障排查经历.docx》由会员分享，可在线阅读，更多相关《Keepalived无法绑定VIP故障排查经历.docx（32页珍藏版）》请在冰豆网上搜索。
Keepalived无法绑定VIP故障排查经历
Keepalived无法绑定VIP故障排查经历
[日期：
2015-03-14]
来源：
Linux社区 作者：
john88wang
[字体：
大 中 小]
一故障描述
我在台湾合作方给定的两台虚拟机上部署HAProxy+Keepalived负载均衡高可用方案。
在配置完Keepalived后，重新启动Keepalived，Keepalived没有绑定VIP。
Keepalived执行程序路径为/data/app_platform/keepalived/sbin/keepalived
配置文件路径为/data/app_platform/keepalived/conf/keepalived.conf
Keepalived的启动脚本为/etc/init.d/keepalived
keepalived.conf的内容
LB1Master
!
ConfigurationFileforkeepalived
global_defs{
notification_email{
admin@
}
notification_email_fromlb1@
smtp_server127.0.0.1
smtp_connect_timeout30
router_idLB1_MASTER
}
vrrp_scriptchk_haproxy{
script"killall-0haproxy"
interval2
weight2
}
vrrp_instanceVI_1{
stateMASTER
interfaceeth1
virtual_router_id51
priority100
advert_int1
authentication{
auth_typePASS
auth_pass1111
}
virtual_ipaddress{
10.1.1.200/24 brd10.1.1.255 deveth1labeleth1:
vip
}
track_script{
chk_haproxy
}
}
重新启动Keepalived查看日志
Mar 318:
09:
00cv00300005248-1Keepalived[20138]:
StoppingKeepalivedv1.2.15（02/28,2015）
Mar 318:
09:
00cv00300005248-1Keepalived[20259]:
StartingKeepalivedv1.2.15（02/28,2015）
Mar 318:
09:
00cv00300005248-1Keepalived[20260]:
StartingHealthcheckchildprocess,pid=20261
Mar 318:
09:
00cv00300005248-1Keepalived[20260]:
StartingVRRPchildprocess,pid=20262
Mar 318:
09:
00cv00300005248-1Keepalived_vrrp[20262]:
RegisteringKernelnetlinkreflector
Mar 318:
09:
00cv00300005248-1Keepalived_vrrp[20262]:
RegisteringKernelnetlinkcommandchannel
Mar 318:
09:
00cv00300005248-1Keepalived_vrrp[20262]:
RegisteringgratuitousARPsharedchannel
Mar 318:
09:
00cv00300005248-1Keepalived_healthcheckers[20261]:
RegisteringKernelnetlinkreflector
Mar 318:
09:
00cv00300005248-1Keepalived_healthcheckers[20261]:
RegisteringKernelnetlinkcommandchannel
Mar 318:
09:
00cv00300005248-1Keepalived_healthcheckers[20261]:
Configurationisusing:
3924Bytes
Mar 318:
09:
00cv00300005248-1Keepalived_healthcheckers[20261]:
UsingLinkWatchkernelnetlinkreflector...
Mar 318:
09:
00cv00300005248-1Keepalived_vrrp[20262]:
Configurationisusing:
55712Bytes
Mar 318:
09:
00cv00300005248-1Keepalived_vrrp[20262]:
UsingLinkWatchkernelnetlinkreflector...
Mar 318:
09:
18cv00300005248-1kernel:
__ratelimit:
1964callbackssuppressed
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
Mar 318:
09:
18cv00300005248-1kernel:
Neighbourtableoverflow.
查看VIP绑定情况
$ifconfigeth1:
vip
eth1:
vip Linkencap:
Ethernet HWaddr00:
16:
3E:
F2:
37:
6B
UPBROADCASTRUNNINGMULTICAST MTU:
1500 Metric:
1
Interrupt:
13
没有VIP绑定
二排查过程
1）检查VIP的配置情况
向合作方确认提供的VIP的详细情况
IPADDR 10.1.1.200
NETMASK 255.255.255.0
GATEWAY 10.1.1.1
Brodcast 10.1.1.255
这里设置的是
10.1.1.200/24 brd10.1.1.255 deveth1labeleth1:
vip
2）检查iptables和selinux的设置情况
$sudoserviceiptablesstop
$sudosetenforce0
setenforce:
SELinuxisdisabled
如果非要开启iptables的话，需要作些设定
iptables-IINPUT-ieth1-d224.0.0.0/8-jACCEPT
serviceiptablessave
keepalived使用224.0.0.18作为Master和Backup健康检查的通信IP
3）检查相关的内核参数
HAProxy+Keepalived架构需要注意的内核参数有：
#ControlsIPpacketforwarding
net.ipv4.ip_forward=1
开启IP转发功能
net.ipv4.ip_nonlocal_bind=1
开启允许绑定非本机的IP
如果使用LVS的DR或者TUN模式结合Keepalived需要在后端真实服务器上特别设置两个arp相关的参数。
这里也设置好。
net.ipv4.conf.lo.arp_ignore=1
net.ipv4.conf.lo.arp_announce=2
net.ipv4.conf.all.arp_ignore=1
net.ipv4.conf.all.arp_announce=2
4）检查VRRP的设置情况
LB1Master
stateMASTER
interfaceeth1
virtual_router_id51
priority100
LB2Backup
stateBACKUP
interfaceeth1
virtual_router_id51
priority99
Master和Backup的virtual_router_id需要一样，priority需要不一样，数字越大，优先级越高
5）怀疑是编译安装Keepalived版本出现了问题
重新下载并编译2.1.13的版本，并重新启动keepalived，VIP仍然没有被绑定。
线上有个平台的keepalived是通过yum安装的，于是打算先用yum安装keepalived后将配置文件复制过去看看是否可以绑定VIP
rpm-ivhhttp:
//ftp.linux.ncsu.edu/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
yum-yinstallkeepalived
cp/data/app_platform/keepalived/conf/keepalived.conf /etc/keepalived/keepalived.conf
重新启动keepalived
然后查看日志
Mar 416:
42:
46xxxxxKeepalived_healthcheckers[17332]:
RegisteringKernelnetlinkreflector
Mar 416:
42:
46xxxxxKeepalived_healthcheckers[17332]:
RegisteringKernelnetlinkcommandchannel
Mar 416:
42:
46xxxxxKeepalived_vrrp[17333]:
Openingfile'/etc/keepalived/keepalived.conf'.
Mar 416:
42:
46xxxxxKeepalived_vrrp[17333]:
Configurationisusing:
65250Bytes
Mar 416:
42:
46xxxxxKeepalived_vrrp[17333]:
UsingLinkWatchkernelnetlinkreflector...
Mar 416:
42:
46xxxxxKeepalived_vrrp[17333]:
VRRPsockpool:
[ifindex（3）,proto（112）,unicast（0）,fd（10,11）]
Mar 416:
42:
46xxxxxKeepalived_healthcheckers[17332]:
Openingfile'/etc/keepalived/keepalived.conf'.
Mar 416:
42:
46xxxxxKeepalived_healthcheckers[17332]:
Configurationisusing:
7557Bytes
Mar 416:
42:
46xxxxxKeepalived_healthcheckers[17332]:
UsingLinkWatchkernelnetlinkreflector...
Mar 416:
42:
46xxxxxKeepalived_vrrp[17333]:
VRRP_Script（chk_haproxy）succeeded
Mar 416:
42:
47xxxxxKeepalived_vrrp[17333]:
VRRP_Instance（VI_1）TransitiontoMASTERSTATE
Mar 416:
42:
48xxxxxKeepalived_vrrp[17333]:
VRRP_Instance（VI_1）EnteringMASTERSTATE
Mar 416:
42:
48xxxxxKeepalived_vrrp[17333]:
VRRP_Instance（VI_1）settingprotocolVIPs.
Mar 416:
42:
48xxxxxKeepalived_vrrp[17333]:
VRRP_Instance（VI_1）SendinggratuitousARPsoneth1for10.1.1.200
Mar 416:
42:
48xxxxxKeepalived_healthcheckers[17332]:
NetlinkreflectorreportsIP10.1.1.200added
Mar 416:
42:
53xxxxxKeepalived_vrrp[17333]:
VRRP_Instance（VI_1）SendinggratuitousARPsoneth1for10.1.1.200
再查看IP绑定情况
$ifconfigeth1:
vip
eth1:
vip Linkencap:
Ethernet HWaddr00:
16:
3E:
F2:
37:
6B
inetaddr:
10.1.1.200 Bcast:
10.1.1.255 Mask:
255.255.255.0
UPBROADCASTRUNNINGMULTICAST MTU:
1500 Metric:
1
Interrupt:
13
再通过yum将keepalived卸载掉
yumremovekeepalived
恢复到原来的启动脚本/etc/init.d/keepalived
重新启动keepalived后还是无法绑定VIP
怀疑是keepalived启动脚本/etc/init.d/keepalived的问题
检查/etc/init.d/keepalived
#Sourcefunctionlibrary.
./etc/rc.d/init.d/functions
exec="/data/app_platform/keepalived/sbin/keepalived"
prog="keepalived"
config="/data/app_platform/keepalived/conf/keepalived.conf"
[-e/etc/sysconfig/$prog]&&./etc/sysconfig/$prog
lockfile=/var/lock/subsys/keepalived
start（）{
[-x$exec]||exit5
[-e$config]||exit6
echo-n$"Starting$prog:
"
daemon$exec$KEEPALIVED_OPTIONS
retval=$?
echo
[$retval-eq0]&&touch$lockfile
return$retval
}
关键是这一行
daemon$exec$KEEPALIVED_OPTIONS
由于没有复制/etc/sysconfig/keepalived,所以将直接执行damon/data/app_platform/keepalived/sbin/keepalived
由于keepalived默认使用的是/etc/keepalived/keepalived.conf作为配置文件，而这里指定了不同的配置文件，所以要修改成为
daemon$exec-D-f$config
重新启动keepalived，查看日志和VIP绑定情况
$ifconfigeth1:
vip
eth1:
vip Linkencap:
Ethernet HWaddr00:
16:
3E:
F2:
37:
6B
inetaddr:
10.1.1.200 Bcast:
10.1.1.255 Mask:
255.255.255.0
UPBROADCASTRUNNINGMULTICAST MTU:
1500 Metric:
1
Interrupt:
13
6）将LB2Backup的keepalived启动脚本也修改一下，观察VIP接管情况
查看LB1Master
$ifconfigeth1:
vip
eth1:
vip Linkencap:
Ethernet HWaddr00:
16:
3E:
F2:
37:
6B
inetaddr:
10.1.1.200 Bcast:
10.1.1.255 Mask:
255.255.255.0
UPBROADCASTRUNNINGMULTICAST MTU:
1500 Metric:
1
Interrupt:
13
查看LB2Backup
$ifconfigeth1:
vip
eth1:
vip Linkencap:
Ethernet HWaddr00:
16:
3E:
F2:
37:
6B
inetaddr:
10.1.1.200 Bcast:
10.1.1.255 Mask:
255.255.255.0
UPBROADCASTRUNNINGMULTICAST MTU:
1500 Metric:
1
Interrupt:
13
问题出现了，LB1Master和LB2Backup都绑定了VIP10.1.1.200，这是不正常的！
！
！
！
在LB1和LB2上登录10.1.1.200看看
[lb1~]$ssh10.1.1.200
Lastlogin:
WedMar 417:
31:
332015from10.1.1.200
[lb1~]$
[lb2~]$ssh10.1.1.200
Lastlogin:
WedMar 417:
54:
572015from101.95.153.246
[b2 ~]$
在LB1上停掉keepalived，ping下10.1.1.200这个IP，发现无法ping通
在LB2上停掉keepalived，ping下10.1.1.200这个IP，发现也无法ping通
然后开启LB1上的keepalived，LB1上可以ping通10.1.1.200,LB2上不行
开启LB2上的keepalived，LB2上可以ping通10.1.1.200
由此得出，LB1和LB2各自都将VIP10.1.1.200绑定到本机的eth1网卡上。
两台主机并没有VRRP通信，没有VRRP的优先级比较。
7）排查影响VRRP通信的原因
重新启动LB1Master的Keepalived查看日志
Mar 515:
45:
36gintama-taiwan-lb1Keepalived_vrrp[32303]:
Configurationisusing:
65410Bytes
Mar 515:
45:
36gintama-taiwan-lb1Keepalived_vrrp[32303]:
UsingLinkWatchkernelnetlinkreflector...
Mar 515:
45:
36gintama-taiwan-lb1Keepalived_vrrp[32303]:
VRRPsockpool:
[ifindex（3）,proto（112）,unicast（0）,fd（10,11）]
Mar 515:
45:
36gintama-taiwan-lb1Keepalived_vrrp[32303]:
VRRP_Script（chk_haproxy）succeeded
Mar 515:
45:
37gintama-taiwan-lb1Keepalived_vrrp[32303]:
VRRP_Instance（VI_1）TransitiontoMASTERSTATE
Mar 515:
45:
38gintama-taiwan-lb1Keepalived_vrrp[32303]:
VRRP_Instance（VI_1）EnteringMASTERSTATE
Mar 515:
45:
38gintama-taiwan-lb1Keepalived_vrrp[32303]:
VRRP_Instance（VI_1）settingprotocolVIPs.
Mar 515:
45:
38gintama-taiwan-lb1Keepalived_vrrp[32303]:
VRRP_Instance（VI_1）SendinggratuitousARPsoneth1for10.1.1.200
Mar 515:
45:
38gintama-taiwan-lb1Keepalived_
升级会员 