一、架构
二、唠叨一会原理：
1、nginx
Nginx进程基于于Master+Slave(worker)多进程模型，自身具有非常稳定的子进程管理功能。在Master进程分配模式下，Master进程永远不进行业务处理，只是进行任务分发，
从而达到Master进程的存活高可靠性，Slave(worker)进程所有的业务信号都 由主进程发出，Slave(worker)进程所有的超时任务都会被Master中止，属于非阻塞式任务模型。
2、keepalived
Keepalived是Linux下面实现VRRP 备份路由的高可靠性运行件。基于Keepalived设计的服务模式能够真正做到主服务器和备份服务器故障时IP瞬间无缝交接,作用:
主要用作RealServer的健康状态检查以及LoadBalance主机和BackUP主机之间failover的实现
3、单点故障
Nginx有很强代理功能，但是一台nginx就形成了单点，现在使用keepalived来解决这个问题，keepalived的故障转移时间很短.
Nginx+keepalived双机实现nginx反向代理服务的高可用,一台nginx挂掉之后不影响应用也不影响内网访问外网.
4、此架构需要考虑的问题
1) Master没挂，则Master占有vip且nginx运行在Master上
2) Master挂了，则backup抢占vip且在backup上运行nginx服务
3) 如果master服务器上的nginx服务挂了，则vip资源转移到backup服务器上
4) 检测后端服务器的健康状态
5、叙述
Master和Backup两边都开启nginx服务，无论Master还是Backup，当其中的一个keepalived服务停止后，vip都会漂移到keepalived服务还在的节点上，
如果要想使nginx服务挂了，vip也漂移到另一个节点，则必须用脚本或者在配置文件里面用shell命令来控制。
首先必须明确后端服务器的健康状态检测keepalived在这种架构上是无法检测的，后端服务器的健康状态检测是有nginx来判断的，但是nginx的检测机制有一定的缺陷，后端服务器某一个宕机之后，nginx还是会分发请求给它，在一定的时间内后端服务响应不了，nginx则会发给另外一个服务器，然后当客户的请求来了，nginx会一段时间内不会把请求分发给已经宕机的服务器，但是过一段时间后，nginx还是会把分发请求发给宕机的服务器上。
三、准备工作
1、环境 系统版本：CentOS 6.4 64bit nginx版本：nginx-1.4.7.tar.gz keepalived版本：keepalived-1.2.7.tar.gz 2、地址规划 nginx+keepalived主 192.168.0.189 master nginx+keepalived辅 192.168.0.200 badkup vip 192.168.0.250 vip
四、安装nginx和keepalived服务
1、安装nginx服务[分别在master机器和backup机器安装nginx服务]
#！/bin/bash
# time 2014-08-18
# Allentuns
#安装依赖环境包
yum -y install gcc gcc-c++ make unzip wget vim openssh-clients
#支持重写rewrite,正则表达式
unzip pcre-8.33.zip
cd pcre-8.33
./configure --prefix=/usr/local/pcre
make && make install
cd ..
#支持网页压缩
tar xf zlib-1.2.8.tar.gz
cd zlib-1.2.8
./configure --prefix=/usr/local/zlib
make && make install
cd ..
#支持https
tar xf openssl-1.0.0l.tar.gz
cd openssl-1.0.0l
./config --prefix=/usr/local/openssl
make && make install
cd ..
#安装mp4和flv模块
tar xf nginx_mod_h264_streaming-2.2.7_update.tar.gz
#创建运行nginx服务的账号和组
groupadd -r nginx
useradd -g nginx -r -s /sbin/nologin nginx
#安装nginx
tar xf nginx-1.4.7.tar.gz
cd nginx-1.4.7
#启用flv和mp4功能
./configure --user=nginx --group=nginx --prefix=/usr/local/nginx --with-pcre=../pcre-8.33 --with-zlib=../zlib-1.2.8 --with-openssl=../openssl-1.0.0l --with-http_stub_status_module --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock --with-http_gzip_static_module --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --http-scgi-temp-path=/var/tmp/nginx/scgi --with-ipv6 --with-http_flv_module --with-http_mp4_module
make && make install
cd ..
#提供nginx启动脚本
cat > /etc/init.d/nginx << EOF
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemin
#
# chkconfig: - 85 15
# description: Nginx is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /usr/local/nginx/conf/nginx.conf
# pidfile: /usr/local/nginx/logs/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "\$NETWORKING" = "no" ] && exit 0
nginx="/usr/local/nginx/sbin/nginx"
prog=\$(basename \$nginx)
NGINX_CONF_FILE="/usr/local/nginx/conf/nginx.conf"
lockfile=/var/lock/subsys/nginx
start() {
[ -x \$nginx ] || exit 5
[ -f \$NGINX_CONF_FILE ] || exit 6
echo -n \$"Starting \$prog: "
daemon \$nginx -c \$NGINX_CONF_FILE
retval=\$?
echo
[ \$retval -eq 0 ] && touch \$lockfile
return \$retval
}
stop() {
echo -n \$"Stopping \$prog: "
killproc \$prog -QUIT
retval=\$?
echo
[ \$retval -eq 0 ] && rm -f \$lockfile
return \$retval
}
restart() {
configtest || return \$?
stop
start
}
reload() {
configtest || return \$?
echo -n \$"Reloading \$prog: "
killproc \$nginx -HUP
RETVAL=\$?
echo
}
force_reload() {
restart
}
configtest() {
\$nginx -t -c \$NGINX_CONF_FILE
}
rh_status() {
status \$prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "\$1" in
start)
rh_status_q && exit 0
\$1
;;
stop)
rh_status_q || exit 0
\$1
;;
restart|configtest)
\$1
;;
reload)
rh_status_q || exit 7
\$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo \$"Usage: \$0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac
EOF
##赋予nginx执行的权限
chmod +x /etc/init.d/nginx
chkconfig --add nginx
#创建文件
mkdir -p /var/tmp/nginx/proxy
#启动nginx服务
/etc/init.d/nginx restart
#查看nginx状态
/bin/ps -ef |grep nginx2、分别在两台机器上创建不同的测试页面[为了测试]
#在master机器创建index.html页面 echo "Welcome To 192.168.0.189" > /usr/local/nginx/html/index.html #在backup机器创建index.html页面 echo "Welcome To 192.168.0.200" > /usr/local/nginx/html/index.html
3、安装keepalived
#至于keepalived的安装可以选择源码编译安装或者yum源安装，都可以；这里选择源码编译安装
#安装keepalived源码安装的依赖包 yum -y install openssl openssl-devel yum -y install popt-devel #源码编译安装keepalived tar xf keepalived-1.2.7.tar.gz cd keepalived-1.2.7 ./configure --prefix=/usr/local/keepalived make && make install #拷贝keepavlied的一些文件到相关目录中 mkdir /etc/keepalived cp /usr/local/keepalived/sbin/keepalived /usr/sbin/ cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/ cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/ cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
修改keepalived配置文件
keepalived的文件路径/etc/keepalived/keepalived.conf
#####【Master-keepalived】#####
! Configuration File for keepalived #全局定义
global_defs {
notification_email { #指定keepalived在发生事件时(比如切换),需要发送的email对象,可以多个,每行一个
zhengyansheng@cdvcloud.com
}
notification_email_from admin@cdvcloud.com
smtp_server 127.0.0.1 #指定发送email的smtp服务器
smtp_connect_timeout 30
router_id LVS_DEVEL #运行keepalived的机器的一个标识
}
vrrp_instance VI_1 {
state MASTER #为主服务器
interface eth1 #监听的本地网卡接口
virtual_router_id 51 #主辅virtual_router_id号必须相同
mcast_src_ip=192.168.0.189 #主nginx的ip地址
priority 100 #优先级为100，此值越大优先级越大 就为master 权重值
advert_int 1 #VRRP Multicast 广播周期秒数；心跳检测时间，单位秒
authentication {
auth_type PASS #vrrp认证方式
auth_pass 1111 #vrrp口令
}
virtual_ipaddress { #VRRP HA 虚拟地址 如果有多个VIP，继续换行填写
192.168.0.250/24 dev eth1
}
}#####【Backup-keepalived】#####
! Configuration File for keepalived
global_defs {
notification_email {
zhengyansheng@cdvcloud.com
}
notification_email_from admin@cdvcloud.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 51
mcast_src_ip=192.168.0.200
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.250/24 dev eth1
}
}启动keepalived服务[所有的服务开机启动]
/etc/init.d/keepalived start
chkconfig nginx on
chkconfig keepalived on
查看vip状态
#首先在master节点上查看vip的状态
[root@master ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:59:45:de brd ff:ff:ff:ff:ff:ff inet 192.168.0.189/24 brd 192.168.0.255 scope global eth1 inet 192.168.0.250/32 scope global eth1 #vip地址已经加入到master节点上 inet6 fe80::20c:29ff:fe59:45de/64 scope link valid_lft forever preferred_lft forever
#其次在backup节点上查看vip的状态
[root@backup ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:61:70:d3 brd ff:ff:ff:ff:ff:ff inet 192.168.0.200/24 brd 192.168.0.255 scope global eth1 inet6 fe80::20c:29ff:fe61:70d3/64 scope link valid_lft forever preferred_lft forever
学会看日志输出
一部分：当启动keepalived服务的时候，会根据配置文件的优先级来竞选谁为master，从日志来看192.168.0.189竞选master
[root@master ~]# /etc/init.d/keepalived start 正在启动 keepalived： [确定] [root@master ~]# tail -f /var/log/messages Aug 16 11:09:48 localhost Keepalived[25707]: Starting Keepalived v1.2.7 (08/16,2014) Aug 16 11:09:48 localhost Keepalived[25708]: Starting Healthcheck child process, pid=25710 Aug 16 11:09:48 localhost Keepalived[25708]: Starting VRRP child process, pid=25711 Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Interface queue is empty Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Netlink reflector reports IP 192.168.0.189 added Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Netlink reflector reports IP fe80::20c:29ff:fe59:45de added Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Registering Kernel netlink reflector Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Registering Kernel netlink command channel Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Registering gratuitous ARP shared channel Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Interface queue is empty Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Netlink reflector reports IP 192.168.0.189 added Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Netlink reflector reports IP fe80::20c:29ff:fe59:45de added Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Registering Kernel netlink reflector Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Registering Kernel netlink command channel Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Opening file ‘/etc/keepalived/keepalived.conf‘. Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Configuration is using : 62766 Bytes Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: Using LinkWatch kernel netlink reflector... Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Opening file ‘/etc/keepalived/keepalived.conf‘. Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Configuration is using : 7073 Bytes Aug 16 11:09:48 localhost Keepalived_vrrp[25711]: VRRP sockpool: [ifindex(2), proto(112), fd(11,12)] Aug 16 11:09:48 localhost Keepalived_healthcheckers[25710]: Using LinkWatch kernel netlink reflector... Aug 16 11:09:51 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) Transition to MASTER STATE Aug 16 11:09:56 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) Entering MASTER STATE Aug 16 11:09:56 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) setting protocol VIPs. Aug 16 11:09:56 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.0.250 Aug 16 11:09:56 localhost Keepalived_healthcheckers[25710]: Netlink reflector reports IP 192.168.0.250 added Aug 16 11:10:01 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.0.250 [root@backup ~]# tail -f /var/log/messages Aug 16 11:09:51 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) Received higher prio advert Aug 16 11:09:51 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) Entering BACKUP STATE Aug 16 11:09:51 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) removing protocol VIPs. Aug 16 11:09:51 localhost Keepalived_healthcheckers[26135]: Netlink reflector reports IP 192.168.0.250 removed [root@master ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:59:45:de brd ff:ff:ff:ff:ff:ff inet 192.168.0.189/24 brd 192.168.0.255 scope global eth1 inet 192.168.0.250/32 scope global eth1 inet6 fe80::20c:29ff:fe59:45de/64 scope link valid_lft forever preferred_lft forever 二部分：当我们停掉主keepalived服务的时候，主master的vip会自动切换到辅机器上，看日志 停止主keepalived服务 [root@master ~]# /etc/init.d/keepalived stop 停止 keepalived： [确定] [root@master ~]# tail -f /var/log/messages Aug 16 11:16:14 localhost Keepalived[25708]: Stopping Keepalived v1.2.7 (08/16,2014) Aug 16 11:16:14 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) sending 0 priority Aug 16 11:16:14 localhost Keepalived_vrrp[25711]: VRRP_Instance(VI_1) removing protocol VIPs. [root@backup ~]# tail -f /var/log/messages Aug 16 11:09:51 localhost Keepalived_healthcheckers[26135]: Netlink reflector reports IP 192.168.0.250 removed Aug 16 11:16:14 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) Transition to MASTER STATE Aug 16 11:16:19 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) Entering MASTER STATE Aug 16 11:16:19 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) setting protocol VIPs. Aug 16 11:16:19 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.0.250 Aug 16 11:16:19 localhost Keepalived_healthcheckers[26135]: Netlink reflector reports IP 192.168.0.250 added Aug 16 11:16:24 localhost Keepalived_vrrp[26136]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth1 for 192.168.0.250 [root@backup ~]# ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000 link/ether 00:0c:29:61:70:d3 brd ff:ff:ff:ff:ff:ff inet 192.168.0.200/24 brd 192.168.0.255 scope global eth1 inet 192.168.0.250/32 scope global eth1 辅节点已经成功接管vip inet6 fe80::20c:29ff:fe61:70d3/64 scope link valid_lft forever preferred_lft forever
测试1：nginx主备切换的时间 C:\Users\Allentuns>ping 192.168.0.250 -t 正在 Ping 192.168.0.250 具有 32 字节的数据: 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 请求超时。 请求超时。 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 来自 192.168.0.250 的回复: 字节=32 时间<1ms TTL=64 192.168.0.250 的 Ping 统计信息: 数据包: 已发送 = 28，已接收 = 26，丢失 = 2 (7% 丢失)， 往返行程的估计时间(以毫秒为单位): 最短 = 0ms，最长 = 0ms，平均 = 0ms Control-C ^C C:\Users\Allentuns>
以上keepalived的配置文件并不能满足整个架构：
因为默认情况下，keepalived工作模式并不能直接监控nginx服务，只有当keepalived服务挂掉后才能主备切换，nginx服务挂掉后不能实现主备服务器的切换，但是我们的目的就是要实现nginx服务keepalived挂掉后，都要主备切换。
以上有两种方法可以实现
keepalived配置文件中可以支持shell脚本，写个监听nginx服务的脚本就可以了
单独写个脚本来监听nginx和keepalived服务
基于第一种情况
#####【Master-keepalived】#####
! Configuration File for keepalived #全局定义
global_defs {
notification_email { #指定keepalived在发生事件时(比如切换),需要发送的email对象,可以多个,每行一个
zhengyansheng@cdvcloud.com
}
notification_email_from admin@cdvcloud.com
smtp_server 127.0.0.1 #指定发送email的smtp服务器
smtp_connect_timeout 30
router_id LVS_DEVEL #运行keepalived的机器的一个标识
}
vrrp_script
chk_nginx { #检测nginx服务是否在运行有很多方式，比如进程，用脚本检测等等
script "killall -0 nginx" #用shell命令检查nginx服务是否存在
interval 1 #时间间隔为1秒检测一次
weight -2 #当nginx的服务不存在了，就把当前的权重-2
fall 2 #测试失败的次数
rise 1 #测试成功的次数
}
vrrp_instance VI_1 {
state MASTER #为主服务器
interface eth1 #监听的本地网卡接口
virtual_router_id 51 #主辅virtual_router_id号必须相同
mcast_src_ip=192.168.0.189 #主nginx的ip地址
priority 100 #优先级为100，此值越大优先级越大 就为master 权重值
advert_int 1 #VRRP Multicast 广播周期秒数；心跳检测时间，单位秒
authentication {
auth_type PASS #vrrp认证方式
auth_pass 1111 #vrrp口令
}
virtual_ipaddress { #VRRP HA 虚拟地址 如果有多个VIP，继续换行填写
192.168.0.250/24 dev eth1
}
track_script {
chk_nginx #引用上面的vrrp_script定义的脚本名称
}
}#####【Backup-keepalived】#####
! Configuration File for keepalived
global_defs {
notification_email {
zhengyansheng@cdvcloud.com
}
notification_email_from admin@cdvcloud.com
smtp_server 127.0.0.1
smtp_connect_timeout 30
router_id LVS_DEVEL
}
vrrp_script
chk_nginx { #检测nginx服务是否在运行有很多方式，比如进程，用脚本检测等等
script "killall -0 nginx" #用shell命令检查nginx服务是否存在
interval 1 #时间间隔为1秒检测一次
weight -2 #当nginx的服务不存在了，就把当前的权重-2
fall 2 #测试失败的次数
rise 1 #测试成功的次数
}
vrrp_instance VI_1 {
state BACKUP
interface eth1
virtual_router_id 51
mcast_src_ip=192.168.0.200
priority 99
advert_int 1
authentication {
auth_type PASS
auth_pass 1111
}
virtual_ipaddress {
192.168.0.250/24 dev eth1
}
track_script {
chk_nginx #引用上面的vrrp_script定义的脚本名称
}
}
#这种情况的监听脚本是2秒中监听一次nginx服务，如果进程不存在，则将keepalived的权重减少2，为98，那么权重为99的为主 自动切换到备用服务器上基于第二种情况
#!/bin/bash while : do nginxpid=`ps -C nginx --no-header | wc -l` if [ $nginxpid -eq 0 ];then /usr/local/nginx/sbin/nginx sleep 5 nginxpid=`ps -C nginx --no-header | wc -l` echo $nginxpid if [ $nginxpid -eq 0 ];then /etc/init.d/keepalived stop fi fi sleep 5 done #这个脚本确实很好用，来自酒哥的博文
至于后端tomcat的负载均衡和主备切换在这里就不多说了！很简单。。。。 网上有很多这方面的教程
我会在下周末之前把nginx的双主架构和hadoop的源码编译贡献给博客上。
博文的笔记和所有的软件都已经上传！期待和大家交流更多linux运维 QQ 467754239
http://yunpan.cn/QaSn4mzCd65YP 访问密码 06b3
本文出自 “郑彦生” 博客，请务必保留此出处http://467754239.blog.51cto.com/4878013/1541421
nginx+keepalived构建主备负载均衡代理服务器,布布扣,bubuko.com
原文：http://467754239.blog.51cto.com/4878013/1541421