nginx+keepalived+tomcat配置高可用web集群

基本架构：

    

角色	ip	安装软件	作用	主机名
nginx主	192.168.247.129	nginx+keepalived	反向代理
nginx备	192.168.247.130	nginx+keepalived	反向代理
tomcat1	192.168.247.128	tomcat	web服务器
tomcat2	192.168.247.131	tomcat	web服务器
nfs主	192.168.247.132	nfs	低端共享存储
nfs备	192.168.247.133	nfs	低端共享存储
虚拟ip	192.168.247.150

集群部署：

    tomcat安装部署：

        编译安装tomcat：（tomcat1，tomcat2）

            安装JDK，运行java环境：（tomcat1，tomcat2）

                卸载centos自带的JDK：

rpm -qa | grep javayum -y remove java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5

                下载安装JDK：

##下载地址：http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html ##下载最新的（64位）：jdk-7u79-linux-x64.rpm

##在/usr下建立一个java目录，以备将java程序安装在此目录下mkdir /usr/javacd /usr/local/srcrpm -ivh jdk-7u79-linux-x64.rpm        ##安装后，在/usr/java目录下就会生成一个新的目录jdk1.7.0_79，该目录下存放的是安装后的文件

##设置环境变量：vi /etc/profile  ##在最后添加：export JAVA_HOME=/usr/java/jdk1.7.0_79export PATH=$PATH:$JAVA_HOME/binexport CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jarsource /etc/profile

##验证是否安装成功：java -version         ##出现如下则正常java version "1.7.0_79"Java(TM) SE Runtime Environment (build 1.7.0_79-b15)Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

            安装tomcat7：（tomcat1，tomcat2）

软件：apache-tomcat-7.0.29.tar.gzcd /usr/local/src/tar -zxf apache-tomcat-7.0.42.tar.gzcp -a apache-tomcat-7.0.42 /usr/local/tomcat7/

/usr/local/tomcat7/bin/startup.sh       ##启动tomcat，出现如下则正常Using CATALINA_BASE:   /usr/local/tomcat7Using CATALINA_HOME:   /usr/local/tomcat7Using CATALINA_TMPDIR: /usr/local/tomcat7/tempUsing JRE_HOME:        /usr/java/jdk1.7.0_79Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar##设置开机自动启动

##访问下：    ip:8080   看是否正常

        配置tomcat：（tomcat1，tomcat2）

            配置server.xml文件：

cd /usr/local/tomcat7/conf/vi server.xml##找到：    
     ##将其注释掉##添加如下（已优化）：    
     ##配置优化：             优化JVM堆内存：                vi /usr/local/tomcat7/bin/catalina.sh                添加：JAVA_OPTS='-server -Xms1024m -Xmx2048m -XX:PermSize=256M -XX:MaxNewSize=256m -XX:MaxPermSize=256m'			    修改日志目录：/data/tomcat7/logs	        vi /usr/local/tomcat7/conf/server.xml			
                 #nginx做反向代理，%{X-Real-IP}i 获取用户的真实ip                                                  ##找到：Valve标签                    ##将pattern参数的值修改为："%{X-Real-IP}i %l %u %t "%r" %s %b"        ##{X-Real-IP}i，获取的是真实客户端ip，不是代理机ip                    ##每个参数的含义：http://twb.iteye.com/blog/182100                    ##nginx前端代理tomcat取真实客户端IP：http://www.cnphp6.com/archives/60908

##重新启动tomcat：[root@tomcat1 ~]# /usr/local/tomcat7/bin/shutdown.sh Using CATALINA_BASE:   /usr/local/tomcat7Using CATALINA_HOME:   /usr/local/tomcat7Using CATALINA_TMPDIR: /usr/local/tomcat7/tempUsing JRE_HOME:        /usr/java/jdk1.7.0_79Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar[root@tomcat1 ~]# /usr/local/tomcat7/bin/startup.sh Using CATALINA_BASE:   /usr/local/tomcat7Using CATALINA_HOME:   /usr/local/tomcat7Using CATALINA_TMPDIR: /usr/local/tomcat7/tempUsing JRE_HOME:        /usr/java/jdk1.7.0_79Using CLASSPATH:       /usr/local/tomcat7/bin/bootstrap.jar:/usr/local/tomcat7/bin/tomcat-juli.jar

##tomcat性能优化：参考：http://www.cnblogs.com/ggjucheng/archive/2013/04/16/3024731.html

    反向代理的部署：

        编译安装nginx：（主，备）

            安装nginx的依赖包：

yum -y install gcc gcc-c++ autoconf automakeyum -y install zlib zlib-devel openssl openssl-devel pcre-devel

            编译安装pcre，使nginx支持rewrite重写：

cd /usr/local/src/wget http://sourceforge.net/projects/pcre/files/pcre/8.35/pcre-8.35.tar.gztar -zxf pcre-8.35.tar.gzcd pcre-8.35./configuremakemake install

            安装nginx：

cd /usr/local/src/wget http://nginx.org/download/nginx-1.6.2.tar.gztar -zxf nginx-1.6.2.tar.gz cd nginx-1.6.2./configure  --prefix=/usr/local/nginx --with-http_stub_status_module --with-http_ssl_module --with-http_gzip_static_module --with-http_flv_module --with-http_gunzip_module --with-http_realip_modulemakemake install###注意--with-http_ssl_module，后面要用ssl

        配置nginx：（主，备）

            创建nginx用户：

useradd -s /sbin/nologin nginx

            修改文件打开数：

ulimit -n 65535##设置开机自动启动

            生成ssl证书（模拟）（忽略）

##创建证书存放目录：cd /usr/local/nginx/mkdir cacd ca##创建服务器私钥：openssl genrsa -des3 -out server.key 1024##创建签名请求的证书（CSR）：openssl req -new -key server.key -out server.csr##在加载SSL支持的Nginx并使用上述私钥时除去必须的口令cp server.key server.key.orgopenssl rsa -in server.key.org -out server.key##最后标记证书使用上述私钥和CSR：openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

         生成ssl证书（模拟）

##创建证书存放目录：mkdir /usr/local/nginx/cacd /usr/local/nginx/ca##生成密钥文件key和csr文件：openssl req -new -newkey rsa:2048 -nodes -keyout server.key -out server.csr##生成服务器认证文件crt：openssl x509 -req -days 365 -in server.csr -signkey server.key -out server.crt

           修改配置文件：

# mv /usr/local/nginx/conf/nginx.conf /usr/local/nginx/conf/nginx.conf.old# vi /usr/local/nginx/conf/nginx.confuser nginx nginx;worker_processes  2;error_log  logs/error.log;pid        logs/nginx.pid;worker_rlimit_nofile 65535;events {    use epoll;    worker_connections  65535;}http {    include       mime.types;    default_type  application/octet-stream;    log_format main '$remote_addr - $remote_user [$time_local] "$request" '                        '$status $body_bytes_sent "$http_referer" '                        '"$http_user_agent" $http_x_forwarded_for';        server_names_hash_bucket_size 128;    client_header_buffer_size 32k;    large_client_header_buffers 4 32k;    sendfile       on;    #tcp_nopush     on;    tcp_nodelay    on;    client_header_timeout 30;    client_body_timeout 30;    send_timeout   30;    client_max_body_size 100M;    keepalive_timeout  60;    proxy_connect_timeout  100;    proxy_send_timeout  100;    proxy_read_timeout  100;    proxy_buffer_size  16k;    proxy_buffers  4 32k;    proxy_busy_buffers_size  64k;        gzip  on;    gzip_min_length 1k;    gzip_buffers 4 16k;    gzip_http_version 1.1;    gzip_comp_level 2;    gzip_types text/plain application/x-javascript text/css application/xml text/javascript;    gzip_vary on;    upstream web1{      ip_hash;      server 192.168.247.128:8080 max_fails=3 fail_timeout=30s;      server 192.168.247.131:8080 max_fails=3 fail_timeout=30s;    }    server {        listen       443;        server_name  www.scj.com;        root /opt/nginx/www/www.scj.com;        index index.html index.htm index.jsp;        ssl on;        ssl_certificate /usr/local/nginx/ca/server.crt;        ssl_certificate_key /usr/local/nginx/ca/server.key;        access_log  logs/www.scj.com/access.log main;        error_page   500 502 503 504  /50x.html;        location = /50x.html {            root   html;        }        location ~ .*\.jsp$ {          proxy_pass http://web1;          proxy_set_header Host $host;          proxy_set_header X-Real-IP $remote_addr;          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;          proxy_next_upstream http_502 http_504 error timeout invalid_header;        }        location ~ .*\.(gif|jpg|png|swf|bmp|jpeg)$ {            expires 10d;        }        location ~ .*\.(js|css)?$ {            expires 1h;        }    }    #server {       #listen 80 default;       #server_name _;       #return 500;    #}}

##创建日志目录和根目录mkdir /usr/local/nginx/logs/www.scj.com        ##日志目录mkdir -p /opt/nginx/www/www.scj.com            ##创建根目录

##启动nginx：# /usr/local/nginx/sbin/nginx# ps -ef | grep nginxroot       7890      1  0 08:00 ?        00:00:00 nginx: master process /usr/local/nginx/sbin/nginxnginx      7891   7890  0 08:00 ?        00:00:00 nginx: worker process      nginx      7892   7890  0 08:00 ?        00:00:00 nginx: worker process      root       7896   7798  0 08:00 pts/0    00:00:00 grep nginx# netstat  -tlnpa | grep nginxtcp        0      0 0.0.0.0:443                 0.0.0.0:*                   LISTEN      7890/nginx##设置开机自动启动

    部署keepalived，实现自动切换：

        安装keepalived：（主，备）

cd /usr/local/src/wget http://www.keepalived.org/software/keepalived-1.2.15.tar.gztar -zxf keepalived-1.2.15.tar.gzcd keepalived-1.2.15./configure --prefix=/usr/local/keepalivedmakemake install

        拷贝文件：（主，备）

cp -a /usr/local/keepalived/etc/rc.d/init.d/keepalived  /etc/init.d/cp -a /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/mkdir /etc/keepalived/cp -a /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/cp -a /usr/local/keepalived/sbin/keepalived /usr/sbin/##注意： /etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 的路径一定要正确，##因为在执行/etc/init.d/keepalived这个启动脚本时，会读取/etc/sysconfig/keepalived 和 /etc/keepalived/keepalived.conf 这两个文件

        修改配置文件：

            nginx主：

[root@nginxmaster ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old[root@nginxmaster ~]# vi /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {    notification_email {      732233048@qq.com    }    notification_email_from root@localhost    smtp_server 127.0.0.1    smtp_connect_timeout 30    router_id nginx_ha}vrrp_instance VI_1 {    state MASTER    interface eth0    virtual_router_id 51    priority 150    advert_int 1    #nopreempt    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        192.168.247.150    }}virtual_server 192.168.247.150 443 {    delay_loop 6    #lb_algo wrr    #lb_kind DR    #persistence_timeout 50    protocol TCP    real_server 192.168.247.129 443 {        #weight 3        notify_down /etc/keepalived/killkeepalived.sh        TCP_CHECK {            connect_timeout 10            nb_get_retry 3            delay_before_retry 3            connect_port 443        }    }}

           nginx备：

[root@nginxslave ~]# mv /etc/keepalived/keepalived.conf /etc/keepalived/keepalived.conf.old[root@nginxslave ~]# vi /etc/keepalived/keepalived.conf! Configuration File for keepalivedglobal_defs {    notification_email {      732233048@qq.com    }    notification_email_from root@localhost    smtp_server 127.0.0.1    smtp_connect_timeout 30    router_id nginx_ha}vrrp_instance VI_1 {    state BACKUP    interface eth0    virtual_router_id 51    priority 100    advert_int 1    #nopreempt    authentication {        auth_type PASS        auth_pass 1111    }    virtual_ipaddress {        192.168.247.150    }}virtual_server 192.168.247.150 443 {    delay_loop 6    #lb_algo wrr    #lb_kind DR    #persistence_timeout 50    protocol TCP    real_server 192.168.247.130 443 {        #weight 3        notify_down /etc/keepalived/killkeepalived.sh        TCP_CHECK {            connect_timeout 10            nb_get_retry 3            delay_before_retry 3            connect_port 443        }    }}

            创建/etc/keepalived/killkeepalived.sh脚本：（主，备）

vi /etc/keepalived/killkeepalived.sh#!/bin/bash# check nginx server statusnginx_status=`netstat  -tlnpa | grep 443 | wc -l`if [ $nginx_status -eq 0 ];then  /usr/local/nginx/sbin/nginx  sleep 1  nginx_status=`netstat  -tlnpa | grep 443 | wc -l`  if [ $nginx_status -eq 0 ];then    /etc/init.d/keepalived stop  fifichmod 755 /etc/keepalived/killkeepalived.sh

        修改keepalived的日志文件：（主，备）

            参考：

            说明：

##centos6.3之后的syslog改名叫rsyslog了，默认在 /etc/rsyslog.conf

            修改/etc/sysconfig/keepalived：

vi /etc/sysconfig/keepalived# Options for keepalived. See `keepalived --help' output and keepalived(8) and# keepalived.conf(5) man pages for a list of all options. Here are the most# common ones :## --vrrp               -P    Only run with VRRP subsystem.# --check              -C    Only run with Health-checker subsystem.# --dont-release-vrrp  -V    Dont remove VRRP VIPs & VROUTEs on daemon stop.# --dont-release-ipvs  -I    Dont remove IPVS topology on daemon stop.# --dump-conf          -d    Dump the configuration data.# --log-detail         -D    Detailed log messages.# --log-facility       -S    0-7 Set local syslog facility (default=LOG_DAEMON)##KEEPALIVED_OPTIONS="-D"KEEPALIVED_OPTIONS="-D -d -S 0"               ##在最后添加此行

            修改/etc/rsyslog.conf:

vi /etc/rsyslog.conf##在最后添加此行：local0.*                                                /var/log/keepalived.log## /etc/init.d/rsyslog restartShutting down system logger:                               [  OK  ]Starting system logger:                                    [  OK  ]

        启动keepalived：

            nginx主：

[root@nginxmaster ~]# /etc/init.d/keepalived startStarting keepalived:                                       [  OK  ]

##查看日志：Sep 23 20:16:27 nginxmaster Keepalived[2909]: Starting Keepalived v1.2.15 (09/23,2015)Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting Healthcheck child process, pid=2912Sep 23 20:16:27 nginxmaster Keepalived[2910]: Starting VRRP child process, pid=2913Sep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP 192.168.247.129 addedSep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 addedSep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink reflectorSep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering Kernel netlink command channelSep 23 20:16:27 nginxmaster Keepalived_vrrp[2913]: Registering gratuitous ARP shared channelSep 23 20:16:27 nginxmaster kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)Sep 23 20:16:27 nginxmaster kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)Sep 23 20:16:27 nginxmaster kernel: IPVS: ipvs loaded.Sep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.129 addedSep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP fe80::20c:29ff:fee0:d236 addedSep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink reflectorSep 23 20:16:27 nginxmaster Keepalived_healthcheckers[2912]: Registering Kernel netlink command channelSep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Opening file '/etc/keepalived/keepalived.conf'.Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Configuration is using : 63272 BytesSep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: Using LinkWatch kernel netlink reflector...Sep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering BACKUP STATESep 23 20:16:28 nginxmaster Keepalived_vrrp[2913]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Opening file '/etc/keepalived/keepalived.conf'.Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Configuration is using : 11705 BytesSep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: Scheduler or persistence engine not foundSep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: IPVS: No such processSep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Using LinkWatch kernel netlink reflector...Sep 23 20:16:28 nginxmaster Keepalived_healthcheckers[2912]: Activating healthchecker for service [192.168.247.129]:443Sep 23 20:16:28 nginxmaster kernel: IPVS: Scheduler module ip_vs_ not foundSep 23 20:16:31 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Transition to MASTER STATESep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Entering MASTER STATESep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) setting protocol VIPs.Sep 23 20:16:32 nginxmaster Keepalived_healthcheckers[2912]: Netlink reflector reports IP 192.168.247.150 addedSep 23 20:16:32 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150Sep 23 20:16:37 nginxmaster Keepalived_vrrp[2913]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150##设置开机自动启动chkconfig keepalived on

            nginx备：

[root@nginxslave ~]# /etc/init.d/keepalived startStarting keepalived:                                       [  OK  ]

##查看日志：Sep 24 09:43:29 nginxslave Keepalived[9481]: Starting Keepalived v1.2.15 (09/24,2015)Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting Healthcheck child process, pid=9484Sep 24 09:43:29 nginxslave Keepalived[9482]: Starting VRRP child process, pid=9485Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP 192.168.247.130 addedSep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab addedSep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink reflectorSep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering Kernel netlink command channelSep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Registering gratuitous ARP shared channelSep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Opening file '/etc/keepalived/keepalived.conf'.Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Configuration is using : 63262 BytesSep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: Using LinkWatch kernel netlink reflector...Sep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP_Instance(VI_1) Entering BACKUP STATESep 24 09:43:29 nginxslave Keepalived_vrrp[9485]: VRRP sockpool: [ifindex(2), proto(112), unicast(0), fd(10,11)]Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP 192.168.247.130 addedSep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Netlink reflector reports IP fe80::20c:29ff:fe1e:8eab addedSep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink reflectorSep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Registering Kernel netlink command channelSep 24 09:43:29 nginxslave kernel: IPVS: Registered protocols (TCP, UDP, SCTP, AH, ESP)Sep 24 09:43:29 nginxslave kernel: IPVS: Connection hash table configured (size=4096, memory=64Kbytes)Sep 24 09:43:29 nginxslave kernel: IPVS: ipvs loaded.Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Opening file '/etc/keepalived/keepalived.conf'.Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Configuration is using : 11695 BytesSep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: Scheduler or persistence engine not foundSep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: IPVS: No such processSep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Using LinkWatch kernel netlink reflector...Sep 24 09:43:29 nginxslave Keepalived_healthcheckers[9484]: Activating healthchecker for service [192.168.247.130]:443Sep 24 09:43:29 nginxslave kernel: IPVS: Scheduler module ip_vs_ not found##设置开机自动启动chkconfig keepalived on

        查看vip绑定在哪台机器上：

[root@nginxmaster ~]#ip addr1: lo: 
     
       mtu 16436 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever2: eth0: 
      
        mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 00:0c:29:e0:d2:36 brd ff:ff:ff:ff:ff:ff    inet 192.168.247.129/24 brd 192.168.247.255 scope global eth0    inet 192.168.247.150/32 scope global eth0    inet6 fe80::20c:29ff:fee0:d236/64 scope link        valid_lft forever preferred_lft forever
      
     

[root@nginxslave ~]#ip addr1: lo: 
     
       mtu 16436 qdisc noqueue state UNKNOWN     link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00    inet 127.0.0.1/8 scope host lo    inet6 ::1/128 scope host        valid_lft forever preferred_lft forever2: eth0: 
      
        mtu 1500 qdisc pfifo_fast state UP qlen 1000    link/ether 00:0c:29:1e:8e:ab brd ff:ff:ff:ff:ff:ff    inet 192.168.247.130/24 brd 192.168.247.255 scope global eth0    inet6 fe80::20c:29ff:fe1e:8eab/64 scope link        valid_lft forever preferred_lft forever
      
     

##由上发现vip150绑定在nginxmaster，正常

测试：

    测试一：把nginx主的nginx服务stop掉

[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop

##查看日志：Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 failed !!!Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not definedSep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected.Sep 23 23:38:00 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent.Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: TCP connection to [192.168.247.129]:443 success.Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Adding service [192.168.247.129]:443 to VS [192.168.247.150]:443Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not definedSep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: Remote SMTP server [127.0.0.1]:25 connected.Sep 23 23:38:06 nginxmaster Keepalived_healthcheckers[3990]: SMTP alert successfully sent##keepalived会先把129移除，然后执行/etc/keepalived/killkeepalived.sh脚本，nginx服务正常启动，又把129加入到集群中

    测试二：把nginx主的keepalived服务stop掉

[root@nginxmaster ~]# /etc/init.d/keepalived stopStopping keepalived:                                       [  OK  ]

##查看nginx主的日志：Sep 23 23:41:05 nginxmaster Keepalived[3988]: Stopping Keepalived v1.2.15 (09/23,2015)Sep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) sending 0 prioritySep 23 23:41:05 nginxmaster Keepalived_vrrp[3991]: VRRP_Instance(VI_1) removing protocol VIPs.Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Netlink reflector reports IP 192.168.247.150 removedSep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443Sep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: Service not definedSep 23 23:41:05 nginxmaster Keepalived_healthcheckers[3990]: IPVS: No such service

##查看nginx备日志：Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: TCP connection to [192.168.247.130]:443 success.Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Adding service [192.168.247.130]:443 to VS [192.168.247.150]:443Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: IPVS: Service not definedSep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Gained quorum 1+0=1 <= 1 for VS [192.168.247.150]:443Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: Remote SMTP server [127.0.0.1]:25 connected.Sep 24 12:57:11 nginxslave Keepalived_healthcheckers[10012]: SMTP alert successfully sent.Sep 24 12:58:05 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226)Sep 24 12:58:05 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226)Sep 24 12:58:07 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 705 seconds.Sep 24 13:05:23 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATESep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATESep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs.Sep 24 13:05:24 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 addedSep 24 13:05:24 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150Sep 24 13:05:29 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150##nginx备变成master，绑定vip 150

    测试三：把nginx主的nginx服务stop，并不再让它成功启动

##删掉日志目录，nginx便不能启动了[root@nginxmaster logs]# rm -rf /usr/local/nginx/logs/www.scj.com/

##关闭nginx服务[root@nginxmaster ~]# /usr/local/nginx/sbin/nginx -s stop

##查看nginx主日志：Sep 23 23:44:33 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: TCP connection to [192.168.247.129]:443 failed !!!Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Removing service [192.168.247.129]:443 from VS [192.168.247.150]:443Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: IPVS: Service not definedSep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Executing [/etc/keepalived/killkeepalived.sh] for service [192.168.247.129]:443 in VS [192.168.247.150]:443Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Lost quorum 1-0=1 > 0 for VS [192.168.247.150]:443Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: Remote SMTP server [127.0.0.1]:25 connected.Sep 23 23:44:38 nginxmaster Keepalived_healthcheckers[4078]: SMTP alert successfully sent.Sep 23 23:44:39 nginxmaster Keepalived[4076]: Stopping Keepalived v1.2.15 (09/23,2015)Sep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) sending 0 prioritySep 23 23:44:39 nginxmaster Keepalived_vrrp[4079]: VRRP_Instance(VI_1) removing protocol VIPs.Sep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: Netlink reflector reports IP 192.168.247.150 removedSep 23 23:44:39 nginxmaster Keepalived_healthcheckers[4078]: IPVS: No such service##nginx服务不能启动了，然后把keepalived服务关掉

##查看nginx备日志Sep 24 13:08:57 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Transition to MASTER STATESep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Entering MASTER STATESep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) setting protocol VIPs.Sep 24 13:08:58 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150Sep 24 13:08:58 nginxslave Keepalived_healthcheckers[10012]: Netlink reflector reports IP 192.168.247.150 addedSep 24 13:09:03 nginxslave Keepalived_vrrp[10013]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.247.150Sep 24 13:09:52 nginxslave dhclient[906]: DHCPREQUEST on eth0 to 192.168.247.254 port 67 (xid=0x10a73226)Sep 24 13:09:52 nginxslave dhclient[906]: DHCPACK from 192.168.247.254 (xid=0x10a73226)Sep 24 13:09:54 nginxslave dhclient[906]: bound to 192.168.247.130 -- renewal in 800 seconds.##nginx备变为master，并绑定vip 150

    测试四：把nginx备的nginx服务stop

##查看nginx备的nginx服务会不会又正常启动

    测试五：把nginx备的keepalived服务stop

##nginx主不会有任何变化

    测试六：把nginx备的nginx服务stop掉，并不再让它正常启动

##nginx备的keepalived服务会被关掉##nginx主不会有任何变化

注意：

    修复后，一定要记得把keepalived服务启动起来

nfs部署：