搭建高可用Kubernetes集群之etcd集群搭建篇（一）@慕课网原创_慕课网

赠人一赞手有余香

之前搭建了一个单个Master多个Node的Kubernetes集群，但是这个不满足生产环境的高可用需求，现在准备使用Etcd+HAProxy搭一个多Master多Node的Kubernetes集群。有以下两种方案可选。

方案一

image.png

方案二

image.png

这里我选择方案二，把etcd和负载均衡集群独立出来到各个主机，这样方便以后扩展或者其他服务可以调用的etcd集群。

环境介绍

操作系统：
CentOS7.4
3.10.0-862.el7.x86_64
SMP Fri Apr 20 16:44:24 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux

主机规划

192.168.88.90 etcd1
192.168.88.92 etcd2
192.168.88.93 etcd3
192.168.88.94 lb1
192.168.88.95 lb2
192.168.88.96 lb3
192.168.88.96 master1
192.168.88.97 master2
192.168.88.98 master3
192.168.88.140 node1
192.168.88.141 node2
192.168.88.142 node3
192.168.88.201 VIP，apiserver的地址

软件版本

docker-ce: 18.06.1
kubernetes : 1.12.4
etcd: 3.2.22
haproxy

准备工作（各个节点都要操作）

在各个节点执行如下命令： (我这里没有关闭swap，后面搭建集群时需要忽略swap打开)

关闭防火墙
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/config
setenforce 0
systemctl stop firewalld
systemctl disable firewalld
配置ipvs模块

vim /etc/sysctl.d/k8s.conf
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1

modprobe br_netfilter
sysctl -p /etc/sysctl.d/k8s.conf

vim  /etc/sysconfig/modules/ipvs.modules#!/bin/bashipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"for i in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do
    /sbin/modinfo -F filename $i & > /dev/null    if [ $? -eq 0 ]; then
        /sbin/modprobe $i
    fidone

/etc/sysconfig/modules/ipvs.modules

配置时间同步
以etcd1作为时间服务器的主节点，把原来的时间服务器注释了，添加如下阿里的时间服务器，以及添加一个子网，允许子网内的机器连接我作为时间服务器。
etcd0节点:
vim /etc/chrony.conf

server ntp1.aliyun.com iburst
server ntp2.aliyun.com iburst
server ntp3.aliyun.com iburst
server ntp4.aliyun.com iburst
server ntp5.aliyun.com iburst
server ntp6.aliyun.com iburst
server ntp7.aliyun.com iburst

allow 192.168.88.0/24

其他节点:
把其他server服务器注释了，只需添加如下两条（防止etcd0宕机导致集群时间不精准）

server 192.168.88.90 iburstserver ntp1.aliyun.com iburst

最后重启chrony服务器
systemctl restart chrony

安装cfssl（只在etcd1节点即可）

wget -O /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64
wget -O /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64
wget -O /bin/cfssl-certinfo  https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64for cfssl in `ls /bin/cfssl*`;do chmod +x $cfssl;done;

*配置hosts文件

192.168.88.90 etcd1
192.168.88.92 etcd2
192.168.88.93 etcd3
192.168.88.94 lb1
192.168.88.95 lb2
192.168.88.96 lb3
192.168.88.97 master1
192.168.88.98 master2
192.168.88.99 master3
192.168.88.140 node1
192.168.88.141 node2
192.168.88.142 node3

安裝etcd（etcd1，etcd2，etcd3）

yum install -y etcd

配置证书

vim etcd-csr.json

{                                     
    "CN": "etcd",                     
    "hosts": [                        
              "127.0.0.1",            
              "192.168.88.90",        
              "192.168.88.92",        
              "192.168.88.93"         
              ],                      
     "key": {                         
              "algo": "rsa",          
              "size": 2048            
            },                        
     "names": [                       
                {                     
                "C": "CN",            
                "ST": "hubei",        
                "L": "hubei",         
                "O": "etcd",          
                "OU": "Etcd Security" 
                 }                    
            ]                         
}

vim ca-config.json

{                                                      
        "signing": {                                   
                "default": {                           
                        "expiry": "87600h"             
                        },                             
                "profiles": {                          
                        "kubernetes": {                
                               "usages": [             
                                  "signing",           
                                  "key encipherment",  
                                  "server auth",       
                                  "client auth"        
                          ],                           
                          "expiry": "87600h"           
                        }                              
                }                                      
         }                                                                                          
 }

vim etcd-ca-csr.json

{    "CN": "etcd",    "key": {        "algo": "rsa",        "size": 2048
        },     "names": [
                {                "C": "CN",                "ST": "hubei",                "L": "hubei",                 "O": "etcd",                "OU": "Etcd Security"
                }
        ]
}

生成证书

cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd

mkdir -pv /etc/etcd/ssl
cp etcd*.pem /etc/etcd/ssl

复制证书到其他节点

scp -r /etc/etcd etcd2:/etc/scp -r /etc/etcd etcd3:/etc/

修改etcd各个节点的etcd.conf配置文件
vim /etc/etcd/etcd.conf

etcd1

#[Member]#ETCD_CORS=""ETCD_DATA_DIR="/var/lib/etcd/default.etcd"#ETCD_WAL_DIR=""ETCD_LISTEN_PEER_URLS="https://192.168.88.90:2380"ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.90:2379"#ETCD_MAX_SNAPSHOTS="5"#ETCD_MAX_WALS="5"ETCD_NAME="etcd1"#ETCD_SNAPSHOT_COUNT="100000"#ETCD_HEARTBEAT_INTERVAL="100"#ETCD_ELECTION_TIMEOUT="1000"#ETCD_QUOTA_BACKEND_BYTES="0"#ETCD_MAX_REQUEST_BYTES="1572864"#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"##[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.88.90:2380"ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.90:2379"#ETCD_DISCOVERY=""#ETCD_DISCOVERY_FALLBACK="proxy"#ETCD_DISCOVERY_PROXY=""#ETCD_DISCOVERY_SRV=""ETCD_INITIAL_CLUSTER="etcd1=https://192.168.88.90:2380,etcd2=https://192.168.88.92:2380,etcd3=https://192.168.88.93:2380"ETCD_INITIAL_CLUSTER_TOKEN="bigboss"#ETCD_INITIAL_CLUSTER_STATE="new"#ETCD_STRICT_RECONFIG_CHECK="true"#ETCD_ENABLE_V2="true"##[Proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"##[Security]ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_CLIENT_CERT_AUTH="false"ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_AUTO_TLS="false"ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_PEER_CLIENT_CERT_AUTH="false"ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_PEER_AUTO_TLS="false"##[Logging]#ETCD_DEBUG="false"#ETCD_LOG_PACKAGE_LEVELS=""#ETCD_LOG_OUTPUT="default"##[Unsafe]#ETCD_FORCE_NEW_CLUSTER="false"##[Version]#ETCD_VERSION="false"#ETCD_AUTO_COMPACTION_RETENTION="0"##[Profiling]#ETCD_ENABLE_PPROF="false"#ETCD_METRICS="basic"##[Auth]#ETCD_AUTH_TOKEN="simple"

etcd2

#[Member]#ETCD_CORS=""ETCD_DATA_DIR="/var/lib/etcd/default.etcd"#ETCD_WAL_DIR=""ETCD_LISTEN_PEER_URLS="https://192.168.88.92:2380"ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.92:2379"#ETCD_MAX_SNAPSHOTS="5"#ETCD_MAX_WALS="5"ETCD_NAME="etcd2"#ETCD_SNAPSHOT_COUNT="100000"#ETCD_HEARTBEAT_INTERVAL="100"#ETCD_ELECTION_TIMEOUT="1000"#ETCD_QUOTA_BACKEND_BYTES="0"#ETCD_MAX_REQUEST_BYTES="1572864"#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"##[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.88.92:2380"ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.92:2379"#ETCD_DISCOVERY=""#ETCD_DISCOVERY_FALLBACK="proxy"#ETCD_DISCOVERY_PROXY=""#ETCD_DISCOVERY_SRV=""ETCD_INITIAL_CLUSTER="etcd1=https://192.168.88.90:2380,etcd2=https://192.168.88.92:2380,etcd3=https://192.168.88.93:2380"ETCD_INITIAL_CLUSTER_TOKEN="bigboss"#ETCD_INITIAL_CLUSTER_STATE="new"#ETCD_STRICT_RECONFIG_CHECK="true"#ETCD_ENABLE_V2="true"##[Proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"##[Security]ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_CLIENT_CERT_AUTH="false"ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_AUTO_TLS="false"ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_PEER_CLIENT_CERT_AUTH="false"ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_PEER_AUTO_TLS="false"##[Logging]#ETCD_DEBUG="false"#ETCD_LOG_PACKAGE_LEVELS=""#ETCD_LOG_OUTPUT="default"##[Unsafe]#ETCD_FORCE_NEW_CLUSTER="false"##[Version]#ETCD_VERSION="false"#ETCD_AUTO_COMPACTION_RETENTION="0"##[Profiling]#ETCD_ENABLE_PPROF="false"#ETCD_METRICS="basic"##[Auth]#ETCD_AUTH_TOKEN="simple"

etcd3

#[Member]                                                                                        #ETCD_CORS=""                                                                                    ETCD_DATA_DIR="/var/lib/etcd/default.etcd"                                                       #ETCD_WAL_DIR=""                                                                                 ETCD_LISTEN_PEER_URLS="https://192.168.88.93:2380"                                               ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.93:2379"                     #ETCD_MAX_SNAPSHOTS="5"                                                                          #ETCD_MAX_WALS="5"                                                                               ETCD_NAME="etcd3"                                                                                #ETCD_SNAPSHOT_COUNT="100000"                                                                    #ETCD_HEARTBEAT_INTERVAL="100"                                                                   #ETCD_ELECTION_TIMEOUT="1000"                                                                    #ETCD_QUOTA_BACKEND_BYTES="0"                                                                    #ETCD_MAX_REQUEST_BYTES="1572864"                                                                #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"                                                               #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"                                                           #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"                                                               #                                                                                                #[Clustering]                                                                                    ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.88.93:2380"                                    ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.93:2379"                  #ETCD_DISCOVERY=""                                                                               #ETCD_DISCOVERY_FALLBACK="proxy"                                                                 #ETCD_DISCOVERY_PROXY=""                                                                         #ETCD_DISCOVERY_SRV=""                                                                           ETCD_INITIAL_CLUSTER="etcd1=https://192.168.88.90:2380,etcd2=https://192.168.88.92:2380,etcd3=https://192.168.88.93:2380
ETCD_INITIAL_CLUSTER_TOKEN="bigboss"                                                             
#ETCD_INITIAL_CLUSTER_STATE="new"                                                                
#ETCD_STRICT_RECONFIG_CHECK="true"                                                               
#ETCD_ENABLE_V2="true"                                                                           
#                                                                                                
#[Proxy]                                                                                         
#ETCD_PROXY="off"                                                                                
#ETCD_PROXY_FAILURE_WAIT="5000"                                                                  
#ETCD_PROXY_REFRESH_INTERVAL="30000"                                                             
#ETCD_PROXY_DIAL_TIMEOUT="1000"                                                                  
#ETCD_PROXY_WRITE_TIMEOUT="5000"                                                                 
#ETCD_PROXY_READ_TIMEOUT="0"                                                                     
#                                                                                                
#[Security]                                                                                      
ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"                                                          
ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"                                                       
#ETCD_CLIENT_CERT_AUTH="false"                                                                   
ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"                                                 
#ETCD_AUTO_TLS="false"                                                                           
ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"                                                     
ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"                                                  
#ETCD_PEER_CLIENT_CERT_AUTH="false"                                                              
ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"                                            
#ETCD_PEER_AUTO_TLS="false"                                                                      
#                                                                                                
#[Logging]                                                                                       
#ETCD_DEBUG="false"                                                                              
#ETCD_LOG_PACKAGE_LEVELS=""                                                                      
#ETCD_LOG_OUTPUT="default"                                                                       
#                                                                                                
#[Unsafe]                                                                                        
#ETCD_FORCE_NEW_CLUSTER="false"                                                                  
#                                                                                                
#[Version]                                                                                       
#ETCD_VERSION="false"                                                                            
#ETCD_AUTO_COMPACTION_RETENTION="0"                                                              
#                                                                                                
#[Profiling]                                                                                     
#ETCD_ENABLE_PPROF="false"                                                                       
#ETCD_METRICS="basic"                                                                            
#                                                                                                
#[Auth]                                                                                          
#ETCD_AUTH_TOKEN="simple"

在各个节点运行如下命令启动etcd及开机启动

systemctl start etcd
systemctl enable etcd

查看etcd是否运行正常

etcdctl --endpoints "https://192.168.88.90:2379,https://192.168.88.92:2379,https://192.168.88.93:2379" --ca-file=/etc/etcd/ssl/etcd-ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem cluster-health

输出如下：

member afaf79328d27196 is healthy: got healthy result from https://127.0.0.1:2379member 7637d1b4acd15c56 is healthy: got healthy result from https://127.0.0.1:2379member 874429f21be24f32 is healthy: got healthy result from https://127.0.0.1:2379cluster is healthy

至此，etcd集群搭建完毕。
持续更新搭建与踩坑过程~~~~
喜欢的朋友别忘记点赞，哈哈哈

作者：olaH
链接：https://www.jianshu.com/p/a48c206635f3