赠人一赞 手有余香
之前搭建了一个单个Master多个Node的Kubernetes集群，但是这个不满足生产环境的高可用需求，现在准备使用Etcd+HAProxy搭一个多Master多Node的Kubernetes集群。有以下两种方案可选。
方案一
image.png
方案二
image.png
这里我选择方案二，把etcd和负载均衡集群独立出来到各个主机，这样方便以后扩展或者其他服务可以调用的etcd集群。
环境介绍
操作系统：
CentOS7.4
3.10.0-862.el7.x86_64
SMP Fri Apr 20 16:44:24 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
主机规划
192.168.88.90 etcd1
192.168.88.92 etcd2
192.168.88.93 etcd3
192.168.88.94 lb1
192.168.88.95 lb2
192.168.88.96 lb3
192.168.88.96 master1
192.168.88.97 master2
192.168.88.98 master3
192.168.88.140 node1
192.168.88.141 node2
192.168.88.142 node3
192.168.88.201 VIP，apiserver的地址
软件版本
docker-ce: 18.06.1
kubernetes : 1.12.4
etcd: 3.2.22
haproxy
准备工作（各个节点都要操作）
在各个节点执行如下命令： (我这里没有关闭swap，后面搭建集群时需要忽略swap打开)
关闭防火墙
sed -ri 's#(SELINUX=).*#\1disabled#' /etc/selinux/configsetenforce 0systemctl stop firewalldsystemctl disable firewalld配置ipvs模块
vim /etc/sysctl.d/k8s.conf net.bridge.bridge-nf-call-ip6tables = 1 net.bridge.bridge-nf-call-iptables = 1
modprobe br_netfiltersysctl -p /etc/sysctl.d/k8s.conf
vim /etc/sysconfig/modules/ipvs.modules#!/bin/bashipvs_mods_dir="/usr/lib/modules/$(uname -r)/kernel/net/netfilter/ipvs"for i in $(ls $ipvs_mods_dir | grep -o "^[^.]*");do /sbin/modinfo -F filename $i & > /dev/null if [ $? -eq 0 ]; then /sbin/modprobe $i fidone
/etc/sysconfig/modules/ipvs.modules
配置时间同步
以etcd1作为时间服务器的主节点，把原来的时间服务器注释了 ，添加如下阿里的时间服务器，以及添加一个子网，允许子网内的机器连接我作为时间服务器。
etcd0节点:vim /etc/chrony.conf
server ntp1.aliyun.com iburst server ntp2.aliyun.com iburst server ntp3.aliyun.com iburst server ntp4.aliyun.com iburst server ntp5.aliyun.com iburst server ntp6.aliyun.com iburst server ntp7.aliyun.com iburst allow 192.168.88.0/24
其他节点:
把其他server服务器注释了，只需添加如下两条（防止etcd0宕机导致集群时间不精准）
server 192.168.88.90 iburstserver ntp1.aliyun.com iburst
最后重启chrony服务器systemctl restart chrony
安装cfssl（只在etcd1节点即可）
wget -O /bin/cfssl https://pkg.cfssl.org/R1.2/cfssl_linux-amd64 wget -O /bin/cfssljson https://pkg.cfssl.org/R1.2/cfssljson_linux-amd64 wget -O /bin/cfssl-certinfo https://pkg.cfssl.org/R1.2/cfssl-certinfo_linux-amd64for cfssl in `ls /bin/cfssl*`;do chmod +x $cfssl;done;
*配置hosts文件
192.168.88.90 etcd1
192.168.88.92 etcd2
192.168.88.93 etcd3
192.168.88.94 lb1
192.168.88.95 lb2
192.168.88.96 lb3
192.168.88.97 master1
192.168.88.98 master2
192.168.88.99 master3
192.168.88.140 node1
192.168.88.141 node2
192.168.88.142 node3
安裝etcd（etcd1，etcd2，etcd3）
yum install -y etcd
配置证书
vim etcd-csr.json
{
"CN": "etcd",
"hosts": [
"127.0.0.1",
"192.168.88.90",
"192.168.88.92",
"192.168.88.93"
],
"key": {
"algo": "rsa",
"size": 2048
},
"names": [
{
"C": "CN",
"ST": "hubei",
"L": "hubei",
"O": "etcd",
"OU": "Etcd Security"
}
]
}vim ca-config.json
{
"signing": {
"default": {
"expiry": "87600h"
},
"profiles": {
"kubernetes": {
"usages": [
"signing",
"key encipherment",
"server auth",
"client auth"
],
"expiry": "87600h"
}
}
}
}vim etcd-ca-csr.json
{ "CN": "etcd", "key": { "algo": "rsa", "size": 2048
}, "names": [
{ "C": "CN", "ST": "hubei", "L": "hubei", "O": "etcd", "OU": "Etcd Security"
}
]
}生成证书
cfssl gencert -ca=etcd-ca.pem -ca-key=etcd-ca-key.pem -config=ca-config.json -profile=kubernetes etcd-csr.json | cfssljson -bare etcd mkdir -pv /etc/etcd/ssl cp etcd*.pem /etc/etcd/ssl
复制证书到其他节点
scp -r /etc/etcd etcd2:/etc/scp -r /etc/etcd etcd3:/etc/
修改etcd各个节点的etcd.conf配置文件vim /etc/etcd/etcd.conf
etcd1
#[Member]#ETCD_CORS=""ETCD_DATA_DIR="/var/lib/etcd/default.etcd"#ETCD_WAL_DIR=""ETCD_LISTEN_PEER_URLS="https://192.168.88.90:2380"ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.90:2379"#ETCD_MAX_SNAPSHOTS="5"#ETCD_MAX_WALS="5"ETCD_NAME="etcd1"#ETCD_SNAPSHOT_COUNT="100000"#ETCD_HEARTBEAT_INTERVAL="100"#ETCD_ELECTION_TIMEOUT="1000"#ETCD_QUOTA_BACKEND_BYTES="0"#ETCD_MAX_REQUEST_BYTES="1572864"#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"##[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.88.90:2380"ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.90:2379"#ETCD_DISCOVERY=""#ETCD_DISCOVERY_FALLBACK="proxy"#ETCD_DISCOVERY_PROXY=""#ETCD_DISCOVERY_SRV=""ETCD_INITIAL_CLUSTER="etcd1=https://192.168.88.90:2380,etcd2=https://192.168.88.92:2380,etcd3=https://192.168.88.93:2380"ETCD_INITIAL_CLUSTER_TOKEN="bigboss"#ETCD_INITIAL_CLUSTER_STATE="new"#ETCD_STRICT_RECONFIG_CHECK="true"#ETCD_ENABLE_V2="true"##[Proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"##[Security]ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_CLIENT_CERT_AUTH="false"ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_AUTO_TLS="false"ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_PEER_CLIENT_CERT_AUTH="false"ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_PEER_AUTO_TLS="false"##[Logging]#ETCD_DEBUG="false"#ETCD_LOG_PACKAGE_LEVELS=""#ETCD_LOG_OUTPUT="default"##[Unsafe]#ETCD_FORCE_NEW_CLUSTER="false"##[Version]#ETCD_VERSION="false"#ETCD_AUTO_COMPACTION_RETENTION="0"##[Profiling]#ETCD_ENABLE_PPROF="false"#ETCD_METRICS="basic"##[Auth]#ETCD_AUTH_TOKEN="simple"
etcd2
#[Member]#ETCD_CORS=""ETCD_DATA_DIR="/var/lib/etcd/default.etcd"#ETCD_WAL_DIR=""ETCD_LISTEN_PEER_URLS="https://192.168.88.92:2380"ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.92:2379"#ETCD_MAX_SNAPSHOTS="5"#ETCD_MAX_WALS="5"ETCD_NAME="etcd2"#ETCD_SNAPSHOT_COUNT="100000"#ETCD_HEARTBEAT_INTERVAL="100"#ETCD_ELECTION_TIMEOUT="1000"#ETCD_QUOTA_BACKEND_BYTES="0"#ETCD_MAX_REQUEST_BYTES="1572864"#ETCD_GRPC_KEEPALIVE_MIN_TIME="5s"#ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s"#ETCD_GRPC_KEEPALIVE_TIMEOUT="20s"##[Clustering]ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.88.92:2380"ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.92:2379"#ETCD_DISCOVERY=""#ETCD_DISCOVERY_FALLBACK="proxy"#ETCD_DISCOVERY_PROXY=""#ETCD_DISCOVERY_SRV=""ETCD_INITIAL_CLUSTER="etcd1=https://192.168.88.90:2380,etcd2=https://192.168.88.92:2380,etcd3=https://192.168.88.93:2380"ETCD_INITIAL_CLUSTER_TOKEN="bigboss"#ETCD_INITIAL_CLUSTER_STATE="new"#ETCD_STRICT_RECONFIG_CHECK="true"#ETCD_ENABLE_V2="true"##[Proxy]#ETCD_PROXY="off"#ETCD_PROXY_FAILURE_WAIT="5000"#ETCD_PROXY_REFRESH_INTERVAL="30000"#ETCD_PROXY_DIAL_TIMEOUT="1000"#ETCD_PROXY_WRITE_TIMEOUT="5000"#ETCD_PROXY_READ_TIMEOUT="0"##[Security]ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_CLIENT_CERT_AUTH="false"ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_AUTO_TLS="false"ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem"ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem"#ETCD_PEER_CLIENT_CERT_AUTH="false"ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem"#ETCD_PEER_AUTO_TLS="false"##[Logging]#ETCD_DEBUG="false"#ETCD_LOG_PACKAGE_LEVELS=""#ETCD_LOG_OUTPUT="default"##[Unsafe]#ETCD_FORCE_NEW_CLUSTER="false"##[Version]#ETCD_VERSION="false"#ETCD_AUTO_COMPACTION_RETENTION="0"##[Profiling]#ETCD_ENABLE_PPROF="false"#ETCD_METRICS="basic"##[Auth]#ETCD_AUTH_TOKEN="simple"
etcd3
#[Member] #ETCD_CORS="" ETCD_DATA_DIR="/var/lib/etcd/default.etcd" #ETCD_WAL_DIR="" ETCD_LISTEN_PEER_URLS="https://192.168.88.93:2380" ETCD_LISTEN_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.93:2379" #ETCD_MAX_SNAPSHOTS="5" #ETCD_MAX_WALS="5" ETCD_NAME="etcd3" #ETCD_SNAPSHOT_COUNT="100000" #ETCD_HEARTBEAT_INTERVAL="100" #ETCD_ELECTION_TIMEOUT="1000" #ETCD_QUOTA_BACKEND_BYTES="0" #ETCD_MAX_REQUEST_BYTES="1572864" #ETCD_GRPC_KEEPALIVE_MIN_TIME="5s" #ETCD_GRPC_KEEPALIVE_INTERVAL="2h0m0s" #ETCD_GRPC_KEEPALIVE_TIMEOUT="20s" # #[Clustering] ETCD_INITIAL_ADVERTISE_PEER_URLS="https://192.168.88.93:2380" ETCD_ADVERTISE_CLIENT_URLS="https://127.0.0.1:2379, https://192.168.88.93:2379" #ETCD_DISCOVERY="" #ETCD_DISCOVERY_FALLBACK="proxy" #ETCD_DISCOVERY_PROXY="" #ETCD_DISCOVERY_SRV="" ETCD_INITIAL_CLUSTER="etcd1=https://192.168.88.90:2380,etcd2=https://192.168.88.92:2380,etcd3=https://192.168.88.93:2380 ETCD_INITIAL_CLUSTER_TOKEN="bigboss" #ETCD_INITIAL_CLUSTER_STATE="new" #ETCD_STRICT_RECONFIG_CHECK="true" #ETCD_ENABLE_V2="true" # #[Proxy] #ETCD_PROXY="off" #ETCD_PROXY_FAILURE_WAIT="5000" #ETCD_PROXY_REFRESH_INTERVAL="30000" #ETCD_PROXY_DIAL_TIMEOUT="1000" #ETCD_PROXY_WRITE_TIMEOUT="5000" #ETCD_PROXY_READ_TIMEOUT="0" # #[Security] ETCD_CERT_FILE="/etc/etcd/ssl/etcd.pem" ETCD_KEY_FILE="/etc/etcd/ssl/etcd-key.pem" #ETCD_CLIENT_CERT_AUTH="false" ETCD_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem" #ETCD_AUTO_TLS="false" ETCD_PEER_CERT_FILE="/etc/etcd/ssl/etcd.pem" ETCD_PEER_KEY_FILE="/etc/etcd/ssl/etcd-key.pem" #ETCD_PEER_CLIENT_CERT_AUTH="false" ETCD_PEER_TRUSTED_CA_FILE="/etc/etcd/ssl/etcd-ca.pem" #ETCD_PEER_AUTO_TLS="false" # #[Logging] #ETCD_DEBUG="false" #ETCD_LOG_PACKAGE_LEVELS="" #ETCD_LOG_OUTPUT="default" # #[Unsafe] #ETCD_FORCE_NEW_CLUSTER="false" # #[Version] #ETCD_VERSION="false" #ETCD_AUTO_COMPACTION_RETENTION="0" # #[Profiling] #ETCD_ENABLE_PPROF="false" #ETCD_METRICS="basic" # #[Auth] #ETCD_AUTH_TOKEN="simple"
在各个节点运行如下命令启动etcd及开机启动
systemctl start etcd systemctl enable etcd
查看etcd是否运行正常
etcdctl --endpoints "https://192.168.88.90:2379,https://192.168.88.92:2379,https://192.168.88.93:2379" --ca-file=/etc/etcd/ssl/etcd-ca.pem --cert-file=/etc/etcd/ssl/etcd.pem --key-file=/etc/etcd/ssl/etcd-key.pem cluster-health
输出如下：
member afaf79328d27196 is healthy: got healthy result from https://127.0.0.1:2379member 7637d1b4acd15c56 is healthy: got healthy result from https://127.0.0.1:2379member 874429f21be24f32 is healthy: got healthy result from https://127.0.0.1:2379cluster is healthy
至此，etcd集群搭建完毕。
持续更新搭建与踩坑过程~~~~
喜欢的朋友别忘记点赞，哈哈哈
作者：olaH
链接：https://www.jianshu.com/p/a48c206635f3
随时随地看视频
