一、前言
上一文介绍了如何搭建Dashboard。本篇将介绍如何搭建Ingress来访问K8S集群的Service。
二、Ingress简介
Ingress是个什么鬼，网上资料很多（推荐官方），大家自行研究。简单来讲，就是一个负载均衡的玩意，其主要用来解决使用NodePort暴露Service的端口时Node IP会漂移的问题。同时，若大量使用NodePort暴露主机端口，管理会非常混乱。
好的解决方案就是让外界通过域名去访问Service，而无需关心其Node IP及Port。那为什么不直接使用Nginx？这是因为在K8S集群中，如果每加入一个服务，我们都在Nginx中添加一个配置，其实是一个重复性的体力活，只要是重复性的体力活，我们都应该通过技术将它干掉。
Ingress就可以解决上面的问题，其包含两个组件Ingress Controller和Ingress：
Ingress
将Nginx的配置抽象成一个Ingress对象，每添加一个新的服务只需写一个新的Ingress的yaml文件即可Ingress Controller
将新加入的Ingress转化成Nginx的配置文件并使之生效
好了，废话不多，走你~
三、准备操作
官方文档
人生苦短，不造轮子，本文将以官方的标准脚本为基础进行搭建，参考请戳官方文档。官方文档中要求依次执行如下命令：
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/namespace.yaml \ | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/default-backend.yaml \ | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/configmap.yaml \ | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/tcp-services-configmap.yaml \ | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/udp-services-configmap.yaml \ | kubectl apply -f -
以上yaml文件创建Ingress用到的Namespace、ConfigMap，以及默认的后端default-backend。最关键的一点是，由于之前我们基于Kubeadm创建了K8S集群，则还必须执行：
curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/rbac.yaml \ | kubectl apply -f - curl https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/with-rbac.yaml \ | kubectl apply -f -
这是由于Kubeadm创建的集群默认开启了RABC，因此Ingress也必须创建相应的RABC权限控制。
导入镜像
但是，直接按照上述方式执行，我们的Ingress很可能会无法使用。所以，我们需要将上述Yaml文件全部wget下来，经过一些修改后才能执行kubectl apply -f创建。另外需要注意的是，这些yaml文件中提到的一些镜像，国内目前无法下载，如：
gcr.io/google_containers/defaultbackend:1.4 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0
本人已经提前下载好，大家请戳：
地址：https://pan.baidu.com/s/1N-bK9hI7JTZZB6AzmaT8PA密码：1a8a
拿到镜像后，在每个节点上执行如下命令导入镜像：
docker load < quay.io#kubernetes-ingress-controller#nginx-ingress-controller_0.14.0.tardocker tag 452a96d81c30 quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0 docker load < gcr.io#google_containers#defaultbackend.tardocker tag 452a96d81c30 gcr.io/google_containers/defaultbackend
如上所示，导入镜像后，别忘记给打tag，否则镜像名称为<none>：
image.png
四、主要文件介绍
这里，我们先对一些重要的文件进行简单介绍。
default-backend.yaml
default-backend的作用是，如果外界访问的域名不存在的话，则默认转发到default-http-backend这个Service，其会直接返回404：
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: default-http-backend labels: app: default-http-backend namespace: ingress-nginx spec: replicas: 1 selector: matchLabels: app: default-http-backend template: metadata: labels: app: default-http-backend spec: terminationGracePeriodSeconds: 60 containers: - name: default-http-backend # Any image is permissible as long as: # 1. It serves a 404 page at / # 2. It serves 200 on a /healthz endpoint image: gcr.io/google_containers/defaultbackend:1.4 livenessProbe: httpGet: path: /healthz port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 ports: - containerPort: 8080 resources: limits: cpu: 10m memory: 20Mi requests: cpu: 10m memory: 20Mi --- apiVersion: v1 kind: Service metadata: name: default-http-backend namespace: ingress-nginx labels: app: default-http-backend spec: ports: - port: 80 targetPort: 8080 selector: app: default-http-backend
rbac.yaml
rbac.yaml负责Ingress的RBAC授权的控制，其创建了Ingress用到的ServiceAccount、ClusterRole、Role、RoleBinding、ClusterRoleBinding。在上文《从零开始搭建Kubernetes集群（四、搭建K8S Dashboard）》中，我们已对这些概念进行了简单介绍。
apiVersion: v1 kind: ServiceAccount metadata: name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: nginx-ingress-clusterrole rules: - apiGroups: - "" resources: - configmaps - endpoints - nodes - pods - secrets verbs: - list - watch - apiGroups: - "" resources: - nodes verbs: - get - apiGroups: - "" resources: - services verbs: - get - list - watch - apiGroups: - "extensions" resources: - ingresses verbs: - get - list - watch - apiGroups: - "" resources: - events verbs: - create - patch - apiGroups: - "extensions" resources: - ingresses/status verbs: - update --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: Role metadata: name: nginx-ingress-role namespace: ingress-nginx rules: - apiGroups: - "" resources: - configmaps - pods - secrets - namespaces verbs: - get - apiGroups: - "" resources: - configmaps resourceNames: # Defaults to "<election-id>-<ingress-class>" # Here: "<ingress-controller-leader>-<nginx>" # This has to be adapted if you change either parameter # when launching the nginx-ingress-controller. - "ingress-controller-leader-nginx" verbs: - get - update - apiGroups: - "" resources: - configmaps verbs: - create - apiGroups: - "" resources: - endpoints verbs: - get --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: RoleBinding metadata: name: nginx-ingress-role-nisa-binding namespace: ingress-nginx roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: nginx-ingress-role subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx --- apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRoleBinding metadata: name: nginx-ingress-clusterrole-nisa-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: nginx-ingress-clusterrole subjects: - kind: ServiceAccount name: nginx-ingress-serviceaccount namespace: ingress-nginx
with-rbac.yaml
with-rbac.yaml是Ingress的核心，用于创建ingress-controller。前面提到过，ingress-controller的作用是将新加入的Ingress进行转化为Nginx的配置。
apiVersion: extensions/v1beta1 kind: Deployment metadata: name: nginx-ingress-controller namespace: ingress-nginx spec: replicas: 1 selector: matchLabels: app: ingress-nginx template: metadata: labels: app: ingress-nginx annotations: prometheus.io/port: '10254' prometheus.io/scrape: 'true' spec: serviceAccountName: nginx-ingress-serviceaccount containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --annotations-prefix=nginx.ingress.kubernetes.io env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace ports: - name: http containerPort: 80 - name: https containerPort: 443 livenessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 readinessProbe: failureThreshold: 3 httpGet: path: /healthz port: 10254 scheme: HTTP periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 securityContext: runAsNonRoot: false
如上，可以看到nginx-ingress-controller启动时传入了参数，分别为前面创建的default-backend-service以及configmap。
五、创建Ingress
1.创建Ingress-controller
需要注意的是，官方提供的with-rbac.yaml文件不能直接使用，我们必须修改两处：
加入hostNetwork配置
如下，在serviceAccountName上方添加hostNetwork: true:
spec: hostNetwork: true serviceAccountName: nginx-ingress-serviceaccount containers: - name: nginx-ingress-controller image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0 args: - /nginx-ingress-controller - --default-backend-service=$(POD_NAMESPACE)/default-http-backend - --configmap=$(POD_NAMESPACE)/nginx-configuration - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services - --udp-services-configmap=$(POD_NAMESPACE)/udp-services - --annotations-prefix=nginx.ingress.kubernetes.io
配置hostNetwork: true是一种直接定义Pod网络的方式。定义后，Ingress-controller的IP就与宿主机k8s-node1一样(192.168.56.101)，并且端口80也是宿主机上的端口。这样，我们通过该192.168.56.101:80，就可以直接访问到Ingress-controller（实际上就是nginx），然后Ingress-controller则会转发我们的请求到相应后端。
加入环境变量
在其env部分加入如下环境变量：
env: - name: POD_NAME valueFrom: fieldRef: fieldPath: metadata.name - name: POD_NAMESPACE valueFrom: fieldRef: fieldPath: metadata.namespace - name: KUBERNETES_MASTER value: http://192.168.56.101:8080
否则，创建后会提示如下错误：
[root@k8s-node1 ingress]# kubectl describe pod nginx-ingress-controller-9fbd7596d-rt9sf -n ingress-nginx省略前面... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 30s default-scheduler Successfully assigned nginx-ingress-controller-9fbd7596d-rt9sf to k8s-node1 Normal SuccessfulMountVolume 30s kubelet, k8s-node1 MountVolume.SetUp succeeded for volume "nginx-ingress-serviceaccount-token-lq2dt" Warning BackOff 21s kubelet, k8s-node1 Back-off restarting failed container Normal Pulled 11s (x3 over 29s) kubelet, k8s-node1 Container image "quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.14.0" already present on machine Normal Created 11s (x3 over 29s) kubelet, k8s-node1 Created container Warning Failed 10s (x3 over 28s) kubelet, k8s-node1 Error: failed to start container "nginx-ingress-controller": Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "exec: \"/nginx-ingress-controller\": stat /nginx-ingress-controller: no such file or directory": unknown
修改with-rbac.yaml后，使用kubectl -f create命令分别执行如下yaml文件，即可创建Ingress-controller：
image.png
创建成功后如下所示：
[root@k8s-node1 ingress]# kubectl get pod -n ingress-nginx -o wideNAME READY STATUS RESTARTS AGE IP NODEdefault-http-backend-5c6d95c48-pdjn9 1/1 Running 0 23s 192.168.36.81 k8s-node1 nginx-ingress-controller-547cd7d9cb-jmvpn 1/1 Running 0 8s 192.168.36.82 k8s-node1
2.创建自定义Ingress
有了ingress-controller，我们就可以创建自定义的Ingress了。这里已提前搭建好了Kibana服务，我们针对Kibana创建一个Ingress：
apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kibana-ingress namespace: defaultspec: rules: - host: myk8s.com http: paths: - path: / backend: serviceName: kibana servicePort: 5601
其中：
rules中的host必须为域名，不能为IP，表示Ingress-controller的Pod所在主机域名，也就是Ingress-controller的IP对应的域名。
paths中的path则表示映射的路径。如映射
/表示若访问myk8s.com，则会将请求转发至Kibana的service，端口为5601。
创建成功后，查看：
[root@k8s-node1 ingress]# kubectl get ingress -o wideNAME HOSTS ADDRESS PORTS AGE kibana-ingress myk8s.com 80 6s
我们再执行kubectl exec nginx-ingress-controller-5b79cbb5c6-2zr7f -it cat /etc/nginx/nginx.conf -n ingress-nginx，可以看到生成nginx配置，篇幅较长，各位自行筛选：
## start server myk8s.com
server {
server_name myk8s.com ;
listen 80;
listen [::]:80;
set $proxy_upstream_name "-";
location /kibana {
log_by_lua_block {
}
port_in_redirect off;
set $proxy_upstream_name "";
set $namespace "kube-system"; set $ingress_name "dashboard-ingress"; set $service_name "kibana";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
# No endpoints available for the request
return 503;
}
location / {
log_by_lua_block {
}
port_in_redirect off;
set $proxy_upstream_name "";
set $namespace "default"; set $ingress_name "kibana-ingress"; set $service_name "kibana";
client_max_body_size "1m";
proxy_set_header Host $best_http_host;
# Pass the extracted client certificate to the backend
# Allow websocket connections
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_set_header X-Real-IP $the_real_ip;
proxy_set_header X-Forwarded-For $the_real_ip;
proxy_set_header X-Forwarded-Host $best_http_host;
proxy_set_header X-Forwarded-Port $pass_port;
proxy_set_header X-Forwarded-Proto $pass_access_scheme;
proxy_set_header X-Original-URI $request_uri;
proxy_set_header X-Scheme $pass_access_scheme;
# Pass the original X-Forwarded-For
proxy_set_header X-Original-Forwarded-For $http_x_forwarded_for;
# mitigate HTTPoxy Vulnerability
# https://www.nginx.com/blog/mitigating-the-httpoxy-vulnerability-with-nginx/
proxy_set_header Proxy "";
# Custom headers to proxied server
proxy_connect_timeout 5s;
proxy_send_timeout 60s;
proxy_read_timeout 60s;
proxy_buffering "off";
proxy_buffer_size "4k";
proxy_buffers 4 "4k";
proxy_request_buffering "on";
proxy_http_version 1.1;
proxy_cookie_domain off;
proxy_cookie_path off;
# In case of errors try the next upstream server before returning an error
proxy_next_upstream error timeout invalid_header http_502 http_503 http_504;
proxy_next_upstream_tries 0;
# No endpoints available for the request
return 503;
}
} ## end server myk8s.com3.设置host
首先，我们需要在Ingress-controller的Pod所在主机上（这里为k8s-node1），将上面提到的域名myk8s.com追加入/etc/hosts文件：
192.168.56.101 myk8s.com
除此之外，如果想在自己的Windows物理机上使用浏览器访问kibana，也需要在C:\Windows\System32\drivers\etc\hosts文件内加入上述内容。设置后，分别在k8s-node1和物理机上测试无误即可：
image.png
image.png
六、测试
在Windows物理机上，使用Chrome访问myk8s.com，也就是相当于访问了192.168.56.101:80：
image.png
随意访问一个错误的地址myk8s.com/abc，返回预期的404：
image.png
七、废话
至此，我们的Ingress已经搭建完毕，实现了在外部通过域名访问K8S集群Service的功能。如果大家有兴趣，可以尝试为Ingress配置TLS，这样就可以访问如Dashboard这种https服务了。
本人水平有限，难免有错误或遗漏之处，望大家指正和谅解，欢迎评论留言。
作者：宅楠军
链接：https://www.jianshu.com/p/feeea0bbd73e