部署master节点

kubernetes master 节点包含的组件：

kube-apiserver

kube-scheduler

kube-controller-manager

本文档介绍部署一个三节点高可用 master 集群的步骤，分别命名为k8s-host1、k8s-host2、k8s-host3：

k8s-host1：172.16.120.154

k8s-host2：172.16.120.155

k8s-host3：172.16.120.156

安装docker

在每台主机安装docker。建议使用docker v1.12 , 但是 v1.11, v1.13 and 17.03 也可以正常工作. 版本 17.06+ 可能工作, 但是没有被k8s团队测试过。

PS：kubelet设置的cgroup driver应该和Docker设置的cgroup driver一致。这边统一为systemd。

yum install -y dockersystemctl enable docker && systemctl start docker

安装kubeadm, kubelet 和 kubectl

cat <
    
      /etc/yum.repos.d/kubernetes.repo[kubernetes]name=Kubernetesbaseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64enabled=1gpgcheck=1repo_gpgcheck=1gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpgEOFsetenforce 0yum install -y kubelet kubeadm kubectlsystemctl enable kubelet && systemctl start kubelet

通过运行命令setenforce 0 禁用selinux，主要为了允许容器可以访问主机文件系统和pod networks的需要。

Some users on RHEL/CentOS 7 have reported issues with traffic being routed incorrectly due to iptables being bypassed。所以需要执行下面命令：

cat <
    
       /etc/sysctl.d/k8s.confnet.bridge.bridge-nf-call-ip6tables = 1net.bridge.bridge-nf-call-iptables = 1EOFsysctl --system

安装master

kubeadm init

kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16

apiserver-advertise-address该参数一般指定为haproxy+keepalived 的vip。

pod-network-cidr 主要是在搭建pod network（calico）时候需要在init时候指定。

出现如下错误：

[root@k8s-node04 ~]# kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16[init] Using Kubernetes version: v1.9.0[init] Using Authorization modes: [Node RBAC][preflight] Running pre-flight checks.        [WARNING FileExisting-crictl]: crictl not found in system path[preflight] Some fatal errors occurred:        [ERROR Swap]: running with swap on is not supported. Please disable swap[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`

解决办法：

1：kubelet启动参数增加--fail-swap-on=false

[Service]Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"Environment="KUBELET_SYSTEM_PODS_ARGS=--pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true"Environment="KUBELET_NETWORK_ARGS=--network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin"Environment="KUBELET_DNS_ARGS=--cluster-dns=10.96.0.10 --cluster-domain=cluster.local"Environment="KUBELET_AUTHZ_ARGS=--authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt"Environment="KUBELET_CADVISOR_ARGS=--cadvisor-port=0"Environment="KUBELET_CGROUP_ARGS=--cgroup-driver=systemd"Environment="KUBELET_CERTIFICATE_ARGS=--rotate-certificates=true --cert-dir=/var/lib/kubelet/pki"Environment="KUBELET_SWAP_ARGS=--fail-swap-on=false"ExecStart=ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_SYSTEM_PODS_ARGS $KUBELET_NETWORK_ARGS $KUBELET_DNS_ARGS $KUBELET_AUTHZ_ARGS $KUBELET_CADVISOR_ARGS $KUBELET_CGROUP_ARGS $KUBELET_CERTIFICATE_ARGS $KUBELET_EXTRA_ARGS $KUBELET_SWAP_ARGS

2：kubeadm init 参数增加ignore-preflight-errors

kubeadm init --apiserver-advertise-address=172.16.120.200  --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors 'Swap'

3：在物理机上可能并不会出现该问题。

最终安装成功

[root@k8s-host1 manifests]# kubeadm init --apiserver-advertise-address=172.16.120.200  --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors 'Swap'[init] Using Kubernetes version: v1.9.0[init] Using Authorization modes: [Node RBAC][preflight] Running pre-flight checks.        [WARNING Swap]: running with swap on is not supported. Please disable swap        [WARNING FileExisting-crictl]: crictl not found in system path[preflight] Starting the kubelet service[certificates] Generated ca certificate and key.[certificates] Generated apiserver certificate and key.[certificates] apiserver serving cert is signed for DNS names [k8s-host1 kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local] and IPs [10.96.0.1 172.16.120.200][certificates] Generated apiserver-kubelet-client certificate and key.[certificates] Generated sa key and public key.[certificates] Generated front-proxy-ca certificate and key.[certificates] Generated front-proxy-client certificate and key.[certificates] Valid certificates and keys now exist in "/etc/kubernetes/pki"[kubeconfig] Wrote KubeConfig file to disk: "admin.conf"[kubeconfig] Wrote KubeConfig file to disk: "kubelet.conf"[kubeconfig] Wrote KubeConfig file to disk: "controller-manager.conf"[kubeconfig] Wrote KubeConfig file to disk: "scheduler.conf"[controlplane] Wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"[controlplane] Wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"[controlplane] Wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"[init] Waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests".[init] This might take a minute or longer if the control plane images have to be pulled.[apiclient] All control plane components are healthy after 30.002346 seconds[uploadconfig] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace[markmaster] Will mark node k8s-host1 as master by adding a label and a taint[markmaster] Master k8s-host1 tainted and labelled with key/value: node-role.kubernetes.io/master=""[bootstraptoken] Using token: 26ea24.cb987cb5cf6f29a6[bootstraptoken] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials[bootstraptoken] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstraptoken] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster[bootstraptoken] Creating the "cluster-info" ConfigMap in the "kube-public" namespace[addons] Applied essential addon: kube-dns[addons] Applied essential addon: kube-proxyYour Kubernetes master has initialized successfully!To start using your cluster, you need to run the following as a regular user:  mkdir -p $HOME/.kube  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config  sudo chown $(id -u):$(id -g) $HOME/.kube/configYou should now deploy a pod network to the cluster.Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:  https://kubernetes.io/docs/concepts/cluster-administration/addons/You can now join any number of machines by running the following on each nodeas root:  kubeadm join --token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab

PS 需要记下 kubeadm join --token 26ea24.cb987cb5cf6f29a6 172.16.120.200:6443 --discovery-token-ca-cert-hash sha256:a2d5e0391695bb122d4a43dc951f93934a2aaa1360e00556401a49ea0ec2bdab，便于以后node加入到k8s集群中。

将kube-apiserver指向上一节部署的etcd集群

首先需要将上节生成的etcd ca等证书cp到mater主机的/etc/etcd/ssl/下

默认kubeadm安装了一个etcd节点，非集群，非加密，所以我们必须对kube-apiserver.yaml做修改。

修改后的kube-apiserver.yaml：

apiVersion: v1kind: Podmetadata:  annotations:    scheduler.alpha.kubernetes.io/critical-pod: ""  labels:    component: kube-apiserver    tier: control-plane  name: kube-apiserver  namespace: kube-systemspec:  containers:  - command:    - kube-apiserver    - --allow-privileged=true    - --requestheader-username-headers=X-Remote-User    - --requestheader-group-headers=X-Remote-Group    - --requestheader-extra-headers-prefix=X-Remote-Extra-    - --service-cluster-ip-range=10.96.0.0/12    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt    - --proxy-client-key-file=/etc/kubernetes/pki/front-proxy-client.key    - --enable-bootstrap-token-auth=true    - --service-account-key-file=/etc/kubernetes/pki/sa.pub    - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt    - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt    - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key    - --proxy-client-cert-file=/etc/kubernetes/pki/front-proxy-client.crt    - --insecure-port=0    - --admission-control=Initializers,NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,ResourceQuota    - --requestheader-allowed-names=front-proxy-client    - --advertise-address=172.16.120.200    - --client-ca-file=/etc/kubernetes/pki/ca.crt    - --secure-port=6443    - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname    - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key    - --authorization-mode=Node,RBAC    - --etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379    - --etcd-cafile=/etc/etcd/ssl/ca.pem    - --etcd-certfile=/etc/etcd/ssl/etcd.pem    - --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem    image: gcr.io/google_containers/kube-apiserver-amd64:v1.9.0    livenessProbe:      failureThreshold: 8      httpGet:        host: 172.16.120.200        path: /healthz        port: 6443        scheme: HTTPS      initialDelaySeconds: 15      timeoutSeconds: 15    name: kube-apiserver    resources:      requests:        cpu: 250m    volumeMounts:    - mountPath: /etc/kubernetes/pki      name: k8s-certs      readOnly: true    - mountPath: /etc/ssl/certs      name: ca-certs      readOnly: true    - mountPath: /etc/pki      name: ca-certs-etc-pki      readOnly: true    - mountPath: /etc/etcd/ssl      name: ca-certs-etc-etcd-ssl      readOnly: true  hostNetwork: true  volumes:  - hostPath:      path: /etc/kubernetes/pki      type: DirectoryOrCreate    name: k8s-certs  - hostPath:      path: /etc/ssl/certs      type: DirectoryOrCreate    name: ca-certs  - hostPath:      path: /etc/pki      type: DirectoryOrCreate    name: ca-certs-etc-pki  - hostPath:      path: /etc/etcd/ssl      type: DirectoryOrCreate    name: ca-certs-etc-etcd-sslstatus: {}

注意添加了以下四项：

--etcd-cafile=/etc/etcd/ssl/ca.pem  --etcd-certfile=/etc/etcd/ssl/etcd.pem  --etcd-keyfile=/etc/etcd/ssl/etcd-key.pem    --etcd-servers=https://172.16.120.151:2379,https://172.16.120.152:2379,https://172.16.120.153:2379# 以及挂载 - mountPath: /etc/etcd/ssl      name: ca-certs-etc-etcd-ssl      readOnly: true       - hostPath:      path: /etc/etcd/ssl      type: DirectoryOrCreate    name: ca-certs-etc-etcd-ssl

确认第一台master三大组件都成功启动

kubectl get componentstatusesNAME                 STATUS    MESSAGE              ERRORcontroller-manager   Healthy   okscheduler            Healthy   oketcd-1               Healthy   {"health": "true"}etcd-0               Healthy   {"health": "true"}etcd-2               Healthy   {"health": "true"}

安装其他主机

我一般的做法是，依旧使用kubeadm init --apiserver-advertise-address=172.16.120.200 --pod-network-cidr=192.168.0.0/16 --ignore-preflight-errors 'Swap'

主要目的是下载k8s相关组件docker镜像等，其实就是让kubeadm把容易忽略的工作都做了。

然后将第一台/etc/kubernetes/下所有文件拷贝到其他master主机对应目录下，直至其他机器master三大组件启动成功。

至此一个完整的master高可用集群搭建完毕。

转载地址：http://zrlpa.baihongyu.com/

你可能感兴趣的文章