服务热线
135-6963-3175
kubeadm安装部署k8s集群
更新时间 2020-9-14 浏览1857次
Kubeadm方式安装部署k8s
所有方式：
minikube：官方提供的快速本地运行的工具，可以学习使用，但不能用于生产。
kubeadm：也是一个工具，通过kubeadm init和kubeadm join，用于快速安装部署。（不建议用于生产）
二进制包：直接下载编译好的包进行部署。
源码安装：对k8s源码进行编译后进行安装部署。（不建议初学者）
yum方式：yum install方式进行安装部署（前面文章已经有讲解）
下面我们通过kubeadm方式进行安装
准备工作：
1、配置ip,dns,hostname,hosts文件
2、关闭防火墙、selinux、swap分区
3、安装依赖包
4、同步系统时间
5、内核参数优化
| 主机名 | ip |
| master01 | 192.168.1.107 |
| node01 | 192.168.1.108 |
| node02 | 192.168.1.109 |
在所有节点执行下面操作：
cat <<EOF >>/etc/hosts 192.168.1.107 master01 192.168.1.108 node01 192.168.1.109 node02 EOF
关闭防火墙、selinux、内存交换
systemctl stop firewalld systemctl disable firewalld setenforce 0 sed -i 's/^SELINUX=.*/SELINUX=disabled/' /etc/selinux/config swapoff -a sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab
同步时间
[root@master01 ~]# ntpdate time.windows.com 25 Sep 00:01:56 ntpdate[19442]: step time server 52.231.114.183 offset -28378.621119 sec
加载内核模块
modprobe ip_vs_rr modprobe br_netfilter
优化内核参数
cat > /etc/sysctl.d/kubernetes.conf << EOF net.bridge.bridge-nf-call-iptables=1 net.bridge.bridge-nf-call-ip6tables=1 net.ipv4.ip_forward=1 net.ipv4.tcp_tw_recycle=0 vm.swappiness=0 # 禁止使用 swap 空间，只有当系统 OOM 时才允许使用它 vm.overcommit_memory=1 # 不检查物理内存是否够用 vm.panic_on_oom=0 # 开启 OOM fs.inotify.max_user_instances=8192 fs.inotify.max_user_watches=1048576 fs.file-max=52706963 fs.nr_open=52706963 net.ipv6.conf.all.disable_ipv6=1 net.netfilter.nf_conntrack_max=2310720 EOF sysctl -p /etc/sysctl.d/kubernetes.conf
安装kubernetes
所有节点添加k8s源和安装docker
2. yum安装kubeadm、kubelet、kubectl
3. 部署主节点，网络插件、工作节点注册到主节点
在每台节点上执行下面操作
添加kubernetes yum源
cat <<EOF >>/etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes repo baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/ gpgcheck=0 gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg enabled=1 EOF
安装并docker
参考docker安装部署章节文章
部署kubernetes Master
此操作在master节点执行
[root@master01 ~]# kubeadm init --apiserver-advertise-address=192.168.1.107 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.17.0 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16 this version of kubeadm only supports deploying clusters with the control plane version >= 1.18.0. Current version: v1.17.0 To see the stack trace of this error execute with --v=5 or higher [root@master01 ~]# kubeadm init --apiserver-advertise-address=192.168.1.107 --image-repository registry.aliyuncs.com/google_containers --kubernetes-version v1.19.0 --service-cidr=10.1.0.0/16 --pod-network-cidr=10.244.0.0/16 W0925 00:03:38.661081 20948 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io] [init] Using Kubernetes version: v1.19.0 [preflight] Running pre-flight checks [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/ [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master01] and IPs [10.1.0.1 192.168.1.107] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost master01] and IPs [192.168.1.107 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost master01] and IPs [192.168.1.107 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 9.505845 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config-1.19" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master01 as control-plane by adding the label "node-role.kubernetes.io/master=''" [mark-control-plane] Marking the node master01 as control-plane by adding the taints [node-role.kubernetes.io/master:NoSchedule] [bootstrap-token] Using token: vxnij3.ckt678nfri7whmh0 [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: https://kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.1.107:6443 --token vxnij3.ckt678nfri7whmh0 \ --discovery-token-ca-cert-hash sha256:a114c1f50c3a4779e37832466e869327f407fdd2bc8c1e75f0c81278a64b1e70 [root@master01 ~]#
此时保存下来日志最后的输出，下面会用到：
kubeadm join 192.168.1.107:6443 --token vxnij3.ckt678nfri7whmh0 \
--discovery-token-ca-cert-hash sha256:a114c1f50c3a4779e37832466e869327f407fdd2bc8c1e75f0c81278a64b1e70
在主节点执行：（配置kubectl）
mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config
安装flannel网络插件（所有节点）
# 手动拉取flannel的docker镜像 docker pull easzlab/flannel:v0.11.0-amd64 # 修改镜像名称 docker tag easzlab/flannel:v0.11.0-amd64 quay.io/coreos/flannel:v0.11.0-amd64
下载并安装flannel资源清单（master节点）
[root@master01 ~]# wget https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml [root@master01 ~]# kubectl apply -f kube-flannel.yml podsecuritypolicy.policy/psp.flannel.unprivileged created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds-amd64 created daemonset.apps/kube-flannel-ds-arm64 created daemonset.apps/kube-flannel-ds-arm created daemonset.apps/kube-flannel-ds-ppc64le created daemonset.apps/kube-flannel-ds-s390x created
kube-flannel.yml文件下载需要翻墙，文件内容可参考下面：
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: psp.flannel.unprivileged
annotations:
seccomp.security.alpha.kubernetes.io/allowedProfileNames: docker/default
seccomp.security.alpha.kubernetes.io/defaultProfileName: docker/default
apparmor.security.beta.kubernetes.io/allowedProfileNames: runtime/default
apparmor.security.beta.kubernetes.io/defaultProfileName: runtime/default
spec:
privileged: false
volumes:
- configMap
- secret
- emptyDir
- hostPath
allowedHostPaths:
- pathPrefix: "/etc/cni/net.d"
- pathPrefix: "/etc/kube-flannel"
- pathPrefix: "/run/flannel"
readOnlyRootFilesystem: false
# Users and groups
runAsUser:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
fsGroup:
rule: RunAsAny
# Privilege Escalation
allowPrivilegeEscalation: false
defaultAllowPrivilegeEscalation: false
# Capabilities
allowedCapabilities: ['NET_ADMIN', 'NET_RAW']
defaultAddCapabilities: []
requiredDropCapabilities: []
# Host namespaces
hostPID: false
hostIPC: false
hostNetwork: true
hostPorts:
- min: 0
max: 65535
# SELinux
seLinux:
# SELinux is unused in CaaSP
rule: 'RunAsAny'
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
rules:
- apiGroups: ['extensions']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames: ['psp.flannel.unprivileged']
- apiGroups:
- ""
resources:
- pods
verbs:
- get
- apiGroups:
- ""
resources:
- nodes
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes/status
verbs:
- patch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: flannel
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: flannel
subjects:
- kind: ServiceAccount
name: flannel
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: flannel
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: kube-flannel-cfg
namespace: kube-system
labels:
tier: node
app: flannel
data:
cni-conf.json: |
{
"name": "cbr0",
"cniVersion": "0.3.1",
"plugins": [
{
"type": "flannel",
"delegate": {
"hairpinMode": true,
"isDefaultGateway": true
}
},
{
"type": "portmap",
"capabilities": {
"portMappings": true
}
}
]
}
net-conf.json: |
{
"Network": "10.244.0.0/16",
"Backend": {
"Type": "vxlan"
}
}
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: kube-flannel-ds
namespace: kube-system
labels:
tier: node
app: flannel
spec:
selector:
matchLabels:
app: flannel
template:
metadata:
labels:
tier: node
app: flannel
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/os
operator: In
values:
- linux
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- operator: Exists
effect: NoSchedule
serviceAccountName: flannel
initContainers:
- name: install-cni
image: quay.io/coreos/flannel:v0.13.0-rc2
command:
- cp
args:
- -f
- /etc/kube-flannel/cni-conf.json
- /etc/cni/net.d/10-flannel.conflist
volumeMounts:
- name: cni
mountPath: /etc/cni/net.d
- name: flannel-cfg
mountPath: /etc/kube-flannel/
containers:
- name: kube-flannel
image: quay.io/coreos/flannel:v0.13.0-rc2
command:
- /opt/bin/flanneld
args:
- --ip-masq
- --kube-subnet-mgr
resources:
requests:
cpu: "100m"
memory: "50Mi"
limits:
cpu: "100m"
memory: "50Mi"
securityContext:
privileged: false
capabilities:
add: ["NET_ADMIN", "NET_RAW"]
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
volumeMounts:
- name: run
mountPath: /run/flannel
- name: flannel-cfg
mountPath: /etc/kube-flannel/
volumes:
- name: run
hostPath:
path: /run/flannel
- name: cni
hostPath:
path: /etc/cni/net.d
- name: flannel-cfg
configMap:
name: kube-flannel-cfg工作节点注册到master（node节点）
使用kubeadm join注册Node节点到Master
[root@localhost ~]# kubeadm join 192.168.1.107:6443 --token vxnij3.ckt678nfri7whmh0 \ > --discovery-token-ca-cert-hash sha256:a114c1f50c3a4779e37832466e869327f407fdd2bc8c1e75f0c81278a64b1e70 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -oyaml' W0925 00:10:47.612187 3661 kubelet.go:205] detected "systemd" as the Docker cgroup driver, the provided value "cgroupfs" in "KubeletConfiguration" will be overrided [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... [kubelet-check] Initial timeout of 40s passed. This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.
注:使用kubeadm init输出的日志中的命令
查看集群的node状态
[root@master01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master01 Ready master 8m20s v1.19.2 node01 NotReady <none> 4s v1.19.2 node02 NotReady <none> 6s v1.19.2 [root@master01 ~]#
查看k8s pods kube-system
[root@master01 ~]# kubectl get pods -n kube-system NAME READY STATUS RESTARTS AGE coredns-6d56c8448f-5mqqd 1/1 Running 0 12m coredns-6d56c8448f-n69xz 1/1 Running 0 12m etcd-master01 1/1 Running 1 12m kube-apiserver-master01 1/1 Running 1 12m kube-controller-manager-master01 1/1 Running 3 12m kube-flannel-ds-9pqnf 0/1 Init:ImagePullBackOff 0 4m34s kube-flannel-ds-m6nbb 1/1 Running 0 7m5s kube-flannel-ds-nsbcb 0/1 Init:ImagePullBackOff 0 4m36s kube-proxy-2t2zj 1/1 Running 0 4m34s kube-proxy-l54fg 1/1 Running 0 4m36s kube-proxy-rrl5f 1/1 Running 0 12m kube-scheduler-master01 1/1 Running 3 12m
部署nginx应用集群
[root@master01 ~]# kubectl create deployment nginx --image=nginx
deployment.apps/nginx created
[root@master01 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-6799fc88d8-hps2l 0/1 ContainerCreating 0 26s <none> node01 <none> <none>
[root@master01 ~]# kubectl describe pod nginx-6799fc88d8-hps2l
Name: nginx-6799fc88d8-hps2l
Namespace: default
Priority: 0
Node: node01/192.168.1.108
Start Time: Fri, 25 Sep 2020 00:17:36 +0800
Labels: app=nginx
pod-template-hash=6799fc88d8
Annotations: <none>
Status: Pending
IP:
IPs: <none>
Controlled By: ReplicaSet/nginx-6799fc88d8
Containers:
nginx:
Container ID:
Image: nginx
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Environment: <none>
Mounts:
/var/run/secrets/kubernetes.io/serviceaccount from default-token-rkkrz (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
default-token-rkkrz:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-rkkrz
Optional: false
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 43s default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Warning FailedScheduling 42s default-scheduler 0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) had taint {node.kubernetes.io/not-ready: }, that the pod didn't tolerate.
Normal Scheduled 20s default-scheduler Successfully assigned default/nginx-6799fc88d8-hps2l to node01
Normal Pulling 20s kubelet Pulling image "nginx"
[root@master01 ~]# kubectl get pod -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
nginx-6799fc88d8-hps2l 1/1 Running 0 55s 10.244.2.2 node01 <none> <none>
[root@master01 ~]# curl 10.244.2.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
body {
width: 35em;
margin: 0 auto;
font-family: Tahoma, Verdana, Arial, sans-serif;
}
</style>
</head>
<body>
<h1>Welcome to nginx!</h1>
<p>If you see this page, the nginx web server is successfully installed and
working. Further configuration is required.</p>
<p>For online documentation and support please refer to
<a href="http://nginx.org/">nginx.org</a>.<br/>
Commercial support is available at
<a href="http://nginx.com/">nginx.com</a>.</p>
<p><em>Thank you for using nginx.</em></p>
</body>
</html>查看有效的token列表
[root@master01 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS vxnij3.ckt678nfri7whmh0 23h 2020-09-26T00:03:52+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token [root@master01 ~]#
token的有效期为24小时，当过期后解决办法：
# 1.查看当前的token列表 [root@K8S00 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS vxnij3.ckt678nfri7whmh0 23h 2020-09-26T00:03:52+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token # 2.重新生成新的token [root@K8S00 ~]# kubeadm token create 369tcl.oe4punpoj9gaijh7 # 3.再次查看当前的token列表 [root@K8S00 ~]# kubeadm token list TOKEN TTL EXPIRES USAGES DESCRIPTION EXTRA GROUPS 369tcl.oe4punpoj9gaijh7 23h 2019-12-24T16:05:18+08:00 authentication,signing <none> system:bootstrappers:kubeadm:default-node-token vxnij3.ckt678nfri7whmh0 23h 2020-09-26T00:03:52+08:00 authentication,signing The default bootstrap token generated by 'kubeadm init'. system:bootstrappers:kubeadm:default-node-token # 4.获取ca证书sha256编码hash值 [root@K8S00 ~]# openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //' 7ae10591aa593c2c36fb965d58964a84561e9ccd416ffe7432550a0d0b7e4f90 # 5.节点加入集群 [root@k8s-node03 ~]# kubeadm join --token 369tcl.oe4punpoj9gaijh7(新的token) --discovery-token-ca-cert-hash sha256:7ae10591aa593c2c36fb965d58964a84561e9ccd416ffe7432550a0d0b7e4f90(ca证书sha256编码hash值) 192.168.1.107:6443 --skip-preflight-chec