【云原生】最新版Kubernetes集群基于Containerd部署

---

文章目录 Kubernetes集群基于Containerd部署资源列表基础环境一、基础环境准备1.1、关闭Swap分区1.2、添加hosts解析1.3、桥接的IPv4流量传递给iptables的链 二、准备Containerd容器运行时2.1、安装Containerd2.2、配置Containerd2.3、启动Containerd 三、部署Kubernetes集群3.1、安装Kubeadm工具3.2、配置crictl工具3.3、配置Containerd镜像加速器3.4、测试crictl工具是否可用3.5、初始化Master节点3.6、Node节点加入集群Node节点都要操作 四、部署网络插件（CNI）4.1、拉取必要镜像4.2、部署网络插件4.2、查看节点状态4.3、查看组件状态4.4、查看所有Pod状态4.5 解决异常pod状态 五、Containerd基本操作5.1、Crictl镜像管理5.2、Crictl容器管理5.3、Crictl Pod管理

---

Kubernetes集群基于Containerd部署 资源列表 操作系统配置主机名IPCentOS 7.9.20092C 2Gmaster192.168.47.102CentOS 7.9.20092C 2Gworker192.168.47.103 基础环境 关闭防火墙 systemctl stop firewalld #关闭开机自启动 systemctl disable firewalld 关闭内核安全机制 setenforce 0 sed -i "s/^SELINUX=.*/SELINUX=disabled/g" /etc/selinux/config 修改主机名 # 192.168.47.102机器执行 hostnamectl set-hostname master # 192.168.47.103机器执行 hostnamectl set-hostname worker01 一、基础环境准备

待安装的每台机器都需要进行操作，以下只对演示环境的机器进行配置并演示。

1.1、关闭Swap分区

集群的每台机器都需要关闭swap分区

# 临时关闭 swapoff -a # 永久关闭 sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab 1.2、添加hosts解析 cat >> /etc/hosts << EOF 192.168.47.102 master 192.168.47.103 worker01 EOF

1.3、桥接的IPv4流量传递给iptables的链 # 加载 overlay 内核模块 modprobe overlay # 加载 br_netfilter 模块 modprobe br_netfilter cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf net.bridge.bridge-nf-call-iptables = 1 net.ipv4.ip_forward = 1 net.bridge.bridge-nf-call-ip6tables = 1 EOF sysctl --system

二、准备Containerd容器运行时

待安装的每台机器都需要进行操作，以下只对演示环境的机器进行配置并演示。

2.1、安装Containerd 在线安装 # 添加 docker 源，containerd也在docker源内的 cat <<EOF | sudo tee /etc/yum.repos.d/docker-ce.repo [docker] name=docker-ce baseurl= mirrors.aliyun /docker-ce/linux/centos/7/x86_64/stable/ enabled=1 gpgcheck=1 gpgkey= mirrors.aliyun /docker-ce/linux/centos/gpg EOF # 快速建立 yum 缓存 yum makecache fast # 安装containerd # 列出所有containerd版本 yum list containerd.io --showduplicates yum -y install containerd.io-1.6.6-3.1.el7.x86_64 2.2、配置Containerd # 生成配置文件 mkdir -p /etc/containerd containerd config default | sudo tee /etc/containerd/config.toml # 修改/etc/containerd/config.toml文件中sandbox_image的值 grep 'sandbox_image' /etc/containerd/config.toml # 在原有的基础上进行修改， sandbox_image = "registry.aliyuncs /google_containers/pause:3.9" 2.3、启动Containerd systemctl enable containerd systemctl start containerd 三、部署Kubernetes集群 3.1、安装Kubeadm工具

所有节点都要操作

在线安装 cat << EOF >> /etc/yum.repos.d/kubernetes.repo [kubernetes] name=Kubernetes baseurl= mirrors.aliyun /kubernetes/yum/repos/kubernetes-el7-x86_64/ enabled=1 gpgcheck=1 repo_gpgcheck=1 gpgkey= mirrors.aliyun /kubernetes/yum/doc/yum-key.gpg mirrors.aliyun /kubernetes/yum/doc/rpm-package-key.gpg EOF # 快速建立yum缓存 yum makecache fast # 安装 kubectl：命令行管理工具、kubeadm：安装K8S集群工具、kubelet管理容器工具 yum install -y kubelet-1.27.0 kubeadm-1.27.0 kubectl-1.27.0 # 切记这个时候不要启动，只需要设置为开机自启 systemctl enable kubelet.service 3.2、配置crictl工具 crictl是CRI兼容的容器运行时命令行接口。你可以使用它来检查和调试Kubernetes节点上的容器运行时和应用程序。crictl和它的源代码在cri-tools 代码库更好 Containerd后，以上我们常用的docker命令也不再使用了，取而代之恶的分别是 crictl 和 ctr 两个命令行客户端crictl是遵循CRI接口规范的一个命令行工具，通常用它来检查和管理kubelet节点上的容器运行时和镜像ctr是containerd的一个客户端工具 # 所有节点都要操作 cat << EOF >> /etc/crictl.yaml runtime-endpoint: unix:///var/run/containerd/containerd.sock image-endpoint: unix:///var/run/containerd/containerd.sock timeout: 10 debug: false EOF 3.3、配置Containerd镜像加速器

所有节点都要操作

# 在/etc/containerd/config.toml找到[plugins."io.containerd.grpc.v1.cri".registry.mirrors] ，默认在153行左右，在此行下面添加两行配置，阿里云的镜像加速器 vim /etc/containerd/config.toml [plugins."io.containerd.grpc.v1.cri".registry.mirrors]. #亲测有用 [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"] endpoint = [" docker.anyhub.us.kg"," dockerhub.jobcher "," docker.ckyl.me"," hub.uuuadc.top"," docker.1panel.live"," hub.rat.dev/"," docker.chenby "," docker.m.daocloud.io"," docker.awsl9527 "," huecker.io"," dockerhub.timeweb.cloud"," noohub.ru"] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."registry.k8s.io"] endpoint=[" 45hrqeao.mirror.aliyuncs ", " k8s.m.daocloud.io", " docker.mirrors.ustc.edu "," hub-mirror.c.163 "] [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"] endpoint = ["registry.aliyuncs /google_containers"] # 保存退出，刷新服务 systemctl restart containerd 3.4、测试crictl工具是否可用 # 以拉取一个pause镜像进行演示是否可用 crictl pull registry.k8s.io/pause:3.9 Image is up to date for sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c # 查看镜像 [root@localhost containerd]# crictl images IMAGE TAG IMAGE ID SIZE registry.k8s.io/pause 3.9 e6f1816883972 322kB

3.5、初始化Master节点

在master节点上操作

# 生成配置文件 kubeadm config print init-defaults > kubeadm-init.yaml # 修改kubeadm-init.yaml文件的advertiseAddress、name、imageRepository，添加Pod网络podSubnet vim kubeadm-init.yaml apiVersion: kubeadm.k8s.io/v1beta3 bootstrapTokens: - groups: - system:bootstrappers:kubeadm:default-node-token token: abcdef.0123456789abcdef ttl: 24h0m0s usages: - signing - authentication kind: InitConfiguration localAPIEndpoint: advertiseAddress: 192.168.47.102 bindPort: 6443 nodeRegistration: criSocket: unix:///var/run/containerd/containerd.sock imagePullPolicy: IfNotPresent name: master taints: null --- apiServer: timeoutForControlPlane: 4m0s apiVersion: kubeadm.k8s.io/v1beta3 certificatesDir: /etc/kubernetes/pki clusterName: kubernetes controllerManager: {} dns: {} etcd: local: dataDir: /var/lib/etcd imageRepository: registry -hangzhou.aliyuncs /google_containers kind: ClusterConfiguration kubernetesVersion: 1.27.0 networking: dnsDomain: cluster.local serviceSubnet: 10.96.0.0/12 podSubnet: 10.244.0.0/16 scheduler: {} # 初始化集群 [root@admin01 ~]# kubeadm init --config=kubeadm-init.yaml [init] Using Kubernetes version: v1.27.0 [preflight] Running pre-flight checks [preflight] Pulling images required for setting up a Kubernetes cluster [preflight] This might take a minute or two, depending on the speed of your internet connection [preflight] You can also perform this action in beforehand using 'kubeadm config images pull' W0216 21:15:32.473341 59711 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0) W0216 21:15:32.761039 59711 checks.go:835] detected that the sandbox image "registry.aliyuncs /google_containers/pause:3.9" of the container runtime is inconsistent with that used by kubeadm. It is recommended that using "registry.k8s.io/pause:3.9" as the CRI sandbox image. [certs] Using certificateDir folder "/etc/kubernetes/pki" [certs] Generating "ca" certificate and key [certs] Generating "apiserver" certificate and key [certs] apiserver serving cert is signed for DNS names [kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local master] and IPs [10.96.0.1 192.168.47.102] [certs] Generating "apiserver-kubelet-client" certificate and key [certs] Generating "front-proxy-ca" certificate and key [certs] Generating "front-proxy-client" certificate and key [certs] Generating "etcd/ca" certificate and key [certs] Generating "etcd/server" certificate and key [certs] etcd/server serving cert is signed for DNS names [localhost master] and IPs [192.168.47.102 127.0.0.1 ::1] [certs] Generating "etcd/peer" certificate and key [certs] etcd/peer serving cert is signed for DNS names [localhost master] and IPs [192.168.47.102 127.0.0.1 ::1] [certs] Generating "etcd/healthcheck-client" certificate and key [certs] Generating "apiserver-etcd-client" certificate and key [certs] Generating "sa" key and public key [kubeconfig] Using kubeconfig folder "/etc/kubernetes" [kubeconfig] Writing "admin.conf" kubeconfig file [kubeconfig] Writing "kubelet.conf" kubeconfig file [kubeconfig] Writing "controller-manager.conf" kubeconfig file [kubeconfig] Writing "scheduler.conf" kubeconfig file [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Starting the kubelet [control-plane] Using manifest folder "/etc/kubernetes/manifests" [control-plane] Creating static Pod manifest for "kube-apiserver" [control-plane] Creating static Pod manifest for "kube-controller-manager" [control-plane] Creating static Pod manifest for "kube-scheduler" [etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests" W0216 21:15:38.631109 59711 images.go:80] could not find officially supported version of etcd for Kubernetes v1.27.0, falling back to the nearest etcd version (3.5.7-0) [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [apiclient] All control plane components are healthy after 10.513140 seconds [upload-config] Storing the configuration used in ConfigMap "kubeadm-config" in the "kube-system" Namespace [kubelet] Creating a ConfigMap "kubelet-config" in namespace kube-system with the configuration for the kubelets in the cluster [upload-certs] Skipping phase. Please see --upload-certs [mark-control-plane] Marking the node master as control-plane by adding the labels: [node-role.kubernetes.io/control-plane node.kubernetes.io/exclude-from-external-load-balancers] [mark-control-plane] Marking the node master as control-plane by adding the taints [node-role.kubernetes.io/control-plane:NoSchedule] [bootstrap-token] Using token: abcdef.0123456789abcdef [bootstrap-token] Configuring bootstrap tokens, cluster-info ConfigMap, RBAC Roles [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to get nodes [bootstrap-token] Configured RBAC rules to allow Node Bootstrap tokens to post CSRs in order for nodes to get long term certificate credentials [bootstrap-token] Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token [bootstrap-token] Configured RBAC rules to allow certificate rotation for all node client certificates in the cluster [bootstrap-token] Creating the "cluster-info" ConfigMap in the "kube-public" namespace [kubelet-finalize] Updating "/etc/kubernetes/kubelet.conf" to point to a rotatable kubelet client certificate and key [addons] Applied essential addon: CoreDNS [addons] Applied essential addon: kube-proxy Your Kubernetes control-plane has initialized successfully! To start using your cluster, you need to run the following as a regular user: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config sudo chown $(id -u):$(id -g) $HOME/.kube/config Alternatively, if you are the root user, you can run: export KUBECONFIG=/etc/kubernetes/admin.conf You should now deploy a pod network to the cluster. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at: kubernetes.io/docs/concepts/cluster-administration/addons/ Then you can join any number of worker nodes by running the following on each as root: kubeadm join 192.168.47.102:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:0d2e1f77f8531ecfbccdace3dd0f78507fe31c6ce1da02cecf5f35788a8ec613 # 初始化成功以后要根据提示执行以下3个命令，才可以操作集群 [root@master ~]# mkdir -p $HOME/.kube [root@master ~]# sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config [root@master ~]# sudo chown $(id -u):$(id -g) $HOME/.kube/config

待添加的节点执行如下命令即可：

kubeadm join 192.168.47.102:6443 --token abcdef.0123456789abcdef \ --discovery-token-ca-cert-hash sha256:0d2e1f77f8531ecfbccdace3dd0f78507fe31c6ce1da02cecf5f35788a8ec613 3.6、Node节点加入集群Node节点都要操作

在待添加的worker节点上执行

# worker01 节点 [root@localhost containerd]# kubeadm join 192.168.47.102:6443 --token abcdef.0123456789abcdef \ > --discovery-token-ca-cert-hash sha256:0d2e1f77f8531ecfbccdace3dd0f78507fe31c6ce1da02cecf5f35788a8ec613 [preflight] Running pre-flight checks [preflight] Reading configuration from the cluster... [preflight] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml' [kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml" [kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env" [kubelet-start] Starting the kubelet [kubelet-start] Waiting for the kubelet to perform the TLS Bootstrap... This node has joined the cluster: * Certificate signing request was sent to apiserver and a response was received. * The Kubelet was informed of the new secure connection details. Run 'kubectl get nodes' on the control-plane to see this node join the cluster.

集群当前状态 需要对notready的节点安装部署网络插件即可。

四、部署网络插件（CNI） 4.1、拉取必要镜像 所有节点都需要拉取以下两个镜像在部署网络插件的过程中，实测是可能会有两个镜像拉取不来下，所以手动拉取一下，不过要使用魔法，拉取不到了评论或私信（开源免费） # 以下是拉取镜像的命令，所有节点都需要有这两个镜像 crictl pull docker.io/flannel/flannel-cni-plugin:v1.1.2 crictl pull docker.io/flannel/flannel:v0.21.5

这里如果拉取失败，我们可以使用在有dokcer环境的机器上先将镜像拉下来，然后传到当前机器上进行解压后执行后续的相关操作。

[root@localhost ~]# ctr -n k8s.io images import flannel.tar unpacking docker.io/flannel/flannel:v0.21.5 (sha256:6f2d991efb758c5530e7de90761dfb29637b7604a807d431312e20189e09f9e6)...done [root@localhost ~]# ctr -n k8s.io images import flannel-cni-plugin.tar unpacking docker.io/flannel/flannel-cni-plugin:v1.1.2 (sha256:539d3bf046c8581557f0747dbad9d3b78a4de112d3c0bf9d291651593060fc9f)...done

4.2、部署网络插件 master节点操作即可 #下载flannel.yaml文件 wget raw.githubusercontent /coreos/flannel/a70459be0084506e4ec919aa1c114638878db11b/Documentation/kube-flannel.yml [root@master ~]# kubectl apply -f kube-flannel.yaml namespace/kube-flannel created clusterrole.rbac.authorization.k8s.io/flannel created clusterrolebinding.rbac.authorization.k8s.io/flannel created serviceaccount/flannel created configmap/kube-flannel-cfg created daemonset.apps/kube-flannel-ds created 4.2、查看节点状态 [root@admin01 ~]# kubectl get nodes NAME STATUS ROLES AGE VERSION master Ready control-plane 20m v1.27.0 worker01 Ready <none> 18m v1.27.0

4.3、查看组件状态 [root@admin01 ~]# kubectl get cs Warning: v1 ComponentStatus is deprecated in v1.19+ NAME STATUS MESSAGE ERROR etcd-0 Healthy {"health":"true","reason":""} controller-manager Healthy ok scheduler Healthy ok

4.4、查看所有Pod状态 [root@admin01 ~]# kubectl get pod -A NAMESPACE NAME READY STATUS RESTARTS AGE kube-flannel kube-flannel-ds-bgnfw 1/1 Running 0 6m26s kube-flannel kube-flannel-ds-bmdr8 1/1 Running 0 2m36s kube-system coredns-5d78c9869d-9vmpk 1/1 Running 0 21m kube-system coredns-5d78c9869d-wzb8q 1/1 Running 0 21m kube-system etcd-master 1/1 Running 0 21m kube-system kube-apiserver-master 1/1 Running 0 21m kube-system kube-controller-manager-master 1/1 Running 0 6m56s kube-system kube-proxy-shnbq 1/1 Running 0 21m kube-system kube-proxy-trzth 1/1 Running 0 19m kube-system kube-scheduler-master 1/1 Running 0 21m

4.5 解决异常pod状态

查看异常pod的描述状态发现 部署flannel网络插件时发现flannel一直处于CrashLoopBackOff状态，查看日志提示没有分配cidr 解决 vim /etc/kubernetes/manifests/kube-controller-manager.yaml 增加参数：

--allocate-node-cidrs=true --cluster-cidr=10.244.0.0/16

systemctl restart kubelet

五、Containerd基本操作 5.1、Crictl镜像管理 # 下载镜像 [root@master ~]# crictl pull nginx:latest # 查看所有镜像 [root@master ~]# crictl images # 查看镜像详细信息 [root@master ~]# crictl inspecti nginx:latest # 删除镜像 [root@master ~]# crictl rmi nginx:latest 5.2、Crictl容器管理 # 列出运行中的容器，查看所有容器加-a选项 [root@master ~]# crictl ps # 查看容器的详细信息 [root@master ~]# crictl inspect 07376dc402441 # 启动容器 [root@master ~]# crictl start 07376dc402441 # 停止容器 [root@master ~]# crictl stop 07376dc402441 # 删除容器 [root@master ~]# crictl rm 07376dc402441 # 打印日志 crictl logs <container-id 或 container-name> # 进入容器执行命令 crictl exec -it <container-id 或 container-name> sh 5.3、Crictl Pod管理 # 打印所有 Pod [root@node2 ~]# crictl pods # 根据pod名称打印pod信息 [root@master ~]# crictl pods --name kube-proxy-trnjn # 根据标签打印 Pod，crictl pods --label key=value [root@master ~]# crictl pods --label k8s-app=kube-proxy