一文搞定 K8s 版本升级1.30.2 升级 1.30.4含节点驱逐、故障恢复方案Kubernetes 集群升级升级必要性升级注意事项升级流程升级步骤升级前准备升级 master升级 kubeadm升级 master 节点组件升级其他 master 节点驱逐 master 节点升级 kubelet 和 kubectl取消节点不可调度升级 worker升级 worker31升级 worker32从故障状态恢复Kubernetes 集群升级摘要本文详细介绍了 Kubernetes 集群从 1.30.2 升级到 1.30.4 的完整操作流程。首先阐述了升级的必要性和注意事项然后按照先 master 节点后 worker 节点的顺序逐步演示了 kubeadm、控制平面组件、kubelet 和 kubectl 的升级步骤。文章包含大量实际操作命令和输出示例涵盖了升级前环境检查、升级计划、组件升级、节点驱逐与恢复等关键环节最后还提供了从故障状态恢复的方法。整个升级过程遵循官方推荐的最佳实践确保集群在升级期间的稳定性和数据安全。学习参考kubernetes 升级升级必要性功能性增强。漏洞修复。等等升级注意事项如果 Kubernetes 集群只有一个master节点则升级 master 节点过程中将导致集群不可用。如果 Kubernetes 集群只有两个worker节点则升级第二个 worker 节点时负载会在第一个worker节点重建进而导致应用短暂不可用不可用时间取决于pod重建时间。升级流程从节点角度看先逐个升级 master 节点再逐个升级 worker 节点。从软件层面看任何节点都需要先升级kubeadm再利用 kubeadm 升级各个组件最后升级kubelet和kubectl等。升级步骤本次实验环境从kubernetes 1.30.2 升级到 1.30.4。升级前准备# 确认当前安装的版本rootmaster30:~# kubectl versionClient Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.2 rootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 53d v1.30.2 worker31.lz.cloud Readynone53d v1.30.2 worker32.lz.cloud Readynone53d v1.30.2# 确认仓库中提供的版本rootmaster30:~# apt list kubeadm -a|grep amd64WARNING:aptdoes not have a stable CLI interface. Use with cautioninscripts. kubeadm/unknown1.30.4-1.1 amd64[upgradable from:1.30.2-1.1]kubeadm/unknown1.30.3-1.1 amd64 kubeadm/unknown,now1.30.2-1.1 amd64[installed,upgradable to:1.30.4-1.1]kubeadm/unknown1.30.1-1.1 amd64 kubeadm/unknown1.30.0-1.1 amd64升级 master升级 kubeadm# 升级 kubeadmrootmaster30:~# apt install -y kubeadm1.30.4-1.1如果 kubeadm 被hold住了使用以下命令解除。# 升级 kubeadmrootmaster30:~# apt-mark unhold kubeadm升级 master 节点组件# 升级帮助rootmaster30:~# kubeadm upgrade apply --help# --dry-run 模拟升级# --etcd-upgrade 升级 etcd默认值true。设置为 false 则不升级etcd。# -f, --force 强制升级即使部分条件不满足。使用该选项还可以避免二次确认升级。# 升级计划rootmaster30:~# kubeadm upgrade plan[preflight]Running pre-flight checks.[upgrade/config]Reading configuration from the cluster...[upgrade/config]FYI: You canlookat this configfilewithkubectl -n kube-system get cm kubeadm-config -o yaml[upgrade]Running cluster health checks[upgrade/health]FATAL:[preflight]Some fatal errors occurred:[ERROR CreateJob]: Jobupgrade-health-check-tk52jinthe namespacekube-systemdid not completein15s: no condition oftypeComplete[preflight]If you know what you are doing, you canmakea check non-fatal with--ignore-preflight-errors...To see the stack trace of this error execute with--v5or higherrootmaster30:~# kubeadm upgrade plan[preflight]Running pre-flight checks.[upgrade/config]Reading configuration from the cluster...[upgrade/config]FYI: You canlookat this configfilewithkubectl -n kube-system get cm kubeadm-config -o yaml[upgrade]Running cluster health checks[WARNING CreateJob]: Jobupgrade-health-check-c45ncinthe namespacekube-systemdid not completein15s: no condition oftypeComplete[upgrade]Fetching available versions to upgrade to[upgrade/versions]Cluster version:1.30.2[upgrade/versions]kubeadm version: v1.30.4 W0905 09:07:28.91866563943version.go:104]could not fetch a Kubernetes version from the internet: unable to get URLhttps://dl.k8s.io/release/stable.txt:Gethttps://cdn.dl.k8s.io/release/stable.txt:context deadline exceeded(Client.Timeout exceededwhileawaiting headers)W0905 09:07:28.91900463943version.go:105]falling back to thelocalclient version: v1.30.4[upgrade/versions]Target version: v1.30.4[upgrade/versions]Latest versioninthe v1.30 series: v1.30.4 Components that must be upgraded manually after you have upgraded the control plane withkubeadm upgrade apply:COMPONENT NODE CURRENT TARGET kubelet master30.lz.cloud v1.30.2 v1.30.4 kubelet worker31.lz.cloud v1.30.2 v1.30.4 kubelet worker32.lz.cloud v1.30.2 v1.30.4 Upgrade to the latest versioninthe v1.30 series: COMPONENT NODE CURRENT TARGET kube-apiserver master30.lz.cloud v1.30.2 v1.30.4 kube-controller-manager master30.lz.cloud v1.30.2 v1.30.4 kube-scheduler master30.lz.cloud v1.30.2 v1.30.4 kube-proxy1.30.2 v1.30.4 CoreDNS v1.11.1 v1.11.1 etcd master30.lz.cloud3.5.12-03.5.12-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.30.4 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have ayesmarkintheMANUAL UPGRADE REQUIREDcolumnrequire manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denotedinthePREFERRED VERSIONcolumn. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________升级 master 组件rootmaster30:~# kubeadm upgrade apply v1.30.4 --force[preflight]Running pre-flight checks.[upgrade/config]Reading configuration from the cluster...[upgrade/config]FYI: You canlookat this configfilewithkubectl -n kube-system get cm kubeadm-config -o yaml[upgrade]Running cluster health checks[WARNING CreateJob]: Jobupgrade-health-check-wpclkinthe namespacekube-systemdid not completein15s: no condition oftypeComplete[upgrade/version]You have chosen to change the cluster version tov1.30.4[upgrade/versions]Cluster version: v1.30.2[upgrade/versions]kubeadm version: v1.30.4[upgrade/prepull]Pulling images requiredforsetting up a Kubernetes cluster[upgrade/prepull]This might take a minute or two, depending on the speed of your internet connection[upgrade/prepull]You can also perform this actioninbeforehand usingkubeadm config images pullW0905 09:12:29.40698766867checks.go:844]detected that the sandbox imageregistry.aliyuncs.com/google_containers/pause:3.9of the container runtime is inconsistent with that used by kubeadm.It is recommended to useregistry.k8s.io/pause:3.9as the CRI sandbox image.[upgrade/apply]Upgrading your Static Pod-hosted control plane to versionv1.30.4(timeout: 5m0s)...[upgrade/etcd]Upgrading to TLSforetcd[upgrade/staticpods]Preparingforetcdupgrade[upgrade/staticpods]Current and new manifests of etcd are equal, skipping upgrade[upgrade/etcd]Waitingforetcd to become available[upgrade/staticpods]Writing new Static Pod manifests to/etc/kubernetes/tmp/kubeadm-upgraded-manifests3246055680[upgrade/staticpods]Preparingforkube-apiserverupgrade[upgrade/staticpods]Renewing apiserver certificate[upgrade/staticpods]Renewing apiserver-kubelet-client certificate[upgrade/staticpods]Renewing front-proxy-client certificate[upgrade/staticpods]Renewing apiserver-etcd-client certificate[upgrade/staticpods]Moved new manifest to/etc/kubernetes/manifests/kube-apiserver.yamland backed up old manifest to/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-09-05-09-12-29/kube-apiserver.yaml[upgrade/staticpods]Waitingforthe kubelet to restart the component[upgrade/staticpods]This can take up to 5m0s[apiclient]Found1Podsforlabel selectorcomponentkube-apiserver[upgrade/staticpods]Componentkube-apiserverupgraded successfully![upgrade/staticpods]Preparingforkube-controller-managerupgrade[upgrade/staticpods]Renewing controller-manager.conf certificate[upgrade/staticpods]Moved new manifest to/etc/kubernetes/manifests/kube-controller-manager.yamland backed up old manifest to/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-09-05-09-12-29/kube-controller-manager.yaml[upgrade/staticpods]Waitingforthe kubelet to restart the component[upgrade/staticpods]This can take up to 5m0s[apiclient]Found1Podsforlabel selectorcomponentkube-controller-manager[upgrade/staticpods]Componentkube-controller-managerupgraded successfully![upgrade/staticpods]Preparingforkube-schedulerupgrade[upgrade/staticpods]Renewing scheduler.conf certificate[upgrade/staticpods]Moved new manifest to/etc/kubernetes/manifests/kube-scheduler.yamland backed up old manifest to/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-09-05-09-12-29/kube-scheduler.yaml[upgrade/staticpods]Waitingforthe kubelet to restart the component[upgrade/staticpods]This can take up to 5m0s[apiclient]Found1Podsforlabel selectorcomponentkube-scheduler[upgrade/staticpods]Componentkube-schedulerupgraded successfully![upload-config]Storing the configuration usedinConfigMapkubeadm-configinthekube-systemNamespace[kubelet]Creating a ConfigMapkubelet-configinnamespace kube-system with the configurationforthe kubeletsinthe cluster[upgrade]Backing up kubelet configfileto /etc/kubernetes/tmp/kubeadm-kubelet-config1570193532/config.yaml[kubelet-start]Writing kubelet configuration tofile/var/lib/kubelet/config.yaml[bootstrap-token]Configured RBAC rules to allow Node Bootstrap tokens to get nodes[bootstrap-token]Configured RBAC rules to allow Node Bootstrap tokens to post CSRsinorderfornodes to get long term certificate credentials[bootstrap-token]Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstrap-token]Configured RBAC rules to allow certificate rotationforallnodeclient certificatesinthe cluster[addons]Applied essential addon: CoreDNS[addons]Applied essential addon: kube-proxy[upgrade/successful]SUCCESS!Your cluster was upgraded tov1.30.4.Enjoy![upgrade/kubelet]Now that your control plane is upgraded, please proceed with upgrading your kubeletsifyou havent alreadydoneso.kubeadm upgrade apply做了以下工作检查你的集群是否处于可升级状态:API 服务器是可访问的所有节点处于Ready状态控制面是健康的强制执行版本偏差策略。确保控制面的镜像是可用的或可拉取到服务器上。如果组件配置要求版本升级则生成替代配置与/或使用用户提供的覆盖版本配置。升级控制面组件或回滚如果其中任何一个组件无法启动。应用新的CoreDNS和kube-proxy清单并强制创建所有必需的 RBAC 规则。如果旧文件在 180 天后过期将创建 API 服务器的新证书和密钥文件并备份旧文件。升级其他 master 节点rootmaster30:~# kubeadm upgrade nodekubeadm upgrade node在其他控制平节点上执行以下操作从集群中获取 kubeadmClusterConfiguration。可选操作备份 kube-apiserver 证书。升级控制平面组件的静态 Pod 清单。为本节点升级 kubelet 配置驱逐 master 节点# 驱逐 master节点并设置为不可调度rootmaster30:~# kubectl drain master30.lz.cloud --ignore-daemonsetsrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready,SchedulingDisabled control-plane 42h v1.30.2 worker31.lz.cloud Readynone42h v1.30.2 worker32.lz.cloud Readynone42h v1.30.2升级 kubelet 和 kubectl# 升级 kubectl和kubeletrootmaster30:~# apt install -y kubelet1.30.4-1.1 kubectl1.30.4-1.1# 重启 kubelet 服务rootmaster30:~# systemctl restart kubelet.service取消节点不可调度# 设置master节点可调度rootmaster30:~# kubectl uncordon master30.lz.cloud# 验证结果rootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 48m v1.30.4 worker31.lz.cloud Readynone37m v1.30.2 worker32.lz.cloud Readynone37m v1.30.2升级 worker升级 worker31# 升级kubeadmrootworker31:~# apt install -y kubeadm1.30.4-1.1# 升级 worker节点组件rootworker31:~# kubeadm upgrade node# 以上命令在工作节点上完成以下工作# 1.从集群取回 kubeadm ClusterConfiguration。# 2.为本节点升级 kubelet 配置。# 驱逐节点上 pod并取消节点不可调度rootmaster30:~# kubectl drain worker31.lz.cloud --ignore-daemonsetsrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 50m v1.30.4 worker31.lz.cloud Ready,SchedulingDisablednone39m v1.30.2 worker32.lz.cloud Readynone39m v1.30.2# 升级 kubectl 和 kubeletrootworker31:~# apt install -y kubelet1.30.4-1.1 kubectl1.30.4-1.1# 重启kubelet服务rootworker31:~# systemctl restart kubelet.service# 取消节点不可调度rootmaster30:~# kubectl uncordon worker31.lz.cloudrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 51m v1.30.4 worker31.lz.cloud Readynone40m v1.30.4 worker32.lz.cloud Readynone40m v1.30.2升级 worker32# 升级 kubeadmrootworker32:~# apt install -y kubeadm1.30.4-1.1# 升级 worker 节点组件rootworker32:~# kubeadm upgrade node# 驱逐节点上pod并取消节点不可调度rootmaster30:~# kubectl drain worker32.lz.cloud --ignore-daemonsetsrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 52m v1.30.4 worker31.lz.cloud Readynone40m v1.30.4 worker32.lz.cloud Ready,SchedulingDisablednone40m v1.30.2# 升级 kubectl 和 kubeletrootworker32:~# apt install -y kubelet1.30.4-1.1 kubectl1.30.4-1.1# 重启 kubelet 服务rootworker32:~# systemctl restart kubelet.service# 取消节点不可调度rootmaster30:~# kubectl uncordon worker32.lz.cloudrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 53d v1.30.4 worker31.lz.cloud Readynone53d v1.30.4 worker32.lz.cloud Readynone53d v1.30.4# 确认集群版本rootmaster30:~# kubectl versionClient Version: v1.30.4 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.4从故障状态恢复如果kubeadm upgrade失败并且没有回滚例如由于执行期间节点意外关闭 你可以再次运行kubeadm upgrade。 此命令是幂等的并最终确保实际状态是你声明的期望状态。要从故障状态恢复你还可以运行kubeadm upgrade apply --force而无需更改集群正在运行的版本。在升级期间kubeadm 向/etc/kubernetes/tmp目录下的如下备份文件夹写入数据kubeadm-backup-etcd-date-timekubeadm-backup-manifests-date-timekubeadm-backup-etcd包含当前控制面节点本地 etcd 成员数据的备份。 如果 etcd 升级失败并且自动回滚也无法修复则可以将此文件夹中的内容复制到/var/lib/etcd进行手工修复。如果使用的是外部的 etcd则此备份文件夹为空。kubeadm-backup-manifests包含当前控制面节点的静态 Pod 清单文件的备份版本。 如果升级失败并且无法自动回滚则此文件夹中的内容可以复制到/etc/kubernetes/manifests目录实现手工恢复。 如果由于某些原因在升级前后某个组件的清单未发生变化则 kubeadm 也不会为之生成备份版本。 一文搞定 K8s 版本升级1.30.2 升级 1.30.4含节点驱逐、故障恢复方案Kubernetes 集群升级升级必要性升级注意事项升级流程升级步骤升级前准备升级 master升级 kubeadm升级 master 节点组件升级其他 master 节点驱逐 master 节点升级 kubelet 和 kubectl取消节点不可调度升级 worker升级 worker31升级 worker32从故障状态恢复Kubernetes 集群升级摘要本文详细介绍了 Kubernetes 集群从 1.30.2 升级到 1.30.4 的完整操作流程。首先阐述了升级的必要性和注意事项然后按照先 master 节点后 worker 节点的顺序逐步演示了 kubeadm、控制平面组件、kubelet 和 kubectl 的升级步骤。文章包含大量实际操作命令和输出示例涵盖了升级前环境检查、升级计划、组件升级、节点驱逐与恢复等关键环节最后还提供了从故障状态恢复的方法。整个升级过程遵循官方推荐的最佳实践确保集群在升级期间的稳定性和数据安全。学习参考kubernetes 升级升级必要性功能性增强。漏洞修复。等等升级注意事项如果 Kubernetes 集群只有一个master节点则升级 master 节点过程中将导致集群不可用。如果 Kubernetes 集群只有两个worker节点则升级第二个 worker 节点时负载会在第一个worker节点重建进而导致应用短暂不可用不可用时间取决于pod重建时间。升级流程从节点角度看先逐个升级 master 节点再逐个升级 worker 节点。从软件层面看任何节点都需要先升级kubeadm再利用 kubeadm 升级各个组件最后升级kubelet和kubectl等。升级步骤本次实验环境从kubernetes 1.30.2 升级到 1.30.4。升级前准备# 确认当前安装的版本rootmaster30:~# kubectl versionClient Version: v1.30.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.2 rootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 53d v1.30.2 worker31.lz.cloud Readynone53d v1.30.2 worker32.lz.cloud Readynone53d v1.30.2# 确认仓库中提供的版本rootmaster30:~# apt list kubeadm -a|grep amd64WARNING:aptdoes not have a stable CLI interface. Use with cautioninscripts. kubeadm/unknown1.30.4-1.1 amd64[upgradable from:1.30.2-1.1]kubeadm/unknown1.30.3-1.1 amd64 kubeadm/unknown,now1.30.2-1.1 amd64[installed,upgradable to:1.30.4-1.1]kubeadm/unknown1.30.1-1.1 amd64 kubeadm/unknown1.30.0-1.1 amd64升级 master升级 kubeadm# 升级 kubeadmrootmaster30:~# apt install -y kubeadm1.30.4-1.1如果 kubeadm 被hold住了使用以下命令解除。# 升级 kubeadmrootmaster30:~# apt-mark unhold kubeadm升级 master 节点组件# 升级帮助rootmaster30:~# kubeadm upgrade apply --help# --dry-run 模拟升级# --etcd-upgrade 升级 etcd默认值true。设置为 false 则不升级etcd。# -f, --force 强制升级即使部分条件不满足。使用该选项还可以避免二次确认升级。# 升级计划rootmaster30:~# kubeadm upgrade plan[preflight]Running pre-flight checks.[upgrade/config]Reading configuration from the cluster...[upgrade/config]FYI: You canlookat this configfilewithkubectl -n kube-system get cm kubeadm-config -o yaml[upgrade]Running cluster health checks[upgrade/health]FATAL:[preflight]Some fatal errors occurred:[ERROR CreateJob]: Jobupgrade-health-check-tk52jinthe namespacekube-systemdid not completein15s: no condition oftypeComplete[preflight]If you know what you are doing, you canmakea check non-fatal with--ignore-preflight-errors...To see the stack trace of this error execute with--v5or higherrootmaster30:~# kubeadm upgrade plan[preflight]Running pre-flight checks.[upgrade/config]Reading configuration from the cluster...[upgrade/config]FYI: You canlookat this configfilewithkubectl -n kube-system get cm kubeadm-config -o yaml[upgrade]Running cluster health checks[WARNING CreateJob]: Jobupgrade-health-check-c45ncinthe namespacekube-systemdid not completein15s: no condition oftypeComplete[upgrade]Fetching available versions to upgrade to[upgrade/versions]Cluster version:1.30.2[upgrade/versions]kubeadm version: v1.30.4 W0905 09:07:28.91866563943version.go:104]could not fetch a Kubernetes version from the internet: unable to get URLhttps://dl.k8s.io/release/stable.txt:Gethttps://cdn.dl.k8s.io/release/stable.txt:context deadline exceeded(Client.Timeout exceededwhileawaiting headers)W0905 09:07:28.91900463943version.go:105]falling back to thelocalclient version: v1.30.4[upgrade/versions]Target version: v1.30.4[upgrade/versions]Latest versioninthe v1.30 series: v1.30.4 Components that must be upgraded manually after you have upgraded the control plane withkubeadm upgrade apply:COMPONENT NODE CURRENT TARGET kubelet master30.lz.cloud v1.30.2 v1.30.4 kubelet worker31.lz.cloud v1.30.2 v1.30.4 kubelet worker32.lz.cloud v1.30.2 v1.30.4 Upgrade to the latest versioninthe v1.30 series: COMPONENT NODE CURRENT TARGET kube-apiserver master30.lz.cloud v1.30.2 v1.30.4 kube-controller-manager master30.lz.cloud v1.30.2 v1.30.4 kube-scheduler master30.lz.cloud v1.30.2 v1.30.4 kube-proxy1.30.2 v1.30.4 CoreDNS v1.11.1 v1.11.1 etcd master30.lz.cloud3.5.12-03.5.12-0 You can now apply the upgrade by executing the following command: kubeadm upgrade apply v1.30.4 _____________________________________________________________________ The table below shows the current state of component configs as understood by this version of kubeadm. Configs that have ayesmarkintheMANUAL UPGRADE REQUIREDcolumnrequire manual config upgrade or resetting to kubeadm defaults before a successful upgrade can be performed. The version to manually upgrade to is denotedinthePREFERRED VERSIONcolumn. API GROUP CURRENT VERSION PREFERRED VERSION MANUAL UPGRADE REQUIRED kubeproxy.config.k8s.io v1alpha1 v1alpha1 no kubelet.config.k8s.io v1beta1 v1beta1 no _____________________________________________________________________升级 master 组件rootmaster30:~# kubeadm upgrade apply v1.30.4 --force[preflight]Running pre-flight checks.[upgrade/config]Reading configuration from the cluster...[upgrade/config]FYI: You canlookat this configfilewithkubectl -n kube-system get cm kubeadm-config -o yaml[upgrade]Running cluster health checks[WARNING CreateJob]: Jobupgrade-health-check-wpclkinthe namespacekube-systemdid not completein15s: no condition oftypeComplete[upgrade/version]You have chosen to change the cluster version tov1.30.4[upgrade/versions]Cluster version: v1.30.2[upgrade/versions]kubeadm version: v1.30.4[upgrade/prepull]Pulling images requiredforsetting up a Kubernetes cluster[upgrade/prepull]This might take a minute or two, depending on the speed of your internet connection[upgrade/prepull]You can also perform this actioninbeforehand usingkubeadm config images pullW0905 09:12:29.40698766867checks.go:844]detected that the sandbox imageregistry.aliyuncs.com/google_containers/pause:3.9of the container runtime is inconsistent with that used by kubeadm.It is recommended to useregistry.k8s.io/pause:3.9as the CRI sandbox image.[upgrade/apply]Upgrading your Static Pod-hosted control plane to versionv1.30.4(timeout: 5m0s)...[upgrade/etcd]Upgrading to TLSforetcd[upgrade/staticpods]Preparingforetcdupgrade[upgrade/staticpods]Current and new manifests of etcd are equal, skipping upgrade[upgrade/etcd]Waitingforetcd to become available[upgrade/staticpods]Writing new Static Pod manifests to/etc/kubernetes/tmp/kubeadm-upgraded-manifests3246055680[upgrade/staticpods]Preparingforkube-apiserverupgrade[upgrade/staticpods]Renewing apiserver certificate[upgrade/staticpods]Renewing apiserver-kubelet-client certificate[upgrade/staticpods]Renewing front-proxy-client certificate[upgrade/staticpods]Renewing apiserver-etcd-client certificate[upgrade/staticpods]Moved new manifest to/etc/kubernetes/manifests/kube-apiserver.yamland backed up old manifest to/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-09-05-09-12-29/kube-apiserver.yaml[upgrade/staticpods]Waitingforthe kubelet to restart the component[upgrade/staticpods]This can take up to 5m0s[apiclient]Found1Podsforlabel selectorcomponentkube-apiserver[upgrade/staticpods]Componentkube-apiserverupgraded successfully![upgrade/staticpods]Preparingforkube-controller-managerupgrade[upgrade/staticpods]Renewing controller-manager.conf certificate[upgrade/staticpods]Moved new manifest to/etc/kubernetes/manifests/kube-controller-manager.yamland backed up old manifest to/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-09-05-09-12-29/kube-controller-manager.yaml[upgrade/staticpods]Waitingforthe kubelet to restart the component[upgrade/staticpods]This can take up to 5m0s[apiclient]Found1Podsforlabel selectorcomponentkube-controller-manager[upgrade/staticpods]Componentkube-controller-managerupgraded successfully![upgrade/staticpods]Preparingforkube-schedulerupgrade[upgrade/staticpods]Renewing scheduler.conf certificate[upgrade/staticpods]Moved new manifest to/etc/kubernetes/manifests/kube-scheduler.yamland backed up old manifest to/etc/kubernetes/tmp/kubeadm-backup-manifests-2024-09-05-09-12-29/kube-scheduler.yaml[upgrade/staticpods]Waitingforthe kubelet to restart the component[upgrade/staticpods]This can take up to 5m0s[apiclient]Found1Podsforlabel selectorcomponentkube-scheduler[upgrade/staticpods]Componentkube-schedulerupgraded successfully![upload-config]Storing the configuration usedinConfigMapkubeadm-configinthekube-systemNamespace[kubelet]Creating a ConfigMapkubelet-configinnamespace kube-system with the configurationforthe kubeletsinthe cluster[upgrade]Backing up kubelet configfileto /etc/kubernetes/tmp/kubeadm-kubelet-config1570193532/config.yaml[kubelet-start]Writing kubelet configuration tofile/var/lib/kubelet/config.yaml[bootstrap-token]Configured RBAC rules to allow Node Bootstrap tokens to get nodes[bootstrap-token]Configured RBAC rules to allow Node Bootstrap tokens to post CSRsinorderfornodes to get long term certificate credentials[bootstrap-token]Configured RBAC rules to allow the csrapprover controller automatically approve CSRs from a Node Bootstrap Token[bootstrap-token]Configured RBAC rules to allow certificate rotationforallnodeclient certificatesinthe cluster[addons]Applied essential addon: CoreDNS[addons]Applied essential addon: kube-proxy[upgrade/successful]SUCCESS!Your cluster was upgraded tov1.30.4.Enjoy![upgrade/kubelet]Now that your control plane is upgraded, please proceed with upgrading your kubeletsifyou havent alreadydoneso.kubeadm upgrade apply做了以下工作检查你的集群是否处于可升级状态:API 服务器是可访问的所有节点处于Ready状态控制面是健康的强制执行版本偏差策略。确保控制面的镜像是可用的或可拉取到服务器上。如果组件配置要求版本升级则生成替代配置与/或使用用户提供的覆盖版本配置。升级控制面组件或回滚如果其中任何一个组件无法启动。应用新的CoreDNS和kube-proxy清单并强制创建所有必需的 RBAC 规则。如果旧文件在 180 天后过期将创建 API 服务器的新证书和密钥文件并备份旧文件。升级其他 master 节点rootmaster30:~# kubeadm upgrade nodekubeadm upgrade node在其他控制平节点上执行以下操作从集群中获取 kubeadmClusterConfiguration。可选操作备份 kube-apiserver 证书。升级控制平面组件的静态 Pod 清单。为本节点升级 kubelet 配置驱逐 master 节点# 驱逐 master节点并设置为不可调度rootmaster30:~# kubectl drain master30.lz.cloud --ignore-daemonsetsrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready,SchedulingDisabled control-plane 42h v1.30.2 worker31.lz.cloud Readynone42h v1.30.2 worker32.lz.cloud Readynone42h v1.30.2升级 kubelet 和 kubectl# 升级 kubectl和kubeletrootmaster30:~# apt install -y kubelet1.30.4-1.1 kubectl1.30.4-1.1# 重启 kubelet 服务rootmaster30:~# systemctl restart kubelet.service取消节点不可调度# 设置master节点可调度rootmaster30:~# kubectl uncordon master30.lz.cloud# 验证结果rootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 48m v1.30.4 worker31.lz.cloud Readynone37m v1.30.2 worker32.lz.cloud Readynone37m v1.30.2升级 worker升级 worker31# 升级kubeadmrootworker31:~# apt install -y kubeadm1.30.4-1.1# 升级 worker节点组件rootworker31:~# kubeadm upgrade node# 以上命令在工作节点上完成以下工作# 1.从集群取回 kubeadm ClusterConfiguration。# 2.为本节点升级 kubelet 配置。# 驱逐节点上 pod并取消节点不可调度rootmaster30:~# kubectl drain worker31.lz.cloud --ignore-daemonsetsrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 50m v1.30.4 worker31.lz.cloud Ready,SchedulingDisablednone39m v1.30.2 worker32.lz.cloud Readynone39m v1.30.2# 升级 kubectl 和 kubeletrootworker31:~# apt install -y kubelet1.30.4-1.1 kubectl1.30.4-1.1# 重启kubelet服务rootworker31:~# systemctl restart kubelet.service# 取消节点不可调度rootmaster30:~# kubectl uncordon worker31.lz.cloudrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 51m v1.30.4 worker31.lz.cloud Readynone40m v1.30.4 worker32.lz.cloud Readynone40m v1.30.2升级 worker32# 升级 kubeadmrootworker32:~# apt install -y kubeadm1.30.4-1.1# 升级 worker 节点组件rootworker32:~# kubeadm upgrade node# 驱逐节点上pod并取消节点不可调度rootmaster30:~# kubectl drain worker32.lz.cloud --ignore-daemonsetsrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 52m v1.30.4 worker31.lz.cloud Readynone40m v1.30.4 worker32.lz.cloud Ready,SchedulingDisablednone40m v1.30.2# 升级 kubectl 和 kubeletrootworker32:~# apt install -y kubelet1.30.4-1.1 kubectl1.30.4-1.1# 重启 kubelet 服务rootworker32:~# systemctl restart kubelet.service# 取消节点不可调度rootmaster30:~# kubectl uncordon worker32.lz.cloudrootmaster30:~# kubectl get nodesNAME STATUS ROLES AGE VERSION master30.lz.cloud Ready control-plane 53d v1.30.4 worker31.lz.cloud Readynone53d v1.30.4 worker32.lz.cloud Readynone53d v1.30.4# 确认集群版本rootmaster30:~# kubectl versionClient Version: v1.30.4 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.30.4从故障状态恢复如果kubeadm upgrade失败并且没有回滚例如由于执行期间节点意外关闭 你可以再次运行kubeadm upgrade。 此命令是幂等的并最终确保实际状态是你声明的期望状态。要从故障状态恢复你还可以运行kubeadm upgrade apply --force而无需更改集群正在运行的版本。在升级期间kubeadm 向/etc/kubernetes/tmp目录下的如下备份文件夹写入数据kubeadm-backup-etcd-date-timekubeadm-backup-manifests-date-timekubeadm-backup-etcd包含当前控制面节点本地 etcd 成员数据的备份。 如果 etcd 升级失败并且自动回滚也无法修复则可以将此文件夹中的内容复制到/var/lib/etcd进行手工修复。如果使用的是外部的 etcd则此备份文件夹为空。kubeadm-backup-manifests包含当前控制面节点的静态 Pod 清单文件的备份版本。 如果升级失败并且无法自动回滚则此文件夹中的内容可以复制到/etc/kubernetes/manifests目录实现手工恢复。 如果由于某些原因在升级前后某个组件的清单未发生变化则 kubeadm 也不会为之生成备份版本。 
算法精解与面试技巧)
