Multus 多网卡方案：IPVLAN 模式

CNI容器网络接口是云原生计算基金会CNCF的一个项目它包含一套规范及开发库。通过 CNI 编写网络插件用于配置容器网络同时 CNI 本身也供了一系列现成的插件。CNI 仅专注于容器的网络连通性以及在容器被删除时清理已分配的网络资源。正是由于这种专注性它获得了广泛的支持和易于实现的规范。IPVLAN 就是 CNI 自带的网络插件之一它也对宿主机的网络接口进行了虚拟化所有 IPVLAN 设备共享同一个 MAC 地址。# 在 ns0 与 ns1 两个网络空间中,他们的网卡设备 ipvl0/ipvl1 ip 不一致,但 MAC 地址均与 host1 主机 eth0 相同 | Host: host1 | | | | ---------------------- ---------------------- | | | NS:ns0 | | NS:ns1 | | | | | | | | | | | | | | | | ipvl0 | | ipvl1 | | | ----------#----------- -----------#---------- | | # # | | ################################ | | # | | # | | # eth0 | Multus CNI 允许将多个网络接口连接到 K8s Pod 中。在安装他之前K8s 集群必须安装一个默认的 CNI 插件例如 Flannel、Calico 等。部署流程通过 Kind 快速生成集群并部署 Multus CNI Whereabouts IPAM主脚本#!/bin/bash set -v # 1.kind 创建集群时关闭默认 CNI 部署 cat EOF | HTTP_PROXY HTTPS_PROXY http_proxy https_proxy kind create cluster --namecni-multus --imagekindest/node:v1.27.3 --config- kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: disableDefaultCNI: true nodes: - role: control-plane - role: worker - role: worker EOF # 2.remove taints kubectl taint nodes $(kubectl get nodes -o name | grep control-plane) node-role.kubernetes.io/control-plane:NoSchedule- # 3. install CNI kubectl apply -f ./k8snetworkplumbingwgK8snetworkplumbingwg 目录内容Calico CNI部署 Calico v3.31.5 示例IPIP作为集群默认 CNI# Cluster type to identify the deployment type - name: CLUSTER_TYPE value: k8s,bgp # Auto-detect the BGP IP address. - name: IP value: autodetect # Enable IPIP - name: CALICO_IPV4POOL_IPIP value: Always # Enable or Disable VXLAN on the default IP pool. - name: CALICO_IPV4POOL_VXLAN value: Never # Enable or Disable VXLAN on the default IPv6 IP pool. - name: CALICO_IPV6POOL_VXLAN value: NeverMultus使用厚插件Thick形式部署 Multus。与薄插件Thin两种部署方式对比薄插件每次创建/删除 Pod 时直接调用 /opt/cni/bin/multus 二进制文件。每次调用都是一个独立进程日志分散在 kubelet 中不方便排查问题、且没有 Metrics 指标厚插件拆成了两个进程shim和daemon。日志集中在 daemon 中统一管理、且通过 daemon 暴露 Metrics 指标。Shim客户端k8s 调用 shim 后会通过绑定挂载到主机文件系统的 Unix 域套接字与 daemon 通信Daemon服务端负责执行所有繁重的拉取操作读取 CRD、计算配置、调用 delegate 插件等。┌─────────┐ ┌───────┐ ┌────────┐ ┌──────────┐ │ │ cni ADD/DEL │ │ REST POST │ │ cni ADD/DEL │ │ │ runtime ├────────────►│ shim ││ daemon ├────────────►│ delegate │ │ │------------│ │ │ │------------│ │ └─────────┘ └───────┘ └────────┘ └──────────┘按照上面引用的 GitHub 仓库地址中提供的部署方式部署即可。WhereaboutsMultus CNI 的 IPAM 地址管理插件。按照 GitHub 仓库中提供的一键部署方式部署即可查看部署结果rootnetwork-demo:~# kubectl get node -o wide NAME STATUS ROLES AGE VERSION INTERNAL-IP cni-multus-control-plane Ready control-plane 4m56s v1.27.3 172.18.0.4 cni-multus-worker Ready none 4m31s v1.27.3 172.18.0.3 cni-multus-worker2 Ready none 4m33s v1.27.3 172.18.0.2 rootnetwork-demo:~# kubectl get pods -A -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE kube-system calico-kube-controllers-547c488cf5 1/1 Running 0 4m31s 10.244.226.132 cni-multus-control-plane kube-system calico-node-5lgrw 1/1 Running 0 4m31s 172.18.0.4 cni-multus-control-plane kube-system calico-node-7fk72 1/1 Running 0 4m31s 172.18.0.3 cni-multus-worker kube-system calico-node-t67s5 1/1 Running 0 4m31s 172.18.0.2 cni-multus-worker2 kube-system coredns-5d78c9869d-p874p 1/1 Running 0 4m40s 10.244.226.131 cni-multus-control-plane kube-system coredns-5d78c9869d-z6bcj 1/1 Running 0 4m40s 10.244.226.130 cni-multus-control-plane kube-system etcd-cni-multus 1/1 Running 0 4m55s 172.18.0.4 cni-multus-control-plane kube-system kube-apiserver-cni-multus 1/1 Running 0 4m55s 172.18.0.4 cni-multus-control-plane kube-system kube-controller-manager-cni-multus 1/1 Running 0 4m56s 172.18.0.4 cni-multus-control-plane kube-system kube-multus-ds-6nvfz 1/1 Running 0 4m31s 172.18.0.2 cni-multus-worker2 kube-system kube-multus-ds-hn6gj 1/1 Running 0 4m31s 172.18.0.3 cni-multus-worker kube-system kube-multus-ds-r8qfk 1/1 Running 0 4m31s 172.18.0.4 cni-multus-control-plane kube-system kube-proxy-4vfdj 1/1 Running 0 4m35s 172.18.0.2 cni-multus-worker2 kube-system kube-proxy-cz8sl 1/1 Running 0 4m40s 172.18.0.4 cni-multus-control-plane kube-system kube-proxy-q9hv5 1/1 Running 0 4m33s 172.18.0.3 cni-multus-worker kube-system kube-scheduler-cni-multus 1/1 Running 0 4m55s 172.18.0.4 cni-multus-control-plane kube-system whereabouts-98fn6 1/1 Running 0 4m31s 172.18.0.3 cni-multus-worker kube-system whereabouts-tqnqk 1/1 Running 0 4m31s 172.18.0.4 cni-multus-control-plane kube-system whereabouts-xfbgj 1/1 Running 0 4m31s 172.18.0.2 cni-multus-worker2 rootnetwork-demo:~# kubectl get crd network-attachment-definitions.k8s.cni.cncf.io NAME CREATED AT network-attachment-definitions.k8s.cni.cncf.io 2026-05-23T00:58:25Z创建测试 Pod先通过 Multus CNI 使用 NetworkAttachmentDefinition CRD 创建 Pod 附加网络模板后在生成 Pod 时引用对应模板#!/bin/bash set -v controller_nodekubectl get nodes --no-headers -o custom-columnsNAME:.metadata.name| grep control-plane worker_nodekubectl get nodes --no-headers -o custom-columnsNAME:.metadata.name| grep worker2 cat EOF | kubectl apply -f - apiVersion: k8s.cni.cncf.io/v1 kind: NetworkAttachmentDefinition metadata: name: ipvlanl2-whereabouts-conf spec: config: { cniVersion: 0.3.0, name: whereaboutsexample, type: ipvlan, mode: l2, master: eth0, ipam: { type: whereabouts, range: 172.18.0.200-172.18.0.205/24 } } EOF cat EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: ipvlan-pod1 annotations: k8s.v1.cni.cncf.io/networks: ipvlanl2-whereabouts-confeth1 spec: containers: - name: nettool image: burlyluo/nettool:latest securityContext: privileged: false capabilities: add: [NET_ADMIN] nodeName: ${controller_node} EOF cat EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: ipvlan-pod-x annotations: k8s.v1.cni.cncf.io/networks: ipvlanl2-whereabouts-confeth1 spec: containers: - name: nettool image: burlyluo/nettool:latest securityContext: privileged: false capabilities: add: [NET_ADMIN] nodeName: ${controller_node} EOF cat EOF | kubectl apply -f - apiVersion: v1 kind: Pod metadata: name: ipvlan-pod2 annotations: k8s.v1.cni.cncf.io/networks: ipvlanl2-whereabouts-confeth1 spec: containers: - name: nettool image: burlyluo/nettool:latest securityContext: privileged: false capabilities: add: [NET_ADMIN] nodeName: ${worker_node} EOF问题解决创建测试 Pod 后可能会一直卡在ContainerCreating状态通过 describe pod 发现创建流程中的关键报错failed to find plugin ipvlan in path [/opt/cni/bin]这是因为 kind 创建 k8s 集群时的基础镜像中没有包含 IPVLAN 这个 CNI 插件。在主机上手动拉一下 CNI 包解压到几个 k8s node 容器中即可修复rootnetwork-demo:~# kubectl get pods -o wide NAME READY STATUS RESTARTS AGE IP NODE ipvlan-pod-x 0/1 ContainerCreating 0 103s none cni-multus-control-plane ipvlan-pod1 0/1 ContainerCreating 0 103s none cni-multus-control-plane ipvlan-pod2 0/1 ContainerCreating 0 102s none cni-multus-worker2 rootnetwork-demo:~# kubectl describe pods ipvlan-pod-x | grep -A4 Events Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal AddedInterface 110s multus Add eth0 [10.244.226.134/32] from k8s-pod-network Warning FailedCreatePodSandBox 109s kubelet Failed to create pod sandbox: rpc error: code Unknown desc failed to setup network for sandbox 31265d3618de74e470e1a39c0670a463566a99176cbb3c7774d56589c81ba3a2: plugin typemultus-shim namemultus-cni-network failed (add): CmdAdd (shim): CNI request failed with status 400: ContainerID:31265d3618de74e470e1a39c0670a463566a99176cbb3c7774d56589c81ba3a2 Netns:/var/run/netns/cni-6db962c0-2e51-298e-9cbd-8599f6f18e71 IfName:eth0 Args:K8S_POD_NAMEipvlan-pod-x;K8S_POD_INFRA_CONTAINER_ID31265d3618de74e470e1a39c0670a463566a99176cbb3c7774d56589c81ba3a2;K8S_POD_UIDe8872b13-c212-4b3b-9158-794adf3a8527;IgnoreUnknown1;K8S_POD_NAMESPACEdefault Path: ERRORED: error configuring pod [default/ipvlan-pod-x] networking: [default/ipvlan-pod-x/e8872b13-c212-4b3b-9158-794adf3a8527:whereaboutsexample]: error adding container to network whereaboutsexample: failed to find plugin ipvlan in path [/opt/cni/bin] : StdinData: {capabilities:{portMappings:true},clusterNetwork:/host/etc/cni/net.d/10-calico.conflist,cniVersion:0.3.1,logLevel:verbose,logToStderr:true,name:multus-cni-network,type:multus-shim}rootnetwork-demo:~# wget https://github.com/containernetworking/plugins/releases/download/v1.9.1/cni-plugins-linux-amd64-v1.9.1.tgz rootnetwork-demo:~# tar tf cni-plugins-linux-amd64-v1.9.1.tgz ./ ./sbr ./tap ./dhcp ./dummy ./bridge ./host-device ./README.md ./LICENSE ./static ./portmap