Kubernetes由Calico更换Flannel插件排错历程

原来使用的是Calico插件，想更换为Flannel，然后噩梦就开始了。。。

直接使用kubectl apply -f安装了flannel插件,在使用kubectl get pod -A后发现metrics-server和coredns一直处于ContainerCreating状态。

[root@master ~]# kubectl get pod -A -o wide
NAMESPACE      NAME                              READY   STATUS              RESTARTS   AGE    IP              NODE     NOMINATED NODE   READINESS GATES
kube-flannel   kube-flannel-ds-h2mmh             1/1     Running             0          52m    10.0.0.200      master   <none>           <none>
kube-system    coredns-6d8c4cb4d-mzpf8           0/1     ContainerCreating   0          140m   10.100.219.67   master   <none>           <none>
kube-system    coredns-6d8c4cb4d-pntnc           0/1     ContainerCreating   0          140m   10.100.219.66   master   <none>           <none>
kube-system    etcd-master                       1/1     Running             0          140m   10.0.0.200      master   <none>           <none>
kube-system    kube-apiserver-master             1/1     Running             0          140m   10.0.0.200      master   <none>           <none>
kube-system    kube-controller-manager-master    1/1     Running             0          140m   10.0.0.200      master   <none>           <none>
kube-system    kube-proxy-2k749                  1/1     Running             0          140m   10.0.0.200      master   <none>           <none>
kube-system    kube-scheduler-master             1/1     Running             0          140m   10.0.0.200      master   <none>           <none>
kube-system    metrics-server-788b8f4cc9-8hz4p   0/1     Pending             0          6s     <none>          <none>   <none>           <none>
kube-system    metrics-server-58445c4b99-kvkt5   0/1     ContainerCreating   0          6s     <none>          master   <none>           <none>

查看了metrics-server的事件信息后发现报如下错误

[root@master ~]# kubectl describe pod -n kube-system metrics-server-58445c4b99-kvkt5 
......
Events:
  Type     Reason                  Age               From               Message
  ----     ------                  ----              ----               -------
  Normal   Scheduled               19s               default-scheduler  Successfully assigned kube-system/metrics-server-58445c4b99-kvkt5 to master
  Warning  FailedCreatePodSandBox  18s               kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "4f8d7f5c66c9b2660ce9a8190449ae94e312bf6a7d08dd211336750218f5178d" network for pod "metrics-server-58445c4b99-kvkt5": networkPlugin cni failed to set up pod "metrics-server-58445c4b99-kvkt5_kube-system" network: error getting ClusterInformation: connection is unauthorized: Unauthorized, failed to clean up sandbox container "4f8d7f5c66c9b2660ce9a8190449ae94e312bf6a7d08dd211336750218f5178d" network for pod "metrics-server-58445c4b99-kvkt5": networkPlugin cni failed to teardown pod "metrics-server-58445c4b99-kvkt5_kube-system" network: error getting ClusterInformation: connection is unauthorized: Unauthorized]
  Normal   SandboxChanged          6s (x2 over 18s)  kubelet            Pod sandbox changed, it will be killed and re-created.


翻译一下就是
无法为 pod “metrics-server-58445c4b99-kvkt5” 设置沙盒容器“4f8d7f5c66c9b2660ce9a8190449ae94e312bf6a7d08dd211336750218f5178d”网络：networkPlugin cni 无法设置 pod “metrics-server-58445c4b99-kvkt5_kube-system” 网络：获取 ClusterInformation 时出错：连接未经授权：未经授权，无法清理沙盒容器“4f8d7f5c66c9b2660ce9a8190449ae94e312bf6a7d08dd211336750218f5178d”pod “metrics-server-58445c4b99-kvkt5”的网络：networkPlugin cni 无法拆解 pod “metrics-server-58445c4b99-kvkt5_kube-system” network： 获取 ClusterInformation 时出错：连接未授权：未授权]

初步分析是Calico没有卸载干净和Flannel出现了冲突。原来kubelet会从默认目录读取配置文件，如果有多个配置文件，那么它会应用按字母顺序首先出现的配置文件中的 CNI 插件，CNI的配置文件默认在/etc/cni/net.d/目录。

[root@master ~]# ll /etc/cni/net.d/
total 12
-rw-r--r--. 1 root root  805 Apr 13 20:23 10-calico.conflist
-rw-r--r--. 1 root root  292 Apr 13 22:27 10-flannel.conflist
-rw-------. 1 root root 2718 Apr 13 20:25 calico-kubeconfig

可以发现CNI配置文件中Calico比Flannel的优先级是要高的，但是我已经卸载了Calico，这就导致了CNI插件是存在故障的，解决方法也比较简单，