一、知识准备
1.calico主要通过ipip协议与bgp协议来实现通信。前者通过ipip隧道作为通信基础，后者则是纯三层的路由交换
2.bgp协议主要由两种方式：BGP Speaker 全互联模式(node-to-node mesh)与BGP Speaker RR模式
3.本文主要探索一下calico bgp的两种模式
二、环境准备
| 组件 | 版本 |
|---|---|
| OS | Ubuntu 18.04.1 LTS |
| docker | 18.06.0-ce |
| k8s | 1.10.1 |
| calico | 3.1.3 |
| ip | hostname |
|---|---|
| 192.168.56.101 | k8s-master |
| 192.168.56.102 | k8s-node1 |
| 192.168.56.103 | k8s-node2 |
| 192.168.56.104 | k8s-node3 |
三、安装
k8s安装
参考官网安装以及社区诸多大神的安装帖子，这里就不班门弄斧了
本文的k8s的环境打开了rbac，etcd加入了证书
calico安装
主要参考官方文档 https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/calico
1.calico rbac
kubectl apply -f \
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/rbac.yaml
2.下载calico.yaml
curl \
https://docs.projectcalico.org/v3.1/getting-started/kubernetes/installation/hosted/calico.yaml \
-O
3.填入etcd地址
ETCD_ENDPOINTS="https://192.168.56.101:2379"
sed -i "s#.*etcd_endpoints:.*# etcd_endpoints: \"${ETCD_ENDPOINTS}\"#g" calico.yaml
sed -i "s#__ETCD_ENDPOINTS__#${ETCD_ENDPOINTS}#g" calico.yaml
4.将etcd证书信息填入。我的etcd证书在/etc/etcd/ssl下
ETCD_CERT=`cat /etc/etcd/ssl/etcd.pem | base64 | tr -d '\n'`
ETCD_KEY=`cat /etc/etcd/ssl/etcd-key.pem | base64 | tr -d '\n'`
ETCD_CA=`cat /etc/etcd/ssl/etcd-root-ca.pem | base64 | tr -d '\n'`
sed -i "s#.*etcd-cert:.*# etcd-cert: ${ETCD_CERT}#g" calico.yaml
sed -i "s#.*etcd-key:.*# etcd-key: ${ETCD_KEY}#g" calico.yaml
sed -i "s#.*etcd-ca:.*# etcd-ca: ${ETCD_CA}#g" calico.yaml
sed -i 's#.*etcd_ca:.*# etcd_ca: "/calico-secrets/etcd-ca"#g' calico.yaml
sed -i 's#.*etcd_cert:.*# etcd_cert: "/calico-secrets/etcd-cert"#g' calico.yaml
sed -i 's#.*etcd_key:.*# etcd_key: "/calico-secrets/etcd-key"#g' calico.yaml
sed -i "s#__ETCD_KEY_FILE__#/etc/etcd/ssl/etcd-key.pem#g" calico.yaml
sed -i "s#__ETCD_CERT_FILE__#/etc/etcd/ssl/etcd.pem#g" calico.yaml
sed -i "s#__ETCD_CA_CERT_FILE__#/etc/etcd/ssl/etcd-root-ca.pem#g" calico.yaml
sed -i "s#__KUBECONFIG_FILEPATH__#/etc/cni/net.d/calico-kubeconfig#g" calico.yaml
5.配置calico bgp 并且修改ip cidr:10.10.0.0/16
sed -i '/CALICO_IPV4POOL_IPIP/{n;s/Always/off/g}' calico.yaml
sed -i '/CALICO_IPV4POOL_CIDR/{n;s/192.168.0.0/10.10.0.0/g}' calico.yaml
6.kubectl安装calico
kubectl apply -f calico.yaml
<font color=#de171c>注意:因为calico-node需要获取操作系统的权限运行，所以要在apiserver、kubelet中加入--allow-privileged=true</font>
查看一下状态:
root@k8s-master:/tmp# kubectl get pods -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE
calico-kube-controllers-98989846-b4n72 1/1 Running 0 18d 192.168.56.102 k8s-node1
calico-node-58pck 2/2 Running 0 18d 192.168.56.103 k8s-node2
calico-node-s2txw 2/2 Running 0 18d 192.168.56.101 k8s-master
calico-node-svmbp 2/2 Running 0 18d 192.168.56.102 k8s-node1
...
7.kubelet配置calico
找到kubelet的配置文件（我的环境在/etc/kubernetes/kubelet），加入
--network-plugin=cni
重启kubelet
8.测试一个pod
cat << EOF | kubectl create -f -
apiVersion: v1
kind: Pod
metadata:
name: network-test
namespace: test
spec:
containers:
- image: busybox:latest
command:
- sleep
- "3600"
name: network-test
EOF
root@k8s-master:~# kubectl -n test get pods -owide
NAME READY STATUS RESTARTS AGE IP NODE
network-test 1/1 Running 0 41s 10.10.169.139 k8s-node2
至此：calico安装已经完成
四、calicoctl使用
1.下载calicoctl
https://github.com/projectcalico/calicoctl/releases/download/v3.1.3/calicoctl-linux-amd64
2.查看当前的calico-node
root@k8s-master:/tmp# calicoctl get node
NAME
k8s-master
k8s-node1
k8s-node2
calicoctl get node -o yaml 查看详细信息
3.查看当前的ippool
root@k8s-master:/tmp# calicoctl get ippool
NAME CIDR
default-ipv4-ippool 10.10.0.0/16
default-ipv6-ippool fdc6:1a69:2b39::/48
4.查看当前模式
root@k8s-master:/tmp# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-------------------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-------------------+-------+----------+-------------+
| 192.168.56.102 | node-to-node mesh | up | 07:39:02 | Established |
| 192.168.56.103 | node-to-node mesh | up | 07:39:02 | Established |
+----------------+-------------------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-master:/tmp# netstat -anp | grep ESTABLISH | grep bird
tcp 0 0 192.168.56.101:33029 192.168.56.102:179 ESTABLISHED 26558/bird
tcp 0 0 192.168.56.101:58055 192.168.56.103:179 ESTABLISHED 26558/bird
当前运行在BGP Speaker 全互联模式(node-to-node mesh)模式，calico集群中的节点之间都会相互建立连接，用于路由交换。适合规模不大的集群中运行，一旦集群节点增大，mesh模式将形成一个巨大服务网格，连接数暴增
5.修改BGP Speaker RR模式
禁止mesh模式，配置bgpPeer
cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPConfiguration
metadata:
name: default
spec:
logSeverityScreen: Info
nodeToNodeMeshEnabled: false
asNumber: 61234
EOF
cat << EOF | calicoctl create -f -
apiVersion: projectcalico.org/v3
kind: BGPPeer
metadata:
name: bgppeer-global
spec:
peerIP: 192.168.56.103
asNumber: 61234
EOF
查看RR模式配置：
root@k8s-master:~# calicoctl get bgpconfig
NAME LOGSEVERITY MESHENABLED ASNUMBER
default Info false 61234
root@k8s-master:~# calicoctl get bgppeer
NAME PEERIP NODE ASN
bgppeer-global 192.168.56.103 (global) 61234
安装routereflector
docker run --privileged --net=host -d \
--name=calico-rr \
-e IP=192.168.56.104 \
-e ETCD_ENDPOINTS=https://192.168.56.101:2379 \
-v /etc/etcd/ssl:/etc/calico/ssl \
-e ETCD_CA_CERT_FILE=/etc/calico/ssl/etcd-root-ca.pem \
-e ETCD_CERT_FILE=/etc/calico/ssl/etcd.pem \
-e ETCD_KEY_FILE=/etc/calico/ssl/etcd-key.pem \
calico/routereflector:v0.6.1
查看效果：
root@k8s-master:~# calicoctl node status
Calico process is running.
IPv4 BGP status
+----------------+-----------+-------+----------+-------------+
| PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO |
+----------------+-----------+-------+----------+-------------+
| 192.168.56.103 | global | up | 09:13:23 | Established |
+----------------+-----------+-------+----------+-------------+
IPv6 BGP status
No IPv6 peers found.
root@k8s-master:~# netstat -anp | grep ESTABLISH | grep bird
tcp 0 0 192.168.56.101:179 192.168.56.103:54903 ESTABLISHED 26558/bird
每台机器都只会与rr建立一条连接，并且与rr通信即可拿到所有路由，大大减少了连接数量
至此，本文结束
在下才疏学浅，有撒汤漏水的，请各位不吝赐教...
