试了下比较流行的几种SDN，感觉flannel还是比较好用，这里简单记录一下。
用的是virtualbox，3个机器，分别为:
genesis: inet 192.168.99.103/24 brd 192.168.99.255 scope global dynamic enp0s3
exodus: inet 192.168.99.105/24 brd 192.168.99.255 scope global dynamic enp0s3
leviticus: inet 192.168.99.106/24 brd 192.168.99.255 scope global dynamic enp0s3
虚机信息如下
[root@localhostyum.repos.d]# uname -mars
Linuxlocalhost.localdomain3.10.0-229.el7.x86_64#1 SMP Fri Mar 6 11:36:42 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@localhostyum.repos.d]# cat /etc/*-release
CentOSLinuxrelease7.3.1611(Core)
NAME="CentOS Linux"VERSION="7 "
(Core)"ID="centos"ID_LIKE="rhel fedora"VERSION_ID="7"
RETTY_NAME="CentOS Linux 7 (Core)"ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"BUG_REPORT_URL="https://bugs.centos.org/"
CENTOS_MANTISBT_PROJECT="CentOS-7"CENTOS_MANTISBT_PROJECT_VERSION="7"REDHAT_SUPPORT_PRODUCT="centos"REDHAT_SUPPORT_PRODUCT_VERSION="7"
CentOSLinuxrelease7.3.1611(Core)
CentOSLinuxrelease7.3.1611(Core)
[root@localhostyum.repos.d]# docker version
Client:Version:1.12.5APIversion:1.24Goversion:go1.6.4Gitcommit:7392c3bBuilt:FriDec1602:23:592016OS/Arch:linux/amd64
随便选择两台机器run一下，在容器中ifconfig:
[root@localhost~]# docker run -it busybox
/#ifconfig
eth0
Linkencap:EthernetHWaddr02:42:AC:11:00:02inetaddr:172.17.0.2Bcast:0.0.0.0Mask:255.255.0.0inet6addr:fe80::42:acff:fe11:2/64Scope:LinkUPBROADCASTRUNNINGMULTICASTMTU:1500Metric:1RXpackets:12errors:0dropped:0overruns:0frame:0TXpackets:6errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:1016(1016.0B)TXbytes:508(508.0B)
lo
Linkencap:LocalLoopbackinetaddr:127.0.0.1Mask:255.0.0.0inet6addr:::1/128Scope:HostUPLOOPBACKRUNNINGMTU:65536Metric:1RXpackets:0errors:0dropped:0overruns:0frame:0TXpackets:0errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:0(0.0B)TXbytes:0(0.0B)
发现参数完全相同，单纯bridge模式下并没有跨主机互通，而host模式是并不建议使用的。
install
先yum install -y etcd flannel，如果没问题是再好不过了。
etcd 3.x支持--config-file参数，需要的话可以从源代码install(需要golang 1.6+)。
先从etcd开始，简单说就是"distributed key value store"。
etcd集群的3种方式:
static
etcd discovery
DNS discovery
DNS discovery主要是用srv record，这里先不搞DNS服务，下面对static和etcd discovery两种方式简单说明一下。
static
参数可以在启动时置顶，或者写到配置文件中，默认配置文件为/etc/etcd/etcd.conf。
genesis的配置如下:
ETCD_NAME=genesis
ETCD_DATA_DIR="/var/lib/etcd/genesis"
ETCD_LISTEN_PEER_URLS="http://192.168.99.103:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.99.103:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.103:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.103:2379"
ETCD_INITIAL_CLUSTER_STATE="new"
ETCD_INITIAL_CLUSTER_TOKEN="etct-fantasy"ETCD_INITIAL_CLUSTER="exodus=http://192.168.99.105:2380,genesis=http://192.168.99.103:2380"
exodus的配置如下:
ETCD_NAME=exodusETCD_DATA_DIR="/var/lib/etcd/exodus"ETCD_LISTEN_PEER_URLS="http://192.168.99.105:2380"ETCD_LISTEN_CLIENT_URLS="http://192.168.99.105:2379,http://127.0.0.1:2379"ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.105:2380"ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.105:2379"ETCD_INITIAL_CLUSTER_STATE="new"ETCD_INITIAL_CLUSTER_TOKEN="etctfantasy"
ETCD_INITIAL_CLUSTER="exodus=http://192.168.99.105:2380,genesis=http://192.168.99.103:2380"
启动方式看自己喜好，如果打算用systemctl启动的话，注意/usr/lib/systemd/system/etcd.service中的内容可能不会如你所愿。
启动后，检查一下集群状态，顺便也看看有哪些成员:
[root@localhost etcd]# etcdctl cluster-health
member7a4f27f78a05e755ishealthy: g
ot healthy resultfromhttp://192.168.99.103:2379
failed to check the health of member8e8718b335c6c9a2 on http://192.168.99.105:2379:
Get http://192.168.99.105:2379/health: dial tcp 192.168.99.105:2379: i/o timeoutmember8e8718b335c6c9a2isunreachable: [http://192.168.99.105:2379] are all unreachableclusterishealthy
提示"member unreachable"，看来是被exodus的防火墙拦住了，我们先粗暴一点。
[root@localhost etcd]# systemctl stop firewalld
[root@localhost etcd]# etcdctl cluster-health
member7a4f27f78a05e755ishealthy: got healthy resultfromhttp://192.168.99.103:2379 member8e8718b335c6c9a2ishealthy: got healthy resultfromhttp://192.168.99.105:2379
cluster is healthy etcd discovery
当然，这样配置的前提是已经知道各个节点的信息。
但实际场景中可能无法预知各个member，所以我们需要让etcd自己去发现(discovery)。
首先，etcd提供了一个public discovery service - discovery.etcd.io，我们用它来生成一个discovery token，并在genesis创建目录:
[root@localhostetcd]# curl https://discovery.etcd.io/newsize=3
https://discovery.etcd.io/6321c0706046c91f2b2598206ffa3272
[root@localhostetcd]# etcdctl set /discovery/6321c0706046c91f2b2598206ffa3272/_config/size 3
修改exodus的配置，用discovery代替之前的cluster:
ETCD_NAME=exodus
ETCD_DATA_DIR="/var/lib/etcd/exodus"
ETCD_LISTEN_PEER_URLS="http://192.168.99.105:2380"
ETCD_LISTEN_CLIENT_URLS="http://192.168.99.105:2379,http://127.0.0.1:2379"
ETCD_INITIAL_ADVERTISE_PEER_URLS="http://192.168.99.105:2380"
ETCD_ADVERTISE_CLIENT_URLS="http://192.168.99.105:2379"
ETCD_DISCOVERY=http://192.168.99.103:2379/v2/keys/discovery/98a976dac265a218f1a1959eb8dde57f
如果启动后一直显示如下错误(参考:raft election):
rafthttp:the clock difference against peer ?????? is too high [??????s > 1s]
简单的解决方法是通过ntp:
[root@localhostetcd]yum install ntp -y
[root@localhostetcd]# systemctl enable ntpd
Createdsymlinkfrom /etc/systemd/system/multi-user.target.wants/ntpd.service to /usr/lib/systemd/system/ntpd.service.
[root@localhostetcd]# systemctl start ntpd
flannel
set一个路径给flannel用:
etcdctlset/coreos.com/network/config'{ "Network": "10.1.0.0/16" }'
以systemctl start flanneld方式启动时，如果出现以下错误
network.go:53]Failedto retrieve networkconfig:100:Keynotfound (/coreos.net) [9]
注意/etc/sysconfig/flanneld中的内容，FLANNEL_ETCD_PREFIX很可能是/atomic.io/network，将其改为/coreos.com/network。
或者也可以通过-etcd-prefix指定。
启动成功后，查看subnet:
[root@localhostetcd]# etcdctl ls /coreos.com/network/subnets
coreos.com/network/subnets/10.1.90.0-24
coreos.com/network/subnets/10.1.30.0-24
coreos.com/network/subnets/10.1.18.0-24
flannel启动成功后会生成/run/flannel/docker，内容如下:
DOCKER_OPT_BIP="--bip=10.1.30.1/24"
DOCKER_OPT_IPMASQ="--ip-masq=true"
DOCKER_OPT_MTU="--mtu=1450"
DOCKER_NETWORK_OPTIONS=" --bip=10.1.30.1/24 --ip-masq=true --mtu=1450 "
用以下方式启动docker:
[root@localhostetcd]# source /run/flannel/docker
[root@localhostetcd]# docker daemon ${DOCKER_NETWORK_OPTIONS} >> /dev/null 2>&1 &
／run/flannel/docker是怎么来的？
参考flanneld的两个启动参数，-subnet-dir和-subnet-file。
在genesis进入容器看看效果:
[root@localhostetcd]# docker run -it busybox
#ifconfig
eth0 Linkencap:EthernetHWaddr02:42:0A:01:5A:02inetaddr:10.1.90.2Bcast:0.0.0.0Mask:255.255.255.0inet6addr:fe80::42:aff:fe01:5a02/64Scope:LinkUPBROADCASTRUNNINGMULTICASTMTU:1450Metric:1RXpackets:6errors:0dropped:0overruns:0frame:0TXpackets:6errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:0RXbytes:508(508.0B)TXbytes:508(508.0B)
lo
Linkencap:LocalLoopbackinetaddr:127.0.0.1Mask:255.0.0.0inet6addr:::1/128Scope:HostUPLOOPBACKRUNNINGMTU:65536Metric:1RXpackets:0errors:0dropped:0overruns:0frame:0TXpackets:0errors:0dropped:0overruns:0carrier:0collisions:0txqueuelen:1RXbytes:0(0.0B)TXbytes:0(0.0B)
在exodus也做类似操作，在exodus中ping一下10.1.90.2，发现是通的。
并在各自容器中也ping一下，检查是否跨主机互通。
