通过docker-machine +consul +overlay实现跨主机通信
一、基础环境配置
三台服务器均执行以下操作
1. 配置固定IP
[root@localhost ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens32
TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=static
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens32
UUID=75963e3f-b289-4bbd-8489-44f6f2b8c7f0
DEVICE=ens32
ONBOOT=yes
IPADDR=192.168.0.10
PREFIX=24
GATEWAY=192.168.0.1
DNS1=114.114.114.114
[root@localhost ~]# systemctl restart network
2. 更改主机名
[root@localhost ~]# hostnamectl set-hostname docker-ce
[root@localhost ~]# exit //重新登陆即可
[root@docker-ce ~]#
在20，30两台服务器上重复上面的操作，docker1是IPADDR=192.168.0.20，docker2是IPADDR=192.168.0.30
3.关闭防火墙
[root@docker-ce ~]# systemctl stop firewalld
[root@docker-ce ~]# systemctl disable firewalld
4. 同步系统时间
[root@docker-ce ~]# yum -y install ntp
[root@docker-ce ~]# systemctl enable ntpd.service
[root@docker-ce ~]# ntpdate cn.pool.ntp.org
[root@docker-ce ~]# hwclock -w
[root@docker-ce ~]# crontab -e
0 2 * * * ntpdate ntpdate cn.pool.ntp.org && hwclock -w
5.关闭selinux
[root@dockerce ~]# vim /etc/sysconfig/selinux
SELINUX=disabled
[root@docker-ce ~]# reboot
二、docker-ce配置
1. 安装docker-machine
方式一：
[root@docker-ce ~]# curl -L https://github.com/docker/machine/releases/download/v0.14.0/docker-machine-`uname -s`-`uname -m` >/tmp/docker-machine && chmod +x /tmp/docker-machine && cp /tmp/docker-machine /usr/local/bin/docker-machine
%Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 617 0 617 0 0 582 0 --:--:-- 0:00:01 --:--:-- 583
100 25.1M 100 25.1M 0 0 644k 0 0:00:40 0:00:40 --:--:-- 2801k
此连接安装时，有时会链接超时,多链接几次或是过会再链接应该会成功。
方式二：
直接下载安装包
[root@docker-ce ~]# yum -y install wget
[root@docker-ce ~]# wgethttps://github.com/docker/machine/releases/download/v0.14.0/docker-machine-Linux-x86_64
[root@docker-ce ~]# mv docker-machine-Linux-x86_64 /usr/local/bin/docker-machine
[root@docker-ce ~]# chmod +x /usr/local/bin/docker-machine
[root@docker-ce ~]# docker-machine version
docker-machine version 0.14.0, build 89b8332
//docker-machine软件包需要翻墙下载
2. docker1,docker2 配置免密钥登录
[root@docker-ce ~]# ssh-keygen //（生成主机密钥对，一路回车即可）
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:27whAu+mDPZzqj25FcbF47Z8nbIav14KbxGeCl6aZac root@docker-ce
The key's randomart image is:
+---[RSA 2048]----+
| |
| . |
| + |
| . o .. |
| . + S. o |
| +.++==. . |
| o o+BB+*.+ |
| . =+o*.EB.B |
| ..BO. .+O. |
+----[SHA256]-----+
[root@docker-ce ~]# ssh-copy-id 192.168.0.20 //（复制公钥到20服务器）
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.0.20 (192.168.0.20)' can't be established.
ECDSA key fingerprint is SHA256:eOpJf2pvOii5sgnKZS+Wb3G3hc/7deEqGzuPcb2Ymhs.
ECDSA key fingerprint is MD5:1f:20:29:32:84:1e:59:cd:47:a5:c9:c3:82:77:50:f6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.0.20's password: //输入远程主机20的root用户的密码
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '192.168.0.20'"
and check to make sure that only the key(s) you wanted were added.
[root@docker-ce ~]# ssh-copy-id 192.168.0.30 //（复制公钥到30服务器）
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '192.168.0.30 (192.168.0.30)' can't be established.
ECDSA key fingerprint is SHA256:eOpJf2pvOii5sgnKZS+Wb3G3hc/7deEqGzuPcb2Ymhs.
ECDSA key fingerprint is MD5:1f:20:29:32:84:1e:59:cd:47:a5:c9:c3:82:77:50:f6.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@192.168.0.30's password: //输入远程主机30的root用户的密码
Number of key(s) added: 1
Now try logging into the machine, with: "ssh '192.168.0.30'"
and check to make sure that only the key(s) you wanted were added.
3. 登陆验证
[root@docker-ce ~]# ssh 192.168.0.20
Last login: Fri May 18 10:42:53 2018 from 192.168.0.110
[root@docker02 ~]# exit
[root@docker-ce ~]# ssh 192.168.0.30
Last login: Fri May 18 10:43:11 2018 from 192.168.0.110
[root@docker03 ~]# exit
4. 批量安装docker环境并配置启动
[root@docker-ce ~]# docker-machine ls （查看安装docker的机器）
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
在进行批量安装的时候，主要做以下几件事：
1> 使用ssh连接到远程主机上（配置无密码登录等原因）
2> 安装docker（从dockerhub上下载最新的安装包进行安装）
3> 将本机生成证书并将证书拷贝到远程主机上
4> 设置远程主机的启动程序并启动远程主机的docker进程
批量安装时花费的时间比较长
[root@docker-ce ~]# docker-machine create -d generic --generic-ip-address=192.168.0.20 docker1（使用generic驱动安装ip为192.168.0.20的主机，并且将其主机名设置为docker1）
Creating CA: /root/.docker/machine/certs/ca.pem
Creating client certificate: /root/.docker/machine/certs/cert.pem
Running pre-create checks...
Creating machine...
(docker1) No SSH key specified. Assuming an existing key at the default location.
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with centos...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env docker1
[root@docker-ce ~]# docker-machine create -d generic --generic-ip-address=192.168.0.30 docker2
Running pre-create checks...
Creating machine...
(docker2) No SSH key specified. Assuming an existing key at the default location.
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with centos...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env docker2
[root@docker-ce ~]# docker-machine ls （检查是否安装成功，查看远程主机上的版本）
NAME ACTIVE DRIVER STATE URL SWARM DOCKER ERRORS
docker1 - generic Running tcp://192.168.0.20:2376 v18.05.0-ce
docker2 - generic Running tcp://192.168.0.30:2376 v18.05.0-ce
5. 其他情况说明
在生成证书的时候，在本机上和远程主机上都会保存相关的证书信息，而且docker-mahine的配置信息也是保存在此位置，相当于控制的主机，如下所示：
[root@docker-ce ~]# ls -l .docker/（在家目录的隐藏目录.docker目录下）
total 4
-rw-------. 1 root root 173 Jan 17 14:52 config.json（保存的是控制机的配置信息，例如docker客户端的版本）
drwx------. 4 root root 35 Jan 13 14:34 machine（保存本机和远程主机上证书信息及相关远程主机的配置信息）
远程主机运行的进程时候，已经做了相关的配置，进程也已经启动，查看启动的进程信息：
[root@docker2 ~]# ls -l /etc/docker/（证书保存的配置，使用https进行通信，保证安全性）
total 20
-rw-r--r--. 1 root root 1029 Jan 20 05:18 ca.pem
-rw-r--r--. 1 root root 56 Jan 19 03:12 daemon.json
-rw-------. 1 root root 245 Dec 12 21:40 key.json
-rw-r--r--. 1 root root 1679 Jan 20 05:18 server-key.pem
-rw-r--r--. 1 root root 1103 Jan 20 05:18 server.pem
[root@docker2 ~]# cat /etc/systemd/system/docker.service.d/10-machine.conf
（启动的配置文件，在单机host环境中不存在此目录，此文件会覆盖默认的启动的配置文件）
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper--tlsverify --tlscacert/etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic
Environment=
[root@docker2 ~]# ps -ef|grep docker（查看启动的docker进程）
root 21741 1 0 05:18 ? 00:00:11 /usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic
root 21746 21741 0 05:19 ? 00:00:11 docker-containerd --config /var/run/docker/containerd/containerd.toml
root 21950 17184 0 05:39 pts/0 00:00:00 grep --color=auto docker
[root@docker2 ~]# netstat -tunlpx |grep dockerd（查看docker监听的端口）
tcp6 0 0 :::2376:::* LISTEN 21741/dockerd
unix 2 [ ACC ] STREAM LISTENING 324400 21741/dockerd /var/run/docker.sock
unix 2 [ ACC ] STREAM LISTENING 323509 21741/dockerd /var/run/docker/metrics.sock
unix 2 [ ACC ] STREAM LISTENING 324567 21741/dockerd /run/docker/libnetwork/fc7eb55d830a15800059dcf61156a80314b2ba7354834170cf198702e586f22e.sock
在使用多机环境的时候需要注意的是，启动的配置文件已经被覆盖了，如果去修改单机环境中的启动的配置文件是不能生效的，必须要修改新创建的目录下的文件内容。
三、跨主机通信
使用overlay驱动进行跨主机通信，在使用此驱动的时候，是通过udp的4789端口进行通信。
1. 创建数据库来保存网络信息
docker-ce配置
直接使用容器consul来进行运行，如下所示：
[root@docker-ce]# docker run -d -p 8500:8500 -h consul --name consul progrium/consul --server -bootstrap
Unable to find image 'progrium/consul:latest' locally
latest: Pulling from progrium/consul
c862d82a67a2: Pull complete
0e7f3c08384e: Pull complete
0e221e32327a: Pull complete
09a952464e47: Pull complete
60a1b927414d: Pull complete
4c9f46b5ccce: Pull complete
417d86672aa4: Pull complete
b0d47ad24447: Pull complete
fd5300bd53f0: Pull complete
a3ed95caeb02: Pull complete
d023b445076e: Pull complete
ba8851f89e33: Pull complete
5d1cefca2a28: Pull complete
Digest: sha256:8cc8023462905929df9a79ff67ee435a36848ce7a10f18d6d0faba9306b97274
Status: Downloaded newer image for progrium/consul:latest
6ac42624732d189e0e8913c69632b266e0fe67103c4f1403ffd8b155cffc1e61
[root@docker-ce]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6ac42624732d progrium/consul "/bin/start --server…" 18 seconds ago Up 14 seconds 53/tcp, 53/udp, 8300-8302/tcp, 8400/tcp, 8301-8302/udp, 0.0.0.0:8500->8500/tcp consul
使用ip加端口直接进行访问，此处是http://192.168.0.10:8500
2. 远程主机配置
在远程主机上要在这个数据库上注册，从而需要修改配置文件，如下所示：
[root@docker1 ~]# vim /etc/systemd/system/docker.service.d/10-machine.conf （加粗内容是需要添加的）
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2376 -H unix:///var/run/docker.sock --storage-driver devicemapper --tlsverify --tlscacert /etc/docker/ca.pem --tlscert /etc/docker/server.pem --tlskey /etc/docker/server-key.pem --label provider=generic --cluster-store=consul://192.168.0.10:8500 --cluster-advertise=ens32:2376
Environment=
参数说明：
--cluster-store主要表示为需要连接的kv存储的地址，--cluster-advertise表示使用哪个网卡和端口来进行通信。ens32表示为网卡的名称或者使用主机的ip地址。
[root@docker1 ~]# systemctl daemon-reload（重新加载配置文件）
[root@docker1 ~]# systemctl restart docker（重新启动服务）
docker2上同样进行上述操作
再次查看kv存储，发现已经注册，注意在点击的时候，需要先点击key/value,然后再点击docker，最后点击nodes才能看到注册的主机
3. 创建overlay网络
[root@docker1 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
ef17c0c56159 bridge bridge local
34963f83928c host host local
a79f72191b90 none null local
[root@docker1 ~]# docker network create -d overlay kel //（创建overlay网络）
8675d048411335e1738996b62fb9e505e270379a3efd1a4af2a9f7b96c03dd1f
[root@docker1 ~]# docker network ls
NETWORK ID NAME DRIVER SCOPE
ef17c0c56159 bridge bridge local
34963f83928c host host local
8675d0484113 kel overlay global
a79f72191b90 none null local
[root@docker2 ~]# docker network ls //（在主机docker1上也能看到此网络）
NETWORK ID NAME DRIVER SCOPE
d78fa5f31c2a bridge bridge local
34963f83928c host host local
8675d0484113 kel overlay global
a79f72191b90 none null local
4. 创建容器测试网络连通性
[root@docker1 ~]# docker pull cirros
Using default tag: latest
latest: Pulling from library/cirros
480d57c7bf4d: Pull complete
0f45bfe9a805: Pull complete
5aa6c26e64dc: Pull complete
Digest: sha256:e67f6f4a0521e326ba2dd697950046aba5ce836edda79cb818d45a56841c7ca2
Status: Downloaded newer image for cirros:latest
[root@docker1 ~]# docker run -itd --name b1 --network=kel cirros //（运行一个网络在kel网络中的容器）
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0A:00:00:04
inet addr:10.0.0.2 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
[root@docker2 ~]# docker run -itd --name b2 --network=kel cirros //（在另一台主机上运行一个kel网段中的容器）
/ # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:0A:00:00:03
inet addr:10.0.0.3 Bcast:10.0.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)
/# ip r //(查看路由）
default via 172.18.0.1 dev eth1
10.0.0.0/24 dev eth0 scope link src 10.0.0.2
172.18.0.0/16 dev eth1 scope link src 172.18.0.2
/ # cat /etc/resolv.conf //(自带的DNS解析）
nameserver 127.0.0.11
options ndots:0
/ # ip addr show //（有两个网络接口）
1: lo: mtu 65536 qdisc noqueue qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
7: eth0@if8: mtu 1450 qdisc noqueue
link/ether 02:42:0a:00:00:02 brd ff:ff:ff:ff:ff:ff
inet 10.0.0.2/24 brd 10.0.0.255 scope global eth0
valid_lft forever preferred_lft forever
10: eth1@if11: mtu 1500 qdisc noqueue
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
inet 172.18.0.2/16 brd 172.18.255.255 scope global eth1
valid_lft forever preferred_lft forever
5. 不同宿主机内容器互通测试
docker1内容器
/ # ping b2 （跨主机ping)
PING b2 (10.0.0.3): 56 data bytes
64 bytes from 10.0.0.3: seq=0 ttl=64 time=19.570 ms
64 bytes from 10.0.0.3: seq=1 ttl=64 time=3.206 ms
--- b2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 3.206/11.388/19.570 ms
docker2内容器
/# ping b1
PING 10.0.0.2 (10.0.0.2): 56 data bytes
64 bytes from 10.0.0.2: seq=0 ttl=64 time=37.227 ms
64 bytes from 10.0.0.2: seq=1 ttl=64 time=0.448 ms
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 0.448/18.837/37.227 ms
原创作品，允许转载，转载时请务必以超链接形式标明文章 原始出处 、作者信息和本声明。
