项目背景：
项目中目前用到了多数据中心的consul，并且生产环境处于安全原因的考虑要开启acl。多数据中心代表的是多机房，不同的机房搭建各自的consul集群。然后2个集群在通过join wan关联上。
虽然是不通的机房，但是其实为了满足高可用，2个机房其实是部署相同的服务，比如说服务jetty。目前在2个机房都存在，这样就涉及到了跨机房（跨dc）访问了。
面临的问题
如果单纯不开启acl的话，跨机房访问很简单，只要在请求上加上&dc=dc2这样既可，而且这些参数springcloud sdk也是支持的。但是如果开启了acl之后，情况就不同了。
有以下两种情况：
- 2个dc使用1个token
这就需要把2个dc关联起来了。官网是有一种方式的。叫做ACL Replication for Multiple Datacenters.
官网地址如下：
https://learn.hashicorp.com/tutorials/consul/access-control-replication-multiple-datacenters
相关截图：
image.png
-
2个dc使用2个token
2个dc是独立的，都是自己使用自己的acl配置。虽然acl配置时可以选择all datacenter或者是global，但是测试时发现dc2用第一个dc的token访问的时候还是会提示acl not found。
image.png
image.png
下面就开始实践一下，这两种方式来实现带acl的多dc访问方式
2个dc使用1个token方式
找一台腾讯云服务器，按照官网说明进行配置，因为只有一台，所以需要把端口避开。
dc1的配置如下，这里需要注意配置中要有primary_datacenter，这个代表它是主dc
[root@VM_0_13_centos consul]# cat dc1.json
{
"bootstrap_expect": 1,
"server": true,
"data_dir": "dc1",
"node_name": "node1",
"client_addr": "0.0.0.0",
"bind_addr": "172.21.0.13",
"ports": {
"http": 18081,
"dns": 18082,
"serf_lan": 18083,
"serf_wan": 18084,
"server": 18085
},
"datacenter": "primary_dc",
"primary_datacenter": "primary_dc",
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"tokens": {
"master":"cd76a0f7-5535-40cc-8696-073462acc6c7",
"agent":"deaa315d-98c5-b9f6-6519-4c8f6574a551"
}
},
"pid_file": "./consul.pid"
}
启动dc1
./consul agent -config-file=dc1.json -ui
开始配置dc2，dc2不需要指定token了。因为开启acl replication模式后dc2用的都是dc1的token，而且dc2也不要忘记配置primary_datacenter参数为dc1
{
"bootstrap_expect": 1,
"server": true,
"data_dir": "dc2",
"node_name": "node2",
"client_addr": "0.0.0.0",
"bind_addr": "172.21.0.13",
"ports": {
"http": 18091,
"dns": 18092,
"serf_lan": 18093,
"serf_wan": 18094,
"server": 18095
},
"datacenter": "second_dc",
"primary_datacenter": "primary_dc",
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"enable_token_replication": true
},
"pid_file": "./consul.pid"
}
启动dc2
./consul agent -config-file=dc2.json -ui
访问dc2的接口查看是否replication设置成功，就看ReplicationType是否是tokens就行了,只能查dc2，查dc1不显示
[root@VM_0_13_centos consul]# curl http://localhost:18091/v1/acl/replication?pretty
{
"Enabled": true,
"Running": true,
"SourceDatacenter": "primary_dc",
"ReplicationType": "tokens",
"ReplicatedIndex": 0,
"ReplicatedRoleIndex": 0,
"ReplicatedTokenIndex": 0,
"LastSuccess": "0001-01-01T00:00:00Z",
"LastError": "0001-01-01T00:00:00Z"
}
登录ui页面就能看到dc1和dc2设置的token是一样的
image.png
image.png
自己验证时还发现几个小问题
- dc1（主dc）添加policy和token都没有问题，dc2（从dc）添加policy后在页面看不到，但是也能添加成功，在dc1页面能看到
- dc2（token）可以添加token，添加完dc1里面也能看到，但是由于dc2的policy是空的，所以dc2无法选择policy，也就说明这个添加也没什么用
- dc1如果坏了，dc2的acl菜单打不开（500报错），但是dc2现有的token还是能够正常使用，相当于只是acl配置无法使用
跨dc访问服务方式如下
# 18081代表dc1 访问自己dc的服务，dc参数也可以不加
curl "127.0.0.1:18081/v1/catalog/service/jetty?dc=primary_dc&token=17bc5403-23e8-b09f-a3ef-799f9325e9c4"
or
curl "127.0.0.1:18081/v1/catalog/service/jetty?token=17bc5403-23e8-b09f-a3ef-799f9325e9c4"
# 返回报文ServiceAddress=127.0.0.1代表dc1注册的服务
[
{
"ID": "2abbfa79-3102-a79e-837e-8345a7460017",
"Node": "node1",
"Address": "172.21.0.13",
"Datacenter": "primary_dc",
"TaggedAddresses": {
"lan": "172.21.0.13",
"wan": "172.21.0.13"
},
"NodeMeta": {
"consul-network-segment": ""
},
"ServiceKind": "",
"ServiceID": "jetty",
"ServiceName": "jetty",
"ServiceTags": [
"dev"
],
"ServiceAddress": "127.0.0.1",
"ServiceWeights": {
"Passing": 1,
"Warning": 1
},
"ServiceMeta": {},
"ServicePort": 10035,
"ServiceEnableTagOverride": false,
"ServiceProxyDestination": "",
"ServiceProxy": {},
"ServiceConnect": {},
"CreateIndex": 120,
"ModifyIndex": 120
}
]
# dc1访问dc2的服务，直接访问，token不用换
curl "127.0.0.1:18081/v1/catalog/service/jetty?dc=second_dc&token=17bc5403-23e8-b09f-a3ef-799f9325e9c4"
# 返回报文ServiceAddress=127.0.0.2代表dc2注册的服务
[
{
"ID": "3f5f8948-ab09-cde6-8aa0-9a4ccdab1245",
"Node": "node2",
"Address": "172.21.0.13",
"Datacenter": "second_dc",
"TaggedAddresses": {
"lan": "172.21.0.13",
"wan": "172.21.0.13"
},
"NodeMeta": {
"consul-network-segment": ""
},
"ServiceKind": "",
"ServiceID": "jetty",
"ServiceName": "jetty",
"ServiceTags": [
"dev"
],
"ServiceAddress": "127.0.0.2",
"ServiceWeights": {
"Passing": 1,
"Warning": 1
},
"ServiceMeta": {},
"ServicePort": 10035,
"ServiceEnableTagOverride": false,
"ServiceProxyDestination": "",
"ServiceProxy": {},
"ServiceConnect": {},
"CreateIndex": 5,
"ModifyIndex": 5
}
]
2个dc使用2个token方式
在上面环境的基础上在增加dc3，由于dc3没有配置primary_datacenter，所以他是普通的dc模式，不是replication模式
[root@VM_0_13_centos consul]# cat dc3.json
{
"bootstrap_expect": 1,
"server": true,
"data_dir": "dc3",
"node_name": "node3",
"client_addr": "0.0.0.0",
"bind_addr": "172.21.0.13",
"ports": {
"http": 18096,
"dns": 28092,
"serf_lan": 28093,
"serf_wan": 28094,
"server": 28095
},
"datacenter": "third_dc",
"acl": {
"enabled": true,
"default_policy": "deny",
"down_policy": "extend-cache",
"enable_token_persistence": true,
"enable_token_replication": true,
"tokens": {
"master":"cd76a0f7-5535-40cc-8696-073462acc6c7",
"agent":"deaa315d-98c5-b9f6-6519-4c8f6574a551"
}
},
"pid_file": "./consul.pid"
}
启动dc3
./consul agent -config-file=dc3.json -ui
看dc3的ui页面，token是独立的
image.png
用dc1访问dc3的服务需要加上dc3的token，不能用dc1的token
# dc1用自己的token无法访问dc3的服务
[root@VM_0_13_centos consul]#
curl "127.0.0.1:18081/v1/catalog/service/jetty?dc=third_dc&token=17bc5403-23e8-b09f-a3ef-799f9325e9c4&pretty"
rpc error making call: ACL not found
#dc1将token换成dc3的token就可以访问，ServiceAddress=127.0.0.3表示是dc3的服务
[root@VM_0_13_centos consul]# curl "127.0.0.1:18081/v1/catalog/service/jetty?dc=third_dc&token=f49c40fb-9bb2-0da3-d158-3abcbc4339bc&pretty"
[
{
"ID": "5e118452-e8dc-b66f-2ba0-dd3dcacd609a",
"Node": "node3",
"Address": "172.21.0.13",
"Datacenter": "third_dc",
"TaggedAddresses": {
"lan": "172.21.0.13",
"wan": "172.21.0.13"
},
"NodeMeta": {
"consul-network-segment": ""
},
"ServiceKind": "",
"ServiceID": "jetty",
"ServiceName": "jetty",
"ServiceTags": [
"dev"
],
"ServiceAddress": "127.0.0.3",
"ServiceWeights": {
"Passing": 1,
"Warning": 1
},
"ServiceMeta": {},
"ServicePort": 10035,
"ServiceEnableTagOverride": false,
"ServiceProxyDestination": "",
"ServiceProxy": {},
"ServiceConnect": {},
"CreateIndex": 15,
"ModifyIndex": 15
}
]
具体项目可以根据实际的情况进行选择使用哪种方式
有问题请指正，欢迎交流
