诉求
- 如何自动的获取到proto文件的message的结构。
- 或者根据最新包动态的解析成最新的JSON串。
- 在服务器收到客户端发来的消息进行反序列化时，根据 消息 msgId ，对消息体进行解析到对应的消息。
流程图.png
前置知识：
【平台化引擎】根据maven坐标—获取到jar的Class文件（URLClassLoader）
Protobuf | 如何在 MAC 上安装 Protobuf 编译 proto 文件
Java使用ProtoBuffer3时报错: Cannot resolve method 'isStringEmpty' in 'GeneratedMessageV3'
原理
protobuf对于每个元素都有一个相应的descriptor，这个descriptor包含该元素的所有元信息，非常类似于Spring中的Bean Definition。下面是各个Descriptor（元数据描述类）的类图：
descriptor的UML.png
- FileDescriptor: 对一个proto文件的描述，它包含文件名、包名、选项（如package, java_package, java_outer_classname等）、文件中定义的所有message、文件中定义的所有enum、文件中定义的所有service、文件中所有定义的extension、文件中定义的所有依赖文件（import）等。在FileDescriptor中还存在一个DescriptorPool实例，它保存了所有的dependencies(依赖文件的FileDescriptor)、name到GenericDescriptor的映射、字段到FieldDescriptor的映射、枚举项到EnumValueDescriptor的映射，从而可以从该DescriptorPool中查找相关的信息，因而可以通过名字从FileDescriptor中查找Message、Enum、Service、Extensions等。可以通过--descriptor_set_out指定生成某个proto文件相对应的FileDescriptorSet文件。
- Descriptor: 对一个message定义的描述，它包含该message定义的名字、所有字段、内嵌message、内嵌enum、关联的FileDescriptor等。可以使用字段名或字段号查找FieldDescriptor。
- FieldDescriptor：对一个字段或扩展字段定义的描述，它包含字段名、字段号、字段类型、字段定义(required/optional/repeated/packed)、默认值、是否是扩展字段以及和它关联的Descriptor/FileDescriptor等。
- EnumDescriptor：对一个enum定义的描述，它包含enum名、全名、和它关联的FileDescriptor。可以使用枚举项或枚举值查找EnumValueDescriptor。
- EnumValueDescriptor：对一个枚举项定义的描述，它包含枚举名、枚举值、关联的EnumDescriptor/FileDescriptor等。
- ServiceDescriptor：对一个service定义的描述，它包含service名、全名、关联的FileDescriptor等。
- MethodDescriptor：对一个在service中的method的描述，它包含method名、全名、参数类型、返回类型、关联的FileDescriptor/ServiceDescriptor等。
实现
1.1 定义pb文件
syntax = "proto3";
option java_package = "com.tellme.cinema";
message UserInfo{
int64 user_id = 1;
string user_name = 2;
}
syntax = "proto3";
import "base.proto";
option java_package = "com.tellme.cinema";
enum MovieType{
CHILDREN = 0;
ADULT = 1;
NORMAL = 2;
OHTER = 3;
}
enum Gender{
MAN = 0;
WOMAN = 1;
OTHER = 2;
}
message Movie{
string name = 1;
MovieType type = 2;
int32 releaseTimeStamp = 3;
string description = 4;
string address = 5;
UserInfo user_info = 6;
}
message Resp{
int32 code = 1;
string message = 2;
}
message Customer{
string name = 1;
Gender gender = 2;
int32 birthdayTimeStamp = 3;
}
message Ticket{
int32 id = 1;
repeated Movie movie = 2;
Customer customer = 3;
}
service TestService {
rpc test(Movie) returns (Resp);
}
syntax = "proto3";
option java_package = "com.tellme.cinema";
message PbInfo{
string root_file_descriptor_name = 1;
string root_message_type_name = 2;
bytes file_descriptor_set = 3;
}
编译proto文件：
protoc --java_out=./tt-component/src/main/java -I=./tt-component/src/main/proto pbInfo.proto cinema1.proto base.proto
1.2 Message使用
（1）简单使用
public static void main(String[] args) throws Exception {
Cinema1.Movie movie = Cinema1.Movie.newBuilder().setAddress("北京").setName("tom")
.setUserInfo(Base.UserInfo.newBuilder().setUserId(1001L).build()).build();
//Message的数组
byte[] bytes = movie.toByteArray();
//知道Message类名
String className = "com.tellme.cinema.Cinema1$Movie";
Class<?> clazz = Class.forName(className);
Method method = clazz.getMethod("getDescriptor");
Descriptors.Descriptor descriptor = (Descriptors.Descriptor) method.invoke(null);
DynamicMessage dynamicMessage2 = DynamicMessage.newBuilder(descriptor).mergeFrom(bytes).build();
String data2 = printer().includingDefaultValueFields()
.preservingProtoFieldNames()
.printingEnumsAsInts()
.print(dynamicMessage2);
System.out.println(data2);
}
（2）复杂使用
传入proto的文件名，然后找到对应的Message对象
private static void test() throws Exception {
Cinema1.Movie movie = Cinema1.Movie.newBuilder().setAddress("北京").setName("tom")
.setUserInfo(Base.UserInfo.newBuilder().setUserId(1001L).build()).build();
//Message的数组
byte[] bytes = movie.toByteArray();
//知道proto类名
String className = "com.tellme.cinema.Cinema1";
Class<?> clazz = Class.forName(className);
//获取到pb文件的描述符
Method method = clazz.getMethod("getDescriptor");
Descriptors.FileDescriptor fileDescriptor = (Descriptors.FileDescriptor) method.invoke(null);
//获取到扩展字段
List<Descriptors.Descriptor> messageTypes = fileDescriptor.getMessageTypes();
for (Descriptors.Descriptor descriptor : messageTypes) {
System.out.println(descriptor.getName());
//获取到描述结构
String jsonString = toJsonString(descriptor);
System.out.println(jsonString);
//获取到文件描述符
if ("Movie".equals(descriptor.getName())) {
DynamicMessage dynamicMessage = DynamicMessage.newBuilder(descriptor).mergeFrom(bytes).build();
String data1 = printer().includingDefaultValueFields()
.preservingProtoFieldNames()
.printingEnumsAsInts()
.print(dynamicMessage);
System.out.println(data1);
}
}
}
（3）解析出来的descriptor对象如何跨服务传输
工具类：
public class DescriptorUtil {
private static String getRootMessageTypeName(String protoClassName) {
if (protoClassName.contains("$")) {
return protoClassName.substring(protoClassName.lastIndexOf("$") + 1);
} else {
return protoClassName.substring(protoClassName.lastIndexOf(".") + 1);
}
}
/**
* 重新构建descriptor对象。
* @param rootFileDescriptorName cinema1.proto
* @param rootMessageTypeName Movie
* @param descData 传入的proto的描述+依赖项的描述
* @return
* @throws Exception
*/
public static Descriptors.Descriptor buildDescriptor(String rootFileDescriptorName, String rootMessageTypeName, byte[] descData)
throws Exception {
System.out.println("rootFileDescriptorName:"+rootFileDescriptorName);
System.out.println("rootMessageTypeName:"+rootMessageTypeName);
//文件依赖关系
DescriptorProtos.FileDescriptorSet fileDescriptorSet = DescriptorProtos.FileDescriptorSet.parseFrom(descData);
Map<String, Descriptors.FileDescriptor> fdMap = new HashMap<>();
Map<String, DescriptorProtos.FileDescriptorProto> name2FileDescriptorProtoMap =
fileDescriptorSet.getFileList().stream().collect(Collectors.toMap(DescriptorProtos.FileDescriptorProto::getName, Function.identity()));
DescriptorProtos.FileDescriptorProto rootFileDescProto = name2FileDescriptorProtoMap.get(rootFileDescriptorName);
Descriptors.FileDescriptor rootFileDescriptor =
Descriptors.FileDescriptor.buildFrom(rootFileDescProto, getDependencies(name2FileDescriptorProtoMap, fdMap, rootFileDescProto));
return rootFileDescriptor.findMessageTypeByName(rootMessageTypeName);
}
/**
* 重建依赖
*/
private static Descriptors.FileDescriptor[] getDependencies(
Map<String, DescriptorProtos.FileDescriptorProto> name2FileDescriptorProtoMap, Map<String, Descriptors.FileDescriptor> fdMap,
DescriptorProtos.FileDescriptorProto fileDescProto) throws Exception {
if (fileDescProto.getDependencyCount() == 0) {
return new Descriptors.FileDescriptor[0];
}
String[] dependencyArray = fileDescProto.getDependencyList().toArray(new String[0]);
int length = dependencyArray.length;
Descriptors.FileDescriptor[] dependencies = new Descriptors.FileDescriptor[length];
for (int i = 0; i < length; i++) {
DescriptorProtos.FileDescriptorProto dependencyFileDescProto = name2FileDescriptorProtoMap.get(dependencyArray[i]);
Descriptors.FileDescriptor fd = fdMap.get(dependencyArray[i]);
if (fd != null) {
dependencies[i] = fd;
} else {
Descriptors.FileDescriptor dependencyFileDesc = Descriptors.FileDescriptor.buildFrom(
dependencyFileDescProto, getDependencies(name2FileDescriptorProtoMap, fdMap, dependencyFileDescProto));
dependencies[i] = dependencyFileDesc;
fdMap.put(dependencyArray[i], dependencyFileDesc);
}
}
return dependencies;
}
/**
* 获取文件描述符列表（包括自身+依赖项）
*/
public static DescriptorProtos.FileDescriptorSet getFileDescriptorSetForDependency(Descriptors.FileDescriptor fileDescriptor) {
Set<Descriptors.FileDescriptor> dependencies = getAllPbDependencies(fileDescriptor);
DescriptorProtos.FileDescriptorSet.Builder fileDescriptorSetBuilder = DescriptorProtos.FileDescriptorSet.newBuilder();
fileDescriptorSetBuilder.addFile(fileDescriptor.toProto());
for (Descriptors.FileDescriptor dependency : dependencies) {
fileDescriptorSetBuilder.addFile(dependency.toProto());
}
return fileDescriptorSetBuilder.build();
}
private static Set<Descriptors.FileDescriptor> getAllPbDependencies(Descriptors.FileDescriptor fileDescriptor) {
Set<Descriptors.FileDescriptor> result = new HashSet<>();
List<Descriptors.FileDescriptor> dependencies = fileDescriptor.getDependencies();
for (Descriptors.FileDescriptor dependency : dependencies) {
result.add(dependency);
result.addAll(getAllPbDependencies(dependency));
}
return result;
}
}
使用：
private static void test1() throws Exception {
Cinema1.Movie movie = Cinema1.Movie.newBuilder().setAddress("北京").setName("tom")
.setUserInfo(Base.UserInfo.newBuilder().setUserId(1001L).build()).build();
//Message的数组
byte[] bytes = movie.toByteArray();
//拿到了对象名
String className = "com.tellme.cinema.Cinema1$Movie";
Class<?> clazz = Class.forName(className);
Method method = clazz.getMethod("getDescriptor");
Descriptors.Descriptor d1 = (Descriptors.Descriptor) method.invoke(null);
//获取到Descriptor配置
Descriptors.FileDescriptor fileDescriptor = d1.getFile();
//类似于远程调用，一个服务解析出pb结构，传递给另一个服务；
PbInfoOuterClass.PbInfo pbInfo = getPbInfoForMock(className, fileDescriptor);
Descriptors.Descriptor descriptor = buildDescriptor(pbInfo.getRootFileDescriptorName(), pbInfo.getRootMessageTypeName(), pbInfo.getFileDescriptorSet().toByteArray());
DynamicMessage dynamicMessage = DynamicMessage.newBuilder(descriptor).mergeFrom(bytes).build();
String data1 = printer().includingDefaultValueFields()
.preservingProtoFieldNames()
.printingEnumsAsInts()
.print(dynamicMessage);
//得到JSON的目标对象，就可以反序列化成map参与后续逻辑
System.out.println(data1);
}
1.3 Method使用
（1）简单使用
private static void test2() throws Exception {
String className = "com.tellme.cinema.Cinema1";
Class<?> clazz = Class.forName(className);
//获取到pb文件的描述符
Method method = clazz.getMethod("getDescriptor");
Descriptors.FileDescriptor fileDescriptor = (Descriptors.FileDescriptor) method.invoke(null);
List<Descriptors.ServiceDescriptor> services = fileDescriptor.getServices();
for (Descriptors.ServiceDescriptor serviceDescriptor : services) {
Descriptors.MethodDescriptor methodDescriptor = serviceDescriptor.findMethodByName("test");
printPrzm(methodDescriptor);
}
}
private static void printPrzm(Descriptors.MethodDescriptor methodDescriptor) {
//输入参数
Descriptors.Descriptor inputType = methodDescriptor.getInputType();
Map<String, Object> param = new HashMap<>();
param.put("address", "北京");
param.put("name", "tom");
HashMap<String, Object> userInfo = Maps.newHashMap();
userInfo.put("userId", 1001);
userInfo.put("userName", "tom");
param.put("userInfo", userInfo);
String jsonString = JSON.toJSONString(param);
System.out.println(jsonString);
//二进制Message
ByteString byteString = MetaInfoUtil.toByteString(inputType, jsonString);
String jsonString1 = toJsonString(inputType, byteString);
System.out.println(jsonString1);
}
（2）进阶使用
private static void test4() throws Exception {
//拿到了对象名
String className = "com.tellme.cinema.Cinema1";
Class<?> clazz = Class.forName(className);
Method method = clazz.getMethod("getDescriptor");
Descriptors.FileDescriptor fileDescriptor = (Descriptors.FileDescriptor) method.invoke(null);
//将文件格式与依赖格式扁平化，
DescriptorProtos.FileDescriptorSet descriptorSet = getFileDescriptorSetForDependency(fileDescriptor);
Descriptors.MethodDescriptor methodDescriptor = getMethodDescriptor("TestService/test", Lists.newArrayList(descriptorSet));
System.out.println(methodDescriptor);
printPrzm(methodDescriptor);
}
1.4 依赖工具类
@Slf4j
public class MetaInfoUtil {
public static Descriptors.MethodDescriptor getMethodDescriptor(String fullMethodName, List<DescriptorProtos.FileDescriptorSet> fileDescriptorSets) {
Preconditions.checkArgument(StringUtils.isNotEmpty(fullMethodName), "fullMethodName 不能为空");
List<DescriptorWrapper> sericeWrapperList =
fileDescriptorSets.stream().map(DescriptorWrapper::fromFileDescriptorSet).collect(Collectors.toList());
for (DescriptorWrapper wrapper : sericeWrapperList) {
//根据方法名，查询方法出入参。
Descriptors.MethodDescriptor methodDescriptor = wrapper.findMethodDescriptor(fullMethodName);
if (methodDescriptor != null) {
return methodDescriptor;
}
}
return null;
}
/**
* 转化为Json串
*/
public static String toJsonString(Descriptors.Descriptor descriptor, ByteString object) {
try {
DynamicMessage message = DynamicMessage.newBuilder(descriptor).mergeFrom(object).build();
return printer()
.includingDefaultValueFields()
.preservingProtoFieldNames()
.printingEnumsAsInts()
.print(message);
} catch (InvalidProtocolBufferException e) {
log.error(e.getMessage());
}
return null;
}
/**
* 转化为Message的二进制流
*/
public static ByteString toByteString(Descriptors.Descriptor descriptor, String json) {
JsonFormat.Parser parser = JsonFormat.parser()
.usingTypeRegistry(JsonFormat.TypeRegistry.newBuilder()
.add(descriptor)
.build())
.ignoringUnknownFields();
DynamicMessage.Builder messageBuilder = DynamicMessage.newBuilder(descriptor);
try {
parser.merge(json, messageBuilder);
return messageBuilder.build().toByteString();
} catch (InvalidProtocolBufferException e) {
log.error(e.getMessage());
throw new RuntimeException(e);
}
}
//打印出Message的结构（JSON化）
public static String toJsonString(Descriptors.Descriptor descriptor) {
DynamicMessage.Builder builder = null;
try {
builder = DynamicMessage.newBuilder(descriptor);
JsonObject jsonObject = convertWithAllFields(builder, Sets.newHashSet());
return jsonObject.toString();
} catch (Throwable e) {
log.error("递归获取json失败，可能会降级只获取json第一级的元素", e);
}
if (builder != null) {
try {
return printer()
.includingDefaultValueFields()
.preservingProtoFieldNames()
.printingEnumsAsInts()
.print(builder);
} catch (InvalidProtocolBufferException e) {
log.error("降级只获取json第一级的元素失败，返回", e);
}
}
return null;
}
private static JsonObject convertWithAllFields(DynamicMessage.Builder builder, Set<String> existField) throws Exception {
Descriptors.Descriptor descriptor = builder.getDescriptorForType();
String jsonString = printer().includingDefaultValueFields().preservingProtoFieldNames().printingEnumsAsInts()
.print(builder);
JsonObject json = new JsonParser().parse(jsonString).getAsJsonObject();
for (Descriptors.FieldDescriptor field : descriptor.getFields()) {
if (field.toProto().getTypeName().equals(".google.protobuf.Any") || !field.getJavaType().equals(MESSAGE)) {
continue; // any类型，什么也不做；非MESSAGE类型，什么也不做
}
if (field.isRepeated()) {
convertRepeatedField(field, builder, existField, json);
} else {
convertNotRepeatedField(field, builder, existField, descriptor, json);
}
}
return json;
}
private static void convertRepeatedField(Descriptors.FieldDescriptor field, DynamicMessage.Builder builder,
Set<String> exist, JsonObject json) throws Exception {
// 是repeated类型且不是map类型(pb定义map不允许是repeated的，但是这里不知道为什么拿到的map也是repeated的，所以过滤下map类型)
if (!field.isMapField()) {
json.add(field.toProto().getName(), convertNotMapField(field, builder, exist));
return;
}
// 如果是map，返回{"key":{}}形式
exist.add(field.getMessageType().getFullName());
Descriptors.Descriptor messageType = field.getMessageType();
Descriptors.FieldDescriptor valueField = messageType.getFields().get(1);
Descriptors.FieldDescriptor keyField = messageType.getFields().get(0);
String key = "";
switch (keyField.getJavaType()) {
case INT:
case LONG:
key = "0";
break;
case BOOLEAN:
key = "false";
break;
default:
key = "";
}
if (valueField.getJavaType().equals(MESSAGE)) {
JsonObject mapObject = new JsonObject();
mapObject.add(key, new JsonObject());
json.add(field.toProto().getName(), mapObject);
} else {
Object defaultValue = valueField.getDefaultValue();
JsonObject jsonObject = new JsonObject();
jsonObject.add(key, new Gson().toJsonTree(defaultValue));
json.add(field.toProto().getName(), jsonObject);
}
}
private static void convertNotRepeatedField(Descriptors.FieldDescriptor field, DynamicMessage.Builder builder, Set<String> exist,
Descriptors.Descriptor descriptor, JsonObject json) throws Exception {
if (field.toProto().hasOneofIndex()) {
// oneof 只获取key, value需要用户自己补全
descriptor.toProto().getOneofDeclList().forEach(proto -> json.add(proto.getName(), new JsonParser().parse("{}").getAsJsonObject()));
return;
}
if (!exist.contains(field.getMessageType().getFullName())) {
json.add(field.toProto().getName(), convertField(field, builder, exist));
} else {
exist.add(field.getMessageType().getFullName());
json.add(field.toProto().getName(), new JsonParser().parse("{}").getAsJsonObject());
}
}
private static JsonArray convertNotMapField(Descriptors.FieldDescriptor field, DynamicMessage.Builder builder,
Set<String> exist) throws Exception {
// 避免message里面包含自己导致的无限递归，如果已经解析过了就不再递归调用
if (!exist.contains(field.getMessageType().getFullName())) {
JsonArray array = new JsonArray(1);
array.add(convertField(field, builder, exist));
return array;
}
return new JsonArray(0);
}
private static JsonObject convertField(Descriptors.FieldDescriptor field, DynamicMessage.Builder builder, Set<String> exist) throws Exception {
exist.add(field.getMessageType().getFullName());
return convertWithAllFields(builder.newBuilderForField(field), exist);
}
}
根据全类名+FileDescriptor获取到MethodDescriptor，以便反射调用。
@Slf4j
class DescriptorWrapper {
private final ImmutableList<FileDescriptor> fileDescriptors;
private DescriptorWrapper(Iterable<FileDescriptor> fileDescriptors) {
this.fileDescriptors = ImmutableList.copyOf(fileDescriptors);
}
public static DescriptorWrapper empty() {
return new DescriptorWrapper(Collections.emptyList());
}
static DescriptorWrapper fromFileDescriptorSet(DescriptorProtos.FileDescriptorSet descriptorSet) {
ImmutableMap<String, FileDescriptorProto> descriptorProtoIndex =
computeDescriptorProtoIndex(descriptorSet);
Map<String, FileDescriptor> descriptorCache = new HashMap<>();
ImmutableList.Builder<FileDescriptor> result = ImmutableList.builder();
for (FileDescriptorProto descriptorProto : descriptorSet.getFileList()) {
try {
result.add(descriptorFromProto(descriptorProto, descriptorProtoIndex, descriptorCache));
} catch (Descriptors.DescriptorValidationException e) {
log.warn("Skipped descriptor " + descriptorProto.getName() + " due to error", e);
}
}
return new DescriptorWrapper(result.build());
}
@Nullable
Descriptors.MethodDescriptor findMethodDescriptor(String fullMethodName) {
int methodIndex = StringUtils.lastIndexOf(fullMethodName, "/");
String packageName;
String serviceName;
if (StringUtils.contains(fullMethodName, ".")) {
int serviceIndex = StringUtils.lastIndexOf(fullMethodName, ".");
packageName = StringUtils.substring(fullMethodName, 0, serviceIndex);
serviceName = StringUtils.substring(fullMethodName, serviceIndex + 1, methodIndex);
} else {
packageName = "";
serviceName = StringUtils.substring(fullMethodName, 0, methodIndex);
}
String methodName = StringUtils.substring(fullMethodName, methodIndex + 1);
return findMethodDescriptor(packageName, serviceName, methodName);
}
private Descriptors.MethodDescriptor findMethodDescriptor(String packageName, String serviceName, String methodName) {
ServiceDescriptor service = findServiceDescriptor(packageName, serviceName);
return service != null ? service.findMethodByName(methodName) : null;
}
private ServiceDescriptor findServiceDescriptor(String packageName, String serviceName) {
for (FileDescriptor fileDescriptor : fileDescriptors) {
if (!fileDescriptor.getPackage().equals(packageName)) {
continue;
}
ServiceDescriptor serviceDescriptor = fileDescriptor.findServiceByName(serviceName);
if (serviceDescriptor != null) {
return serviceDescriptor;
}
}
return null;
}
private static ImmutableMap<String, FileDescriptorProto> computeDescriptorProtoIndex(
DescriptorProtos.FileDescriptorSet fileDescriptorSet) {
ImmutableMap.Builder<String, FileDescriptorProto> resultBuilder = ImmutableMap.builder();
for (FileDescriptorProto descriptorProto : fileDescriptorSet.getFileList()) {
resultBuilder.put(descriptorProto.getName(), descriptorProto);
}
return resultBuilder.build();
}
private static FileDescriptor descriptorFromProto(
FileDescriptorProto descriptorProto,
ImmutableMap<String, FileDescriptorProto> descriptorProtoIndex,
Map<String, FileDescriptor> descriptorCache) throws Descriptors.DescriptorValidationException {
// First, check the cache.
String descriptorName = descriptorProto.getName();
if (descriptorCache.containsKey(descriptorName)) {
return descriptorCache.get(descriptorName);
}
// Then, fetch all the required dependencies recursively.
ImmutableList.Builder<FileDescriptor> dependencies = ImmutableList.builder();
for (String dependencyName : descriptorProto.getDependencyList()) {
if (!descriptorProtoIndex.containsKey(dependencyName)) {
throw new IllegalArgumentException("Could not find dependency: " + dependencyName);
}
FileDescriptorProto dependencyProto = descriptorProtoIndex.get(dependencyName);
dependencies.add(descriptorFromProto(dependencyProto, descriptorProtoIndex, descriptorCache));
}
// Finally, construct the actual descriptor.
FileDescriptor[] empty = new FileDescriptor[0];
final FileDescriptor fileDescriptor =
FileDescriptor.buildFrom(descriptorProto, dependencies.build().toArray(empty));
descriptorCache.putIfAbsent(descriptorProto.getName(), fileDescriptor);
return fileDescriptor;
}
}
