场景:
执行spark-submit --queue=root.XXXX --class com.OrderProfitAgg kafkaspark.jar 时候抛出异常
pom.xml如下:
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.10</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming-kafka-0-8_2.11</artifactId>
<version>2.1.0</version>
</dependency>
<dependency>
<groupId>com.alibaba</groupId>
<artifactId>fastjson</artifactId>
<version>1.2.19</version>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.38</version>
</dependency>
<dependency>
<groupId>commons-dbcp</groupId>
<artifactId>commons-dbcp</artifactId>
<version>1.4</version>
</dependency>
原因:
类org.apache.spark.Logging只在spark 1.5.2及更早得版本才有,高于1.5.2得版本不支持。
解决方法:
下载:spark-core_2.11-1.5.2.logging.jar
然后把spark-core_2.11-1.5.2.logging.jar复制到$SPARK_HOME/jars目录下重新执行spark-submit即可。