1.把hive-site.xml放到resources下并导入必要的依赖
import org.apache.spark.sql.SparkSession
object ReadHive extends App {
System.setProperty("HADOOP_USER_NAME", "hdfs")
val spark: SparkSession = SparkSession.builder().appName(s"${ReadHive.getClass.getSimpleName}")
.master("local[*]")
.config("spark.sql.warehouse.dir", "/user/hive/warehouse")
.enableHiveSupport()
.getOrCreate()
spark.sql("use yyt_db_bi")
spark.sql("select count(1) from ws_gl_account_d").show()
spark.stop()
}
2.需要的依赖
<dependencies>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.0</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.2.0</version>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.2.2</version>
<configuration>
<recompileMode>incremental</recompileMode>
</configuration>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
注意:在windows中如果没有配置hadoop的会报错需要hadoop.dll(放在System32下)文件和一个winutils的文件并设置好环境变量。
HADOOP_HOME=C:\Program Files\hadoop
在Path后面加上;%HADOOP_HOME%\bin