准备
代码列子
1.安装scala插件
开发工具 intellij-IDEA
image.png
2.构建文件
在这里的例子,构建工具采用的是maven,sbt我们在实践中,发现拉取依赖包慢,而且每次更新或者添加依赖的时候,都会遍历检查所有的依赖,非常耗cpu,影响开发,建议maven.
插件
<scala.version>2.11.8</scala.version>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
<args>
<arg>-target:jvm-1.5</arg>
</args>
</configuration>
</plugin>
<plugin>
<groupId>org.scala-tools</groupId>
<artifactId>maven-scala-plugin</artifactId>
<configuration>
<scalaVersion>${scala.version}</scalaVersion>
</configuration>
</plugin>
//依赖:
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>2.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>2.2.1</version>
</dependency>
3.开发代码
数据people.json
{"name":"zhangsan","age":25}
{"name":"wangwu","age":20}
{"name":"lisi","age":28}
{"name":"mazi","age":18}
新建HelloWorld scala object.
val spark = SparkSession
.builder()
.master("local[2]")
.appName("hello world")
.config("spark.some.config.option", "some-value")
.getOrCreate()
import spark.implicits._
val peopleDF = spark.read.json("src/main/resources/people.json")
val newPeopleDF = peopleDF.map(row=>{
val name = row.getAs[String]("name")
val age = row.getAs[Long]("age")
(name,age-18)
}).toDF("name","理黄花大闺女的年龄差")
newPeopleDF.show()
输出:
image.png
