Spark 1.6.3安装

设置主机名

[root@localhost ~]# hostnamectl set-hostname sam01
修改hosts文件，加入主机名
[root@sam01 ~]# vi /etc/hosts
127.0.0.1   sam01 sam01
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6

查看主机名

[root@sam01 ~]# hostname
sam01

修改当前用户.bash_profile

[sam@localhost ~]$ vi .bash_profile
# .bash_profile

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
        . ~/.bashrc
fi

# User specific environment and startup programs

PATH=$PATH:$HOME/.local/bin:$HOME/bin

export JAVA_HOME=$HOME/tools/jdk1.7.0_80
export CLASSPATH=$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export SCALA_HOME=$HOME/tools/scala-2.11.8
export MAVEN_HOME=$HOME/tools/apache-maven-3.3.9
export SBT_HOME=$HOME/tools/sbt
export HADOOP_HOME=$HOME/tools/hadoop-2.6.5
export HADOOP_CONF_DIR=$HOME/tools/hadoop-2.6.5/etc/hadoop
export SPARK_HOME=$HOME/tools/spark-1.6.3-bin-hadoop2.6
PATH=$HADOOP_HOME/sbin:$HADOOP_HOME/bin:$SPARK_HOME/bin:$SCALA_HOME/bin:$JAVA_HOME/bin:$MAVEN_HOME/bin:$SBT_HOME/bin:$PATH
export PATH

使配置文件立即生效

[sam@localhost ~]$ source .bash_profile

检查环境

[sam@localhost ~]$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

[sam@localhost ~]$ scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL

[sam@localhost ~]$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-11T00:41:47+08:00)
Maven home: /home/sam/tools/apache-maven-3.3.9
Java version: 1.7.0_80, vendor: Oracle Corporation
Java home: /home/sam/tools/jdk1.7.0_80/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"

[sam@localhost ~]$ sbt -batch sbt-version
[info] Loading project definition from /home/sam/project
[info] Set current project to sam (in build file:/home/sam/)
[info] 0.13.15

修改Spark配置文件

[sam@localhost conf]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/conf
[sam@localhost conf]$ cp spark-env.sh.template spark-env.sh
在spark-env.sh最后添加下面
export SCALA_HOME=/home/sam/tools/scala-2.11.8
export SPARK_MASTER_IP=sam01
export SPARK_WORKER_MEMORY=1G
export JAVA_HOME=/home/sam/tools/jdk1.7.0_80

配置slaves,在slaves最后添加主机名

[sam@localhost conf]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/conf
[sam@localhost conf]$ cp slaves.template slaves
sam01

启动master

[sam@localhost sbin]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/sbin
[sam@sam01 sbin]$ ./start-master.sh
starting org.apache.spark.deploy.master.Master, logging to /home/sam/tools/spark-1.6.3-bin-hadoop2.6/logs/spark-sam-org.apache.spark.deploy.master.Master-1-sam01.out

启动worker

[sam@localhost sbin]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/sbin
[sam@localhost sbin]$ ./start-slaves.sh spark://sam01:7077
sam01: starting org.apache.spark.deploy.worker.Worker, logging to /home/sam/tools/spark-1.6.3-bin-hadoop2.6/logs/spark-sam-org.apache.spark.deploy.worker.Worker-1-sam01.out

启动Shell

[sam@sam01 bin]$ pwd
/home/sam/tools/spark-1.6.3-bin-hadoop2.6/bin
[sam@sam01 bin]$ ./spark-shell --master spark://sam01:7077

访问http://sam01:8080/
(如不能通过主机名访问，可修改hosts文件加入ip与主机名对应关系)查看状态

ssh配置

[sam@sam01 ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/sam/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /home/sam/.ssh/id_rsa.
Your public key has been saved in /home/sam/.ssh/id_rsa.pub.
The key fingerprint is:
aa:a9:c1:26:7b:3c:25:a7:f2:1a:c3:10:5d:50:8a:2b sam@sam01
The key's randomart image is:
+--[ RSA 2048]----+
|  .oo            |
| o o             |
|o o              |
| o               |
|E       S        |
|+.. o  .         |
|+o+=  .          |
|.*=. o           |
|o=ooo            |
+-----------------+

[sam@sam01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub sam@sam01
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sam@sam01's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'sam@sam01'"
and check to make sure that only the key(s) you wanted were added.

[sam@sam01 ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub sam@sam02
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
sam@sam02's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh 'sam@sam02'"
and check to make sure that only the key(s) you wanted were added.