Spark Standalone Cluster环境搭建过程中的一些命令代码有必要记录一下，正好看到以下这篇文章转载过来了。
1.在master虚拟机设置spark-env.sh
（1）复制模板文件来创建spark-env.sh
cp /usr/local/spark/conf/spark-env.sh.template /usr/local/spark/conf/spark-env.sh
（2）修改spark-env.sh
sudo vim /usr/local/spark/conf/spark-env.sh
export SPARK_MASTER_IP=master
export SPARK_WORKER_CORES=1
export SPARK_WORKER_MEMORY=128m
export SPARK_WORKER_INSTANCES=4
2.复制spark程序到data1、data2、data3
（1）复制spark程序到data1
ssh data1
sudo mkdir /usr/local/spark
sudo chown hduser:hduser /usr/local/spark
exit
sudo scp -r /usr/local/spark hduser@data1:/usr/local
（2）复制spark程序到data2
ssh data2
sudo mkdir /usr/local/spark
sudo chown hduser:hduser /usr/local/spark
exit
sudo scp -r /usr/local/spark hduser@data2:/usr/local
（3）复制spark程序到data3
ssh data3
sudo mkdir /usr/local/spark
sudo chown hduser:hduser /usr/local/spark
exit
sudo scp -r /usr/local/spark hduser@data3:/usr/local
3.在master虚拟机编辑slaves文件
1.修改slaves文件
sudo vim /usr/local/spark/conf/slaves
data1
data2
data3
4.启动Spark Standalone Cluster
/usr/local/spark/sbin/start-all.sh
或
/usr/local/spark/sbin/start-master.sh
/usr/local/spark/sbin/start-slaves.sh
5.在spark standalone运行pyspark
pyspark --master spark://master:7077 --num-executors 1 --total-executor-cores 3 --executor-memory 512m
---------------------
作者：剑海风云
来源：CSDN
原文：https://blog.csdn.net/nanxiaotao/article/details/90482958
版权声明：本文为博主原创文章，转载请附上博文链接！
