此spark-submit脚本可以配合定时任务,定时任务做定时5分钟(时间随你定,建议大于1分钟),用于监控spark实时任务,若spark实时任务挂掉,5分钟后会重新启动
spark-submit脚本:
#!/bin/bash
count=`ps -ef | grep totalUriStreaming | grep -v grep | wc -l`
kinit -kt /root/hdfs.keytab hdfs/p37.awifi.com@AWIFI.COM #此为kebros安全认证,若集群中没有kebros安全认证则不需要
if [ $count == 0 ]; then
echo "Start totalLogUriNums ..."
nohup spark-submit \
--class com.awifi.bigdata.syslog.totalLogUriNums.totalUriStreaming \
--master yarn \
--driver-memory 2G \
--executor-memory 4G \
--deploy-mode client \
--num-executors 12 \
--executor-cores 2 \
--jars /opt/gofunTest/jars/commons-pool2-2.0.jar,/opt/gofunTest/jars/jedis-2.8.1.jar,/opt/gofunTest/jars/fastjson-1.2.30.jar,/opt/gofunTest/jars/mysql-connector-java-5.1.41-bin.jar \
/opt/gofunTest/totalUriNums/sparkAnalysis.jar > /opt/gofunTest/totalUriNums/totalUriNums.out 2>&1 &
fi
定时任务:(可以看我另一篇关于定时任务的博客)
*/5 * * * * root /xxx/xxx/xxx/spark-submit.sh