编译
- 通过git下载azkaban3.12分支。试过master编译略痛苦，一直报错。
git clone -b 3.12.0 https://github.com/azkaban/azkaban.git
- 打开gitbash
./gradlew clean./gradlew build installDist -x test
- 编译后的压缩包位置都在类似
./azkaban-solo-server/build/distributions/azkaban-solo-server-0.1.0-SNAPSHOT.tar.gz这种路径下(随便举个栗子)
安装
- 创建azkaban在mysql中的表结构
运行mysql的命令 ： mysql -uroot -proot
数据数据库命令 ： create database azkaban;
打开某个数据库 ： use azkaban;
将azkaban sql表结构导入mysql : source /home/zhanqian/azkaban-sql-3.12.0/create-all-sql-3.12.0.sql
1.1 创建azkaban用户
CREATE USER 'azkaban'@'%' IDENTIFIED BY 'azkaban'; #创建用户noah_dba
grant all on azkaban.* to azkaban@'%' identified by 'azkaban'; #授权azkaban给azkaban
flush privileges;#刷新
不出意外后续启动会报错：
2019-03-12 10:32:49 ERROR StdOutErrRedirect:55 - SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
2019-03-12 10:32:49 ERROR StdOutErrRedirect:55 - SLF4J: Defaulting to no-operation (NOP) logger implementation
2019-03-12 10:32:49 ERROR StdOutErrRedirect:55 - SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
2019-03-12 10:32:49 INFO AzkabanWebServer:253 - Loading user manager class azkaban.user.XmlUserManager
2019-03-12 10:32:49 INFO XmlUserManager:176 - Loading user azkaban
2019-03-12 10:32:49 INFO XmlUserManager:176 - Loading user metrics
2019-03-12 10:32:49 INFO XmlUserManager:176 - Loading user admin
2019-03-12 10:32:49 INFO ExecutorManager:200 - Initializing local executor localhost:12321
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - Exception in thread "main"
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - azkaban.executor.ExecutorManagerException: Error fetching active flows
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.executor.JdbcExecutorLoader.fetchActiveFlows(JdbcExecutorLoader.java:216)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorManager.loadRunningFlows(ExecutorManager.java:432)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.executor.ExecutorManager.<init>(ExecutorManager.java:127)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.webapp.AzkabanWebServer.loadExecutorManager(AzkabanWebServer.java:281)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.webapp.AzkabanWebServer.<init>(AzkabanWebServer.java:202)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:762)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - Caused by: java.sql.SQLException: Unknown column 'ex.executor_id' in 'on clause' Query: SELECT ex.exec_id exec_id, ex.enc_type enc_type, ex.flow_data flow_data, et.host host, et.port port, ax.update_time axUpdateTime, et.id executorId, et.active executorStatus FROM execution_flows ex INNER JOIN active_executing_flows ax ON ex.exec_id = ax.exec_id INNER JOIN executors et ON ex.executor_id = et.id Parameters: []
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at org.apache.commons.dbutils.AbstractQueryRunner.rethrow(AbstractQueryRunner.java:363)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:350)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at org.apache.commons.dbutils.QueryRunner.query(QueryRunner.java:306)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - at azkaban.executor.JdbcExecutorLoader.fetchActiveFlows(JdbcExecutorLoader.java:212)
2019-03-12 10:32:50 ERROR StdOutErrRedirect:55 - ... 5 more
你需要继续导入更新SQL（参考：azkaban3.12 编译与安装）
source /home/zhanqian/azkaban-sql-3.12.0/update.execution_flows.3.0.sql #
source /home/zhanqian/azkaban-sql-3.12.0/update.active_executing_flows.3.0.sql #
- 修改azkaban.properties的配置（azkaban-exec-server-3.12.0和azkaban-web-server-3.12.0），如果没找到conf目录，从azkaban-solo-server里面拷贝。
azkaban.name=Test
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/
default.timezone.id=Asia/Shanghai #默认时区为美国时区，需要修改不然时间对不上
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
#user.manager.xml.file=conf/azkaban-users.xml
user.manager.xml.file=/opt/azkaban/azkaban-web-server-3.12.0/conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
database.type=mysql
mysql.port=3306
mysql.host=hadoop-master #如果mysql服务器在本机可以改成localhost
mysql.database=azkaban
mysql.user=root
mysql.password=root
mysql.numconnections=100
# Velocity dev mode
velocity.dev.mode=false
# Azkaban Jetty server properties.
jetty.maxThreads=25
jetty.ssl.port=8443
# jetty.use.ssl=false
jetty.port=8081
jetty.keystore=keystore
jetty.password=azkaban
jetty.keypassword=azkaban
jetty.truststore=keystore
jetty.trustpassword=azkaban
...
启动
- 启动azkaban-web-server
cd /opt/azkaban/azkaban-web-server-3.12.0 #一定要在该目录下，默认启动找的配置文件为 conf/azkaban.properties
sh bin/azkaban-web-start.sh
- 启动azkaban-exe-server
踩坑
- 报错1：上传文件报错
2019-03-14 14:37:35 INFO JdbcProjectLoader:439 - Running update for noah4-client.zip chunk 0
2019-03-14 14:37:37 INFO ProjectManagerServlet:1765 - Installation Failed.
azkaban.project.ProjectManagerException: Error Chunking during uploading files to db...
at azkaban.project.JdbcProjectLoader.uploadProjectFile(JdbcProjectLoader.java:453)
at azkaban.project.JdbcProjectLoader.uploadProjectFile(JdbcProjectLoader.java:358)
at azkaban.project.ProjectManager.uploadProject(ProjectManager.java:540)
at azkaban.webapp.servlet.ProjectManagerServlet.ajaxHandleUpload(ProjectManagerServlet.java:1714)
at azkaban.webapp.servlet.ProjectManagerServlet.handleUpload(ProjectManagerServlet.java:1790)
at azkaban.webapp.servlet.ProjectManagerServlet.handleMultiformPost(ProjectManagerServlet.java:198)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doPost(LoginAbstractAzkabanServlet.java:316)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:688)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:770)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
解决（Ubuntu16.04）：
1. vi /etc/mysql/mysql.conf.d/mysqld.cnf
2. 修改max_allowed_packet = 160M #将16M改成160M
3. 重启 service mysql restart
- 报错
22-04-2018 01:56:02 CST foo ERROR - Job run failed!
azkaban.utils.UndefinedPropertyException: Missing required property 'azkaban.native.lib'
at azkaban.utils.Props.getString(Props.java:421)
at azkaban.jobExecutor.ProcessJob.run(ProcessJob.java:234)
at azkaban.execapp.JobRunner.runJob(JobRunner.java:784)
at azkaban.execapp.JobRunner.doRun(JobRunner.java:600)
at azkaban.execapp.JobRunner.run(JobRunner.java:561)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
解决
出现这个错误时，在azkaban-exec-server-3.46.0/plugins/jobtypes 目录下添加commonprivate.properties配置文件，内容中添加：azkaban.native.lib=false
原文链接：https://blog.csdn.net/qq_37279279/article/details/80031972
使用
command命令截图.png
noah-dp-spark.job
type=command
command=spark2-submit \
--class noah.MyJob \
--master yarn \
--deploy-mode cluster \
--driver-memory 1g \
--executor-memory 1g \
--executor-cores 1 \
--queue thequeue \
--jars mybatis-scala-core_2.11-1.0.3.jar,hbase-client-1.3.0.jar,hbase-common-1.2.0.jar,mybatis-3.4.4.jar,mysql-connector-java-5.1.35.jar,hbase-protocol-1.3.0.jar,hbase-server-1.3.0.jar,htrace-core-3.1.0-incubating.jar,emoji-java-4.0.0.jar \
noah-dp-spark.jar
10
zip包解压后的文件截图.png
