azkaban 环境搭建

azkaban 环境搭建，插件安装，简单demo，功能演示。

一. azkaban 环境搭建

1. azkaban 代码编译

  • 打开 azkaban 源码地址,将代码下载下来。

    1
    2
    3
    4
    5
    6
    7
     ## 选择自己适合的版本，我选的 3.49.0版本，azkaban高版本与 azkaban插件(3.0)不兼容

    git clone https://github.com/azkaban/azkaban.git

    git tag  

    git checkout 3.49.0

  • 编译代码(azkaban 用的是gradle进行的项目构建管理)

    1
    2
    ##没装gradle的，使用下面命令跳过测试构建环境
    ./gradlew clean build  -x test

  • 编译包获取

这个包只能安装到linux环境，因为exection调用了linux /proc/loadavg 与 /proc/meminfo 等命令。

  • mac 环境下需要修改azkaban.execapp.ServerStatisticsServlet 类，将判断条件去掉

2. 环境准备

  • 创建mysql账号及数据库
1
2
3
4
5
6
7
8
9
10
11
12
13
###创建azkaban用户
CREATE USER 'azkaban'@'%' IDENTIFIED BY '******';

###赋权
GRANT ALL PRIVILEGES ON `azkaban_new`.* TO 'azkaban'@'%';

###创建数据库
CREATE DATABASE IF NOT EXISTS azkaban_new DEFAULT CHARSET utf8 COLLATE utf8_general_ci;
flush privileges;

## 解压包
tar -zxvf azkaban-db-3.49.0.tar.gz
source azkaban-db-3.49.0/create-all-sql-3.49.0.sql
  • 配置环境变量(/etc/profile)
1
2
export HADOOP_HOME=/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3
export HADOOP_CONF_DIR=/etc/hadoop/conf
注意: 一定要配置，否则下面的hdfs-viewer插件，将无法使用。

  • 解压文件

    1
    2
    3
    cd /opt
    tar -zxvf azkaban-exec-server-3.49.0.tar.gz
    tar -zxvf azkaban-web-server-3.49.0.tar.gz

3. 配置azkaban-web

  • 在服务根目录下配置ssl
1
2
3
4
cd azkaban-web-server-3.49.0

##生成keystore文件，需要填写一些信息
keytool -keystore keystore -alias jetty -genkey -keyalg RSA

  • 修改conf/azkaban.properties 文件
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

azkaban.name=nick-wei
azkaban.label=My Local Azkaban
azkaban.color=#FF3601
azkaban.default.servlet.path=/index
web.resource.dir=web/

#时区设置
default.timezone.id=Asia/Shanghai
# Azkaban UserManager class
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml
# Loader for projects
executor.global.properties=conf/global.properties
azkaban.project.dir=projects
# Velocity dev mode
velocity.dev.mode=false

# Azkaban Jetty server properties.
#jetty.use.ssl=false
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=weixingxing
jetty.keypassword=weixingxing
jetty.truststore=keystore
jetty.trustpassword=weixingxing


# Azkaban Executor settings
executor.port=12321


# email 设置
mail.sender=xxxx@163.com
mail.host=smtp.163.com
mail.user=xxxx@163.com
mail.password=xxxx
#mail.port=465
mail.port=25


job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache
# JMX stats
jetty.connector.stats=true
executor.connector.stats=true


# mysql 配置
database.type=mysql
mysql.port=3306
mysql.host=xxx
mysql.database=azkaban_new
  • 修改 conf/azkaban-users.xml 文件
1
2
3
4
5
6
7
8
9
##此文件是用户权限控制文件，可以修改也可以使用默认的
<azkaban-users>
  <user groups="azkaban" password="azkaban" roles="admin" username="azkaban"/>
  <user password="metrics" roles="metrics" username="metrics"/>

  <user username="admin" password="admin" roles="admin,metrics" />
  <role name="admin" permissions="ADMIN"/>
  <role name="metrics" permissions="METRICS"/>
</azkaban-users>

3. 配置azkaban-exec-server

  • 修改 conf/azkaban.properties
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
## 时区设置
default.timezone.id=Asia/Shanghai

# 权限设置
user.manager.class=azkaban.user.XmlUserManager
user.manager.xml.file=conf/azkaban-users.xml

# 全局配置
executor.global.properties=conf/global.properties
## project 本地缓存存放地址
azkaban.project.dir=projects
# exec server 服务端口配置
executor.port=12321
executor.connector.stats=true

# Velocity dev mode
velocity.dev.mode=false

# Azkaban Jetty server properties.
jetty.maxThreads=25
jetty.ssl.port=8443
jetty.port=8081
jetty.keystore=keystore
jetty.password=weixingxing
jetty.keypassword=weixingxing
jetty.truststore=keystore
jetty.trustpassword=weixingxing
#jetty.use.ssl=false
# JMX stats
jetty.connector.stats=true

# 邮箱设置
mail.sender=
mail.host=
job.failure.email=
job.success.email=
lockdown.create.projects=false
cache.directory=cache

# mysql 配置
database.type=mysql
mysql.port=3306
mysql.host=h1
mysql.database=azkaban_new
mysql.user=azkaban
mysql.password=azkaban
mysql.numconnections=100


#Multiple Executor,exec需要的资源
azkaban.use.multiple.executors=true
azkaban.executorselector.filters=StaticRemainingFlowSize,MinimumFreeMemory,CpuStatus
azkaban.executorselector.comparator.NumberOfAssignedFlowComparator=1
azkaban.executorselector.comparator.Memory=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1
azkaban.executorselector.comparator.LastDispatched=1
azkaban.executorselector.comparator.CpuUsage=1

##jobtype插件地址
azkaban.jobtype.plugin.dir=plugins/jobtype

二. azkaban-plugins 环境搭建

1. azkaban-plugins 代码编译

1
2
## 选择azkaban插件(3.0)最高版本
git clone https://github.com/azkaban/azkaban-plugins.git
  • 编译代码（azkaban-plugins 用的是ant进行的项目构建管理）
1
2
cd azkaban-plugins
ant
注意：在更目录下，修改build.xml文件，注释掉 pigvisualizer 引用

  • 编译包获取
1
find . -type f \( -iname "azkaban-jobtype*.tar.gz" -o -iname "azkaban-javaviewer*.tar.gz" -o -iname "azkaban-jobsummary*.tar.gz" -o -iname "azkaban-hdfs-viewer*.tar.gz" -o -iname "azkaban-reportal*.tar.gz"  \)

2. 配置azkaban-web 插件

  • (1) 配置hdfs-viewer插件
1
2
3
4
5
cd plugins/viewer/
##解压文件
tar -zxvf azkaban-hdfs-viewer-3.0.0.tar.gz
##重命名插件
mv azkaban-hdfs-viewer-3.0.0 hdfs

修改配置文件: hdfs/conf/plugin.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
viewer.name=HDFS
viewer.path=hdfs
viewer.order=1
viewer.hidden=false
viewer.external.classpaths=extlib/*
viewer.servlet.class=azkaban.viewer.hdfs.HdfsBrowserServlet
#hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_1_0

##hadoop2.x 需要用这个管理器
hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0
azkaban.should.proxy=false
proxy.user=azkaban
proxy.keytab.location=
allow.group.proxy=false
file.max.lines=1000
注意：一定要在/etc/profile 里面配置HADOOP_HOME 和 HADOOP_CONF_DIR
  • (2) 配置javaviewer插件
1
2
3
4
5
cd plugins/viewer/
##解压文件
tar -zxvf azkaban-javaviewer-3.0.0.tar.gz
##重命名插件
mv azkaban-javaviewer-3.0.0 javaviewer

修改配置文件: javaviewer/conf/plugin.properties

1
2
3
4
5
6
7
8
viewer.name=JavaViewer
viewer.path=javaviewer
viewer.order=1
viewer.hidden=false
#viewer.external.classpaths=extlib/*
viewer.external.classpaths=extlib
viewer.servlet.class=azkaban.viewer.javaviewer.JavaViewerServlet
viewer.jobtypes=hadoopJava
  • (3) 配置jobsummary插件
1
2
3
4
5
cd plugins/viewer/
##解压文件
tar -zxvf azkaban-jobsummary-3.0.0.tar.gz
##重命名插件
mv azkaban-jobsummary-3.0.0 jobsummary

修改配置文件: jobsummary/conf/plugin.properties

1
2
3
4
5
6
7
viewer.name=Summary
viewer.path=jobsummary
viewer.order=1
viewer.hidden=true
viewer.external.classpaths=extlib
viewer.servlet.class=azkaban.viewer.jobsummary.JobSummaryServlet
viewer.jobtypes=java,hadoopJava,pig,hive
  • (4) 配置reportal插件
1
2
3
4
5
cd plugins/viewer/
##解压文件
tar -zxvf azkaban-reportal-3.0.0.tar.gz
##重命名插件
mv azkaban-reportal-3.0.0 reportal

修改配置文件: reportal/conf/plugin.properties

1
2
3
4
5
6
7
8
9
10
11
12
viewer.name=Reportal
viewer.path=reportal
viewer.order=1
viewer.hidden=false
viewer.external.classpaths=extlib
viewer.servlet.class=azkaban.viewer.reportal.ReportalServlet
azkaban.should.proxy=true
proxy.user=azkaban
proxy.keytab.location=
allow.group.proxy=true
reportal.output.filesystem=hdfs
hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0

3. 配置azkaban-exec 插件

  • (1) 配置azkaban-jobtype和reportal 插件
1
2
3
4
5
cd plugins/jobtypes/

##解压文件
tar -zxvf azkaban-jobtype-3.0.0.tar.gz
tar -zxvf azkaban-reportal-3.0.0.tar.gz

修改配置文件: plugins/jobtypes/common.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
azkaban.should.proxy=false
## reportal 插件配置
reportal.execution.user=azkaban
#reportal.storage.user=reportal
#reportal.default.timezone=America/Los_Angeles
reportal.default.timezone=Asia/Shanghai

##必须设置hdfs，否则azkaban-web 接收不到文件
reportal.output.filesystem=hdfs
#reportal.output.dir=/tmp/reportal

## pig,hive,spark,hadoop 的home路径配置
#pig.home=
hadoop.home=/opt/cloudera/parcels/CDH/lib/hadoop
hive.home=/opt/cloudera/parcels/CDH/lib/hive
spark.home=/opt/cloudera/parcels/CDH/lib/spark

##通用hadoop包，引用配置
hadoop.classpath=${hadoop.home}/etc/hadoop,${hadoop.home}/share/hadoop/common/*,${hadoop.home}/share/hadoop/common/lib/*,${hadoop.home}/share/hadoop/hdfs/*,${hadoop.home}/share/hadoop/hdfs/lib/*,${hadoop.home}/share/hadoop/yarn/*,${hadoop.home}/share/hadoop/yarn/lib/*,${hadoop.home}/share/hadoop/mapreduce/*,${hadoop.home}/share/hadoop/mapreduce/lib/*,${hadoop.home}/client-0.20/*

## 通用jar包引用配置
jobtype.global.classpath=${hadoop.home}/etc/hadoop,${hadoop.home}/share/hadoop/common/*,${hadoop.home}/share/hadoop/common/lib/*,${hadoop.home}/share/hadoop/hdfs/*,${hadoop.home}/share/hadoop/hdfs/lib/*,${hadoop.home}/share/hadoop/yarn/*,${hadoop.home}/share/hadoop/yarn/lib/*,${hadoop.home}/share/hadoop/mapreduce/*,${hadoop.home}/share/hadoop/mapreduce/lib/*,${hadoop.home}/client-0.20/*,/opt/cloudera/parcels/CDH-5.15.2-1.cdh5.15.2.p0.3/lib/avro/*,${hive.home}/conf/

## configs for jobtype security settings
execute.as.user=false

修改配置文件: plugins/jobtypes/commonprivate.properties

1
2
3
4
5
6
7
8
9
10
11
12

hadoop.security.manager.class=azkaban.security.HadoopSecurityManager_H_2_0

 azkaban.should.proxy=false
 obtain.binary.token=false
# obtain.namenode.token=true
# obtain.jobtracker.token=true

# global classpath items for all jobs. e.g. hadoop-core jar, hadoop conf
#jobtype.global.classpath=${hadoop.home}/*,${hadoop.home}/conf
# global jvm args for all jobs. e.g. java.io.temp.dir, java.library.path
#jobtype.global.jvm.args=
  • (2) jobtypes hive 配置修改

修改 hive/plugin.properties

1
2
3
4
hive.aux.jars.path=${hive.home}/aux/lib

hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}
jobtype.jvm.args=${hive.jvm.args}

修改 hive/private.properties

1
2
3
4
5
6
7
8
9
jobtype.classpath=${hadoop.home}/conf,${hadoop.home}/lib/*,${hive.home}/lib/*,${hive.home}/conf,${hive.aux.jar.path}
jobtype.class=azkaban.jobtype.HadoopHiveJob

hive.aux.jar.path=${hive.home}/aux/lib

## definitely set hive home otherwise we dont have conf

#jobtype.global.classpath=${hive.classpath.items}
#hive.classpath.items=${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${hadoop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/
  • (3) jobtypes reportalhive 配置修改

修改 reportalhive/plugin.properties

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
job.class=azkaban.jobtype.ReportalHiveRunner

##不知道为什么,commonprivate.properties中的通用设置不好使
hive.home=/opt/cloudera/parcels/CDH/lib/hive
hadoop.home=/opt/cloudera/parcels/CDH/lib/hadoop
CDH-5.15.2-1.cdh5.15.2.p0.3

env.HIVE_HOME=${hive.home}
env.HIVE_AUX_JARS_PATH=${hive.aux.jars.path}

hive.jvm.args=-Dhive.querylog.location=. -Dhive.exec.scratchdir=/tmp/hive-${user.to.proxy} -Dhive.aux.jars.path=${hive.aux.jars.path}
jobtype.global.jvm.args=${hive.jvm.args}
hive.aux.jars.path=${hive.home}/aux/lib

hadoop.dir.bin=${hadoop.home}/bin
hadoop.dir.conf=${hadoop.home}/etc/hadoop
不想编译的话可以从我的百度云盘获取：azkaban-3.49.0 ,提取码：* (联系我)

源码地址

三. azkaban 运行

1. azkaban-exec 服务启动

1
2
cd azkaban-exec-server-3.49.0
sh ./bin/start-exec.sh

executors 表中的active 字段，修改成1。否则web服务启动会报错

2. azkaban-web 服务启动

1
2
cd azkaban-web-server-3.49.0
sh ./bin/start-web.sh

3. 运行效果

扫一扫，分享到微信

微信分享二维码

tag:

    缺失模块。
    1、请确保node版本大于6.2
    2、在博客根目录（注意不是yilia根目录）执行以下命令：
    npm i hexo-generator-json-content --save

    3、在根目录_config.yml里添加配置： 

      jsonContent:
    meta: false
    pages: false
    posts:
      title: true
      date: true
      path: true
      text: false
      raw: false
      content: false
      slug: false
      updated: false
      comments: false
      link: false
      permalink: false
      excerpt: false
      categories: false
      tags: true

健康快乐