ClouderaManager 基于 Parcel&CSD 安装、监控自定义服务

背景

因业务需求，想通过 ClouderaManager ( 以下简称 CM) 管理、监控自定义服务

准备工作

安装 CM 扩展工具

工具地址

https://github.com/cloudera/cm_ext

安装时需跳过单元测试

mvn install -DskipTests

操作步骤

创建 Extensions (Parcel & CSD)

Parcel

职责

负责分发部署安装包

原理

1. 当parcel 处于可用状态，建立符号链接，以便被系统识别(二进制或文件)

2. 为 CM管理的进程设置扩展的环境变量，当 CM启动一个已经配置好的Parcel 进程时，CM Agent将获取 Parcel 提供的shell脚本，此时脚本可以设置或修改正在启动进程的环境变量

目录结构

├── ECHO-1.0

│ └── meta

│ ├── echo_env.sh

│ └── parcel.json

├── ECHO-1.0-el6.parcel

├── ECHO-1.0-el6.parcel.sha1

└── manifest.json

创建步骤

创建 Parcel

Parcel 名称拟为 ECHO-1.0

mkdir -p ECHO-1.0/meta

创建 parcel.json

{

"schema_version":1,

"name":"ECHO",

"version":"1.0",

"setActiveSymlink":true,

"conflicts":"",

"provides":[

"ECHO"

"scripts":{

"defines":"echo_env.sh"

"packages":[

{

"name":"echo",

"version":"1.0"

}

"components":[

{

"name":"echo",

"version":"1.0",

"pkg_version":"1.0",

"pkg_release":"1.0"

}

"users":{},

"groups":[]

}

创建echo_env.sh

#!/bin/bash

ECHO_DIRNAME=${PARCEL_DIRNAME}

export CDH_ECHO_HOME=$PARCELS_ROOT/$ECHO_DIRNAME

验证文件

java -jar cm_ext/validator/target/validator.jar -p ECHO-1.0/meta/parcel.json

验证目录

java -jar cm_ext/validator/target/validator.jar -d ECHO-1.0

打包 parcel

tar czvf ECHO-1.0-el6.parcel ECHO-1.0 —owner=root —group=root

验证 parcel 包

java -jar cm_ext/validator/target/validator.jar -f ECHO-1.0-el6.parcel

对外提供 Parcel 服务

mkdir parcelService && cd parcelService

python cm_ext/make_manifest/make_manifest.py ./parcelService

把 parcel 包以及 manifest 放到 parcelServer目录

启动服务

python -m SimpleHTTPServer 8001

或以 Apache、Nginx 对外提供服务

CSD ( Custom Service Descriptor )

职责

负责管理和监控

目录结构

├── ECHO-1.0.jar

├── descriptor

│ └── service.sdl

└── scripts

└── control.sh

SDL ( Service Descriptor Language ) 模板

https://github.com/cloudera/cm_ext/wiki/Service-Descriptor-Language-Reference

创建步骤

mkdir csd && cd csd

mkdir descriptor

创建 service.sdl

{

"name" : "ECHO",

"label" : "Echo",

"description" : "The echo service",

"version" : "1.0",

"runAs" : {

"user" : "root",

"group" : "root"

"roles" : [

{

"name" : "ECHO_WEBSERVER",

"label" : "Web Server",

"pluralLabel" : "Web Servers",

"parameters" : [

{

"name" : "port_num",

"label" : "Webserver port",

"description" : "The web server port number",

"required" : "true",

"type" : "port",

"default" : 8080

}

"startRunner" : {

"program" : "scripts/control.sh",

"args" : [ "start" ],

"environmentVariables" : {

"WEBSERVER_PORT" : "${port_num}"

}

]

}

mkdir scripts

创建 control.sh

#!/bin/bash

CMD=$1

case $CMD in

(start)

echo "Starting the web server on port [$WEBSERVER_PORT]"

exec python -m SimpleHTTPServer $WEBSERVER_PORT

;;

(*)

echo "Don't understand [$CMD]"

;;

esac

验证

java -jar cm_ext/validator/target/validator.jar -s csd/descriptor/service.sdl

打jar包

jar -cvf ECHO-1.0.jar *

cp -a ECHO-1.0.jar cm主机/opt/cloudera/csd

重启服务

service cloudera-scm-server restart

参考

https://github.com/cloudera/cm_ext/wiki/CSD-Primer

界面操作

进入 CM

点击右上角 Parcels (礼物形状的 Icon)

点击配置

新增远程 Parcel 存储库 URL

URL必须配置域名，IP 地址无效

更新

重启服务

更新 CSD jar 包

监控

基线监控 Baseline Monitoring (自带，无需在 server.sdl 中配置)

程序指标(句柄、CPU、Memory等)

程序健康测试(进程停止提醒等)

图表

角色和主机指标

创建仪表盘并将其保存为角色页面的默认值

支持的系统版本

el5: Redhat Enterprise Linux 5 and clones (CentOS, Scientific Linux, etc)

el6: Redhat Enterprise Linux 6 and clones (CentOS, Scientific Linux, etc)

el7: Redhat Enterprise Linux 7 and clones (CentOS, Scientific Linux, etc)

sles11: SuSE Linux Enterprise Server 11.x

lucid: Ubuntu Linux 10.04 LTS (No CDH 5.x parcel provided)

precise: Ubuntu Linux 12.04 LTS

trusty: Ubuntu Linux 14.04 LTS (Newly supported in CM 5.2. No CDH 4.x parcel provided)

squeeze: Debian 6.x (No CDH 5.x parcel provided))

wheezy: Debian 7.x (Newly supported in CM 5.0. No CDH 4.x parcel provided)

命名需要加上系统版本

CDH-5.0.0-0.cdh5b1.p0.57-squeeze.parcel

IMPALA-1.2.3-1.p0.97-el6.parcel

GPLEXTRAS-5.5.0-1.cdh5.5.0.p0.7-el7.parcel

参考文档

https://github.com/cloudera/cm_ext

https://github.com/cloudera/cm_csds

如有错误，欢迎指正~