本文章仅作为个人笔记
CDH官网地址:https://www.cloudera.com/
CDH镜像地址:
cm5.11.1地址:http://archive.cloudera.com/cm5/repo-as-tarball/5.11.1/cm5.11.1-centos7.tar.gz
http://archive.cloudera.com/cm5/repo-as-tarball/5.11.1/cm5.11.1-centos7.tar.gz.sha1
cdh5.11.1地址:http://archive.cloudera.com/cdh5/parcels/5.11.1/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel
http://archive.cloudera.com/cdh5/parcels/5.11.1/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha1
http://archive.cloudera.com/cdh5/parcels/5.11.1/manifest.json
jdk1.8.121官网下载地址:http://download.oracle.com/otn/java/jdk/8u121-b13/e9e7ea248e2c4826b92b3f075a80e441/jdk-8u121-linux-i586.rpm
mysql5.6.36官网下载地址:https://dev.mysql.com/get/Downloads/MySQL-5.6/mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz
mysql连接驱动下载地址:https://dev.mysql.com/downloads/connector/j/
笔者使用VMware进行模拟安装,为了满足基础配置,这里分配3台虚拟机(hadoopb01(4g/4核/40g/192.168.75.136/master),hadoopb02(2g/2核/40g/192.168.75.144/slave),hadoopb03(2g/2核/40g/192.168.75.145/slave)),另外wget下载文件非常缓慢并且不方便反复测试及安装,笔者这里下载好所有资源文件放在本地(.sha1文件结尾的文件统一重命名为.sha)。(如果虚拟机开机后无网络(使用net桥接方式),可与命令行运行dhclient分配ip,若后续断开了可运行dhclient -r后再运行dhclient)
在3台服务器共同操作:
安装基本工具: yum install vim wget net-tools unzip -y
关闭SELINUX:
运行 setenforce 0 临时关闭
永久关闭SELINUX:
echo "SELINUX=disabled" > /etc/selinux/config
echo "SELINUXTYPE=targeted" >> /etc/selinux/config
关闭iptables防火墙:
运行 systemctl stop firewalld.service 停止防火墙
运行 systemctl status firewalld.service 查看防火墙状态
运行 systemctl disable firewalld.service 关闭防火墙开机启动
更改主机名:
运行 hostname hadoopb01 更改主机名,根据不同主机选择不同主机名(hadoopb01/hadoopb02/hadoopb03)
运行 echo "HOSTNAME=hadoopb01" > /etc/sysconfig/network ,根据不同主机选择不同主机名(hadoopb01/hadoopb02/hadoopb03)
运行 echo "hadoopb01" > /etc/hostname ,根据不同主机选择不同主机名(hadoopb01/hadoopb02/hadoopb03)
安装jdk(参考http://www.jianshu.com/p/c5fc4dd47754)
密钥生成:
运行 ssh-keygen 一路回车回车生成
ip与hostname绑定:
echo "192.168.75.136 hadoopb01" >> /etc/hosts
echo "192.168.75.144 hadoopb02" >> /etc/hosts
echo "192.168.75.145 hadoopb03" >> /etc/hosts
运行python --version 检查python版本,如果不是python2.6.6建议改为python2.6.6
时间同步:
运行 yum install ntp -y 安装ntp服务
运行 systemctl enable ntpd 使ntp服务开机启动
在从服务器操作(hadoopb02(2g/2核/40g/192.168.75.144/slave),hadoopb03(2g/2核/40g/192.168.75.145/slave):
配置时钟同步:
echo "driftfile /var/lib/ntp/ntp.drift" > /etc/ntp.conf
echo "statistics loopstats peerstats clockstats" >> /etc/ntp.conf
echo "filegen loopstats file loopstats type day enable" >> /etc/ntp.conf
echo "filegen peerstats file peerstats type day enable" >> /etc/ntp.conf
echo "filegen clockstats file clockstats type day enable" >> /etc/ntp.conf
echo "server 192.168.75.136 prefer" >> /etc/ntp.conf
echo "fudge 192.168.75.136 stratum 5" >> /etc/ntp.conf
echo "restrict -4 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "restrict -6 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "restrict 127.0.0.1" >> /etc/ntp.conf
echo "restrict ::1" >> /etc/ntp.conf
运行 service ntpd restart 重启ntpd并使配置生效
运行 ntpdate –u 192.168.75.136 同步时间
在主服务器操作(hadoopb01(4g/4核/40g/192.168.75.136/master):
运行 ssh-copy-id hadoopb01
ssh-copy-id hadoopb02
ssh-copy-id hadoopb03 完成各服务器免密登录
配置时钟同步:
echo "driftfile /var/lib/ntp/ntp.drift" > /etc/ntp.conf
echo "restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap" >> /etc/ntp.conf
echo "server 210.72.145.44" >> /etc/ntp.conf
echo "server 202.112.10.36 perfer" >> /etc/ntp.conf
echo "server 59.124.196.83" >> /etc/ntp.conf
echo "restrict 210.72.145.44 nomodify notrap noquery" >> /etc/ntp.conf
echo "restrict 202.112.10.36 nomodify notrap noquery" >> /etc/ntp.conf
echo "restrict 59.124.196.83 nomodify notrap noquery" >> /etc/ntp.conf
echo "server 127.127.1.0" >> /etc/ntp.conf
echo "fudge 127.127.1.0 stratum 10" >> /etc/ntp.conf
运行 service ntpd restart 重启ntpd并使配置生效
运行 ntpdate –u 202.112.10.36 同步时间
使用filezilla将上文所述文件传输至主服务器(放置于root用户~目录)
安装配置mysql5.6(可参考http://www.jianshu.com/p/c5fc4dd47754)
创建相关用户及数据库:
create database cmf default character set utf8
grant all on cmf.* to 'cmf'@'localhost' identified by 'cmf_password'
grant all on cmf.* to 'cmf'@'%' identified by 'cmf_password'
grant all on cmf.* to 'cmf'@'${m_hosts[${c_index}]}' identified by 'cmf_password'
drop database cmf
create database cmf
create database hive default character set utf8
grant all on hive.* to 'hive'@'localhost' identified by 'hive_password'
grant all on hive.* to 'hive'@'%' identified by 'hive_password'
grant all on hive.* to 'hive'@'${m_hosts[${c_index}]}' identified by 'hive_password'
drop database hive
create database hive
create database hue default character set utf8
grant all on hue.* to 'hue'@'localhost' identified by 'hue_password'
grant all on hue.* to 'hue'@'%' identified by 'hue_password'
grant all on hue.* to 'hue'@'${m_hosts[${c_index}]}' identified by 'hue_password'
drop database hue
create database hue
create database am default character set utf8
grant all on am.* to 'am'@'localhost' identified by 'am_password'
grant all on am.* to 'am'@'%' identified by 'am_password'
grant all on am.* to 'am'@'${m_hosts[${c_index}]}' identified by 'am_password'
drop database am
create database am
create database os default character set utf8
grant all on os.* to 'os'@'localhost' identified by 'os_password'
grant all on os.* to 'os'@'%' identified by 'os_password'
grant all on os.* to 'os'@'${m_hosts[${c_index}]}' identified by 'os_password'
drop database os
create database os
flush privileges
安装并启动httpd服务:
运行 yum install -y httpd 安装httpd服务
运行 systemctl enable httpd 设置httpd服务开机启动
运行 service httpd start 启动httpd服务
创建parcels服务(依次运行如下命令):
cd /var/www/html
mkdir parcels
cd parcels
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel ./
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha ./
mv ~/manifest.json ./
创建cm5服务(依次运行如下命令):
mkdir /opt/rpminstall
cd /opt/rpminstall
mv ~/cm5.11.1-centos7.tar.gz ./
mv ~/cm5.11.1-centos7.tar.gz.sha ./
tar -zxf cm5.11.1-centos7.tar.gz -C /var/www/html/
cd /var/www/html/
mkdir -p cm5/redhat/7/x86_64/
mv cm cm5/redhat/7/x86_64/
chown apache:apache -R /var/www/html
cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64
yum install -y cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
yum install -y cloudera-manager-server-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
mkdir /usr/share/java
cd /usr/share/java
mv ~/mysql-connector-java-5.1.42.zip ./
unzip mysql-connector-java-5.1.42.zip
cd mysql-connector-java-5.1.42
cp mysql-connector-java-5.1.42-bin.jar ../mysql-connector-java.jar
echo "com.cloudera.cmf.db.type=mysql" > /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.host=localhost" >> /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.name=cmf" >> /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.user=cmf" >> /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.password=cmf_password" >> /etc/cloudera-scm-server/db.properties
运行 service cloudera-scm-server start 启动 cloudera-scm 服务
运行 tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log 查看运行日志,待日志停止且显示Started Jetty server.表示服务运行完成。
与浏览器输入 http://192.168.75.136:7180开始正式的集群部署
至此完成安装
文章的最后附上个人的脚本(吐槽下简书的书写为了格式会导致粘贴部分文案行不对号,如果要使用这个脚本需要更改( 主机ip,主机名)):
最后为了使脚本能够运行,需要运行 chmod +x filepath 为文件添加权限,运行时执行 bash filepath n 例如(bash cdh.sh 0),运行此脚本可能mysql初始化不完全,如果可以,最好还是弄懂所有的步骤。
====================================================================================================master
#!/bin/bash
c_index=$1
m_names=("master" "slave01" "slave02")
m_hosts=("192.168.1.131" "192.168.1.129" "192.168.1.226")
function auto_ssh_keygen(){
expect -c "set timeout -1;
spawn ssh-keygen;
expect {
*id_rsa* {
send \n\r;
exp_continue;
};
*Overwrite* {
send y\n\r;
exp_continue;
};
*passphrase* {
send \n\r;
exp_continue;
};
*again* {
send \n\r;
exp_continue;
};
};
"
}
function auto_ssh_copy(){
expect -c "set timeout -1;
spawn ssh-copy-id $1;
expect {
*yes/no* {
send yes\n\r;
exp_continue;
};
*password* {
send $2\n\r;
exp_continue;
};
};
"
}
#初始化必要工具
yum install expect -y
yum install unzip -y
#关闭selinux
setenforce 0
echo "SELINUX=disabled" > /etc/selinux/config
echo "SELINUXTYPE=targeted" >> /etc/selinux/config
#关闭防火墙
systemctl stop firewalld.service
systemctl status firewalld.service
systemctl disable firewalld.service
#定义主机名
hostname ${m_names[${c_index}]}
echo "HOSTNAME=${m_names[${c_index}]}" > /etc/sysconfig/network
#ip与hostname绑定
echo "${m_names[${c_index}]}" > /etc/hostname
t_index=0
for m_name in ${m_names[@]};
do
echo "${m_hosts[t_index]} $m_name" >> /etc/hosts
let t_index++
done
#时间同步
yum install ntp -y
systemctl enable ntpd
if [ ${c_index} == 0 ]
then
echo "driftfile /var/lib/ntp/ntp.drift" > /etc/ntp.conf
echo "restrict 192.168.75.0 mask 255.255.255.0 nomodify notrap" >> /etc/ntp.conf
echo "server 210.72.145.44" >> /etc/ntp.conf
echo "server 202.112.10.36 perfer" >> /etc/ntp.conf
echo "server 59.124.196.83" >> /etc/ntp.conf
echo "restrict 210.72.145.44 nomodify notrap noquery" >> /etc/ntp.conf
echo "restrict 202.112.10.36 nomodify notrap noquery" >> /etc/ntp.conf
echo "restrict 59.124.196.83 nomodify notrap noquery" >> /etc/ntp.conf
echo "server 127.127.1.0" >> /etc/ntp.conf
echo "fudge 127.127.1.0 stratum 10" >> /etc/ntp.conf
service ntpd restart
ntpdate –u 202.112.10.36
#免密登陆
for m_name in ${m_names[@]};
do
auto_ssh_copy $m_name root
done
else
echo "driftfile /var/lib/ntp/ntp.drift" > /etc/ntp.conf
echo "statistics loopstats peerstats clockstats" >> /etc/ntp.conf
echo "filegen loopstats file loopstats type day enable" >> /etc/ntp.conf
echo "filegen peerstats file peerstats type day enable" >> /etc/ntp.conf
echo "filegen clockstats file clockstats type day enable" >> /etc/ntp.conf
echo "server ${m_hosts[0]} prefer" >> /etc/ntp.conf
echo "fudge ${m_hosts[0]} stratum 5" >> /etc/ntp.conf
echo "restrict -4 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "restrict -6 default kod notrap nomodify nopeer noquery" >> /etc/ntp.conf
echo "restrict 127.0.0.1" >> /etc/ntp.conf
echo "restrict ::1" >> /etc/ntp.conf
service ntpd restart
ntpdate –u ${m_hosts[0]}
fi
#生成私钥
rm -rf /root/.ssh/id_rsa
rm -rf /root/.ssh/id_rsa.pub
auto_ssh_keygen
#安装jdk
rpm -Uvh jdk-8u121-linux-x64.rpm
echo "JAVA_HOME=/usr/java/jdk1.8.0_121" > /etc/profile.d/java.sh
echo "PATH=\$PATH:\$JAVA_HOME/bin" >> /etc/profile.d/java.sh
echo "CLASSPATH=.:\$JAVA_HOME/lib/jt.jar:\$JAVA_HOME/lib/tools.jar" >> /etc/profile.d/java.sh
echo "export JAVA_HOME PATH CLASSPATH" >> /etc/profile.d/java.sh
source /etc/profile
if [ ${c_index} == 0 ]
then
#安装并初始化配置mysql
tar -zxvf mysql-5.6.36-linux-glibc2.5-x86_64.tar.gz
mv mysql-5.6.36-linux-glibc2.5-x86_64 /usr/local/mysql
touch /etc/my.cnf
echo "[mysql]" > /etc/my.cnf
echo "default-character-set=utf8" >> /etc/my.cnf
echo "socket=/var/lib/mysql/mysql.sock" >> /etc/my.cnf
echo "[mysqld]" >> /etc/my.cnf
echo "skip-name-resolve" >> /etc/my.cnf
echo "port=3306" >> /etc/my.cnf
echo "socket=/var/lib/mysql/mysql.sock" >> /etc/my.cnf
echo "basedir=/usr/local/mysql" >> /etc/my.cnf
echo "datadir=/usr/local/mysql/data" >> /etc/my.cnf
echo "max_connections=200" >> /etc/my.cnf
echo "character-set-server=utf8" >> /etc/my.cnf
echo "default-storage-engine=INNODB" >> /etc/my.cnf
echo "lower_case_table_names=1" >> /etc/my.cnf
echo "max_allowed_packet=16M" >> /etc/my.cnf
mkdir -p /var/lib/mysql/mysql
groupadd mysql
useradd -g mysql mysql
chown -R mysql:mysql /usr/local/mysql
chown -R mysql:mysql /var/lib/mysql
yum install -y perl-Module-Install.noarch
/usr/local/mysql/scripts/mysql_install_db --user=mysql --basedir=/usr/local/mysql
chown -R mysql:mysql /usr/local/mysql
chown 777 /etc/my.cnf
cp /usr/local/mysql/support-files/mysql.server /etc/rc.d/init.d/mysqld
chmod +x /etc/rc.d/init.d/mysqld
chkconfig --add mysqld
echo "export PATH=\$PATH:/usr/local/mysql/bin" > /etc/profile.d/mysql.sh
source /etc/profile
service mysqld restart
/usr/local/mysql/bin/mysql -e "delete from mysql.user where user = ''"
/usr/local/mysql/bin/mysql -e "update mysql.user set password = (select password('root')) where user = 'root'"
/usr/local/mysql/bin/mysql -e "create database cmf default character set utf8"
/usr/local/mysql/bin/mysql -e "grant all on cmf.* to 'cmf'@'localhost' identified by 'cmf_password'"
/usr/local/mysql/bin/mysql -e "grant all on cmf.* to 'cmf'@'%' identified by 'cmf_password'"
/usr/local/mysql/bin/mysql -e "grant all on cmf.* to 'cmf'@'${m_hosts[${c_index}]}' identified by 'cmf_password'"
/usr/local/mysql/bin/mysql -e "drop database cmf"
/usr/local/mysql/bin/mysql -e "create database cmf"
/usr/local/mysql/bin/mysql -e "create database hive default character set utf8"
/usr/local/mysql/bin/mysql -e "grant all on hive.* to 'hive'@'localhost' identified by 'hive_password'"
/usr/local/mysql/bin/mysql -e "grant all on hive.* to 'hive'@'%' identified by 'hive_password'"
/usr/local/mysql/bin/mysql -e "grant all on hive.* to 'hive'@'${m_hosts[${c_index}]}' identified by 'hive_password'"
/usr/local/mysql/bin/mysql -e "drop database hive"
/usr/local/mysql/bin/mysql -e "create database hive"
/usr/local/mysql/bin/mysql -e "create database hue default character set utf8"
/usr/local/mysql/bin/mysql -e "grant all on hue.* to 'hue'@'localhost' identified by 'hue_password'"
/usr/local/mysql/bin/mysql -e "grant all on hue.* to 'hue'@'%' identified by 'hue_password'"
/usr/local/mysql/bin/mysql -e "grant all on hue.* to 'hue'@'${m_hosts[${c_index}]}' identified by 'hue_password'"
/usr/local/mysql/bin/mysql -e "drop database hue"
/usr/local/mysql/bin/mysql -e "create database hue"
/usr/local/mysql/bin/mysql -e "create database am default character set utf8"
/usr/local/mysql/bin/mysql -e "grant all on am.* to 'am'@'localhost' identified by 'am_password'"
/usr/local/mysql/bin/mysql -e "grant all on am.* to 'am'@'%' identified by 'am_password'"
/usr/local/mysql/bin/mysql -e "grant all on am.* to 'am'@'${m_hosts[${c_index}]}' identified by 'am_password'"
/usr/local/mysql/bin/mysql -e "drop database am"
/usr/local/mysql/bin/mysql -e "create database am"
/usr/local/mysql/bin/mysql -e "create database os default character set utf8"
/usr/local/mysql/bin/mysql -e "grant all on os.* to 'os'@'localhost' identified by 'os_password'"
/usr/local/mysql/bin/mysql -e "grant all on os.* to 'os'@'%' identified by 'os_password'"
/usr/local/mysql/bin/mysql -e "grant all on os.* to 'os'@'${m_hosts[${c_index}]}' identified by 'os_password'"
/usr/local/mysql/bin/mysql -e "drop database os"
/usr/local/mysql/bin/mysql -e "create database os"
/usr/local/mysql/bin/mysql -e "flush privileges"
#安装httpd服务
yum install -y httpd
systemctl enable httpd
service httpd restart
cd /var/www/html
mkdir parcels
cd parcels
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel ./
mv ~/CDH-5.11.1-1.cdh5.11.1.p0.4-el7.parcel.sha ./
mv ~/manifest.json ./
mkdir /opt/rpminstall
cd /opt/rpminstall
mv ~/cm5.11.1-centos7.tar.gz ./
mv ~/cm5.11.1-centos7.tar.gz.sha ./
tar -zxf cm5.11.1-centos7.tar.gz -C /var/www/html/
cd /var/www/html/
mkdir -p cm5/redhat/7/x86_64/
mv cm cm5/redhat/7/x86_64/
chown apache:apache -R /var/www/html
cd /var/www/html/cm5/redhat/7/x86_64/cm/5/RPMS/x86_64
yum install -y cloudera-manager-daemons-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
yum install -y cloudera-manager-server-5.11.1-1.cm5111.p0.9.el7.x86_64.rpm
mkdir /usr/share/java
cd /usr/share/java
mv ~/mysql-connector-java-5.1.42.zip ./
unzip mysql-connector-java-5.1.42.zip
cd mysql-connector-java-5.1.42
cp mysql-connector-java-5.1.42-bin.jar ../mysql-connector-java.jar
echo "com.cloudera.cmf.db.type=mysql" > /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.host=localhost" >> /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.name=cmf" >> /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.user=cmf" >> /etc/cloudera-scm-server/db.properties
echo "com.cloudera.cmf.db.password=cmf_password" >> /etc/cloudera-scm-server/db.properties
service cloudera-scm-server restart
source /etc/profile
fi
======================================================================================================end
坑锦集:
1.脚本运行完成后可能需要再次于master运行source /etc/profile 使环境变量生效。
2.如果遇到hive安装失败的问题可能需要再开个浏览器窗口访问cdh搭建主网站更改hive配置,且需要使用独立的mysql数据库及用户。
3.ntp时间同步服务如果报警,可于crontab -e添加定时同步时间。
4.当3台机器服务时,可将master也添加namenode节点。