硬件配置:
阿里云(强调!!!!!!!!!!! 虚拟机请自行配置静态ip)
| 物理CPU数 | 逻辑CPU总核数 | 内存 | 存储 | CPU |
|---|---|---|---|---|
| 1--4核 | 8核 | 64G | 500G | Intel(R) Xeon(R) CPU E5-2682 v4 @ 2.50GHz |
操作系统：
Centos 7.4
CDH版本:
5.14
查看centos版本:
cat /etc/redhat-release
查看内核版本信息
uname -a
查看内网ip
ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno16780032: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 00:0c:29:7f:5c:48 brd ff:ff:ff:ff:ff:ff
inet ************** brd ************ scope global eno16780032
valid_lft forever preferred_lft forever
inet6 fe80::20c:29ff:fe7f:5c48/64 scope link
valid_lft forever preferred_lft forever
集群规划：
| 集群 | 角色 |
|---|---|
| hadoop01 | master |
| hadoop02 | slave1 |
| hadoop03 | slave2 |
| hadoop04 | slave3 |
基础环境配置：
1.集群免密登录:
①修改主机名
查看主机名:
hostname
临时修改主机名:
hostname hadoop01
hostname hadoop02
hostname hadoop03
hostname hadoop04
永久修改主机名:
[root@hadoop01 ~]# vi /etc/sysconfig/network
hostname=hadoop01
[root@hadoop02 ~]# vi /etc/sysconfig/network
hostname=hadoop02
[root@hadoop03 ~]# vi /etc/sysconfig/network
hostname=hadoop03
[root@hadoop04 ~]# vi /etc/sysconfig/network
hostname=hadoop04
② hosts文件配置:
vi /etc/hosts
hadoop01 hadoop02 hadoop03 都修改同一的配置(内网ip 主机名 域名)
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.184.11 hadoop01 www.xy01.com
192.168.184.12 hadoop02 www.xy02.com
192.168.184.13 hadoop03 www.xy03.com
192.168.184.14 hadoop04 www.xy04.com
③ssh 免密配置:
两两配置免密:
1.生成密钥(每台机器上都执行):
ssh-keygen -t rsa
ssh-copy-id hadoop01
ssh-copy-id hadoop02
ssh-copy-id hadoop03
ssh-copy-id hadoop03
2.JDK.1.8.0_191 安装:
①解压
tar -zxvf jdk-8u191-linux-x64.tar.gz -C /usr/local/
②配置环境变量
export JAVA_HOME=/usr/local/jdk1.8.0_191
export CLASSPATH=$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin
配置完之后 执行
source /etc/profile
使配置文件生效
3.时间同步设置:
(若有则无需设置)
查看时间
[root@hadoop01 home]# date
Fri Feb 22 22:58:41 CST 2019
[root@hadoop02 home]# date
Fri Feb 22 22:58:45 CST 2019
[root@hadoop03 home]# date
Fri Feb 22 22:58:50 CST 2019
[root@hadoop04 home]# date
Fri Feb 22 22:58:57 CST 2019
安装ntp(hadoop01 hadoop02 hadoop03 hadoop04)
yum install ntp
chkconfig ntpd on
ntpdate -u ntp.sjtu.edu.cn
*/2 * * * * /usr/sbin/ntpdate -u ntp.sjtu.edu.cn >> /dev/null 2>&1
date 查看是否同步
4.关闭防火墙和selinux:
查看防火墙状态:s
systemctl status firewalld.service
关闭防火墙:
systemctl stop firewalld.service
永久防火墙开机自启动:
systemctl disable firewalld
开机关闭SELinux:
SELINUX=disabled
vi /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
CDH 5.14.4 在线安装:
设置(所有节点)yum源:
1、打开centos的yum文件夹
输入命令
cd /etc/yum.repos.d/
2、用wget下载repo文件
输入命令:
wget http://mirrors.aliyun.com/repo/Centos-7.repo
如果wget命令不生效，说明还没有安装wget工具，输入
yum -y install wget
当前目录是/etc/yum.repos.d/，刚刚下载的Centos-7.repo也在这个目录上
3、备份系统原来的repo文件
mv CentOs-Base.repo CentOs-Base.repo.bak
即是重命名 CentOs-Base.repo -> CentOs-Base.repo.bak
4、替换系统原理的repo文件
mv Centos-7.repo CentOs-Base.repo
即是重命名 Centos-7.repo -> CentOs-Base.repo
5、执行yum源更新命令
yum clean all
yum makecache
yum update
主节点hadoop01 yum安装mysql:
下载mysql源:
wget http://dev.mysql.com/get/mysql57-community-release-el7-7.noarch.rpm
安装到yum 源:
yum localinstall -y mysql57-community-release-el7-7.noarch.rpm
安装MySQL:
yum install -y mysql-community-server
启动MySQL服务:
systemctl start mysqld.service
查看初始密码
grep 'temporary password' /var/log/mysqld.log
修改密码:
mysql -u root -p 初始密码
SET PASSWORD = PASSWORD('your new password');
ALTER USER 'root'@'localhost' PASSWORD EXPIRE NEVER;
flush privileges;
设置远程访问权限:
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY 'your password' WITH GRANT OPTION;
java 设置(所有节点)
cdh启用时, 默认会从/usr/java/default路径读取java环境, 所以此处需要做个软链
#生成软链
mkdir /usr/java
ln -s /usr/local/jdk1.8.0_191 /usr/java/default
登录mysql添加数据库, 用户并授权:
create database cm DEFAULT CHARACTER SET utf8
GRANT ALL PRIVILEGES ON cm.* TO 'cm'@'%' IDENTIFIED BY 'admin123@XY'
hadoop上执行
下载对应系统cdh官方yum源:
cd /etc/yum.repos.d/
wget http://archive.cloudera.com/cm5/redhat/7/x86_64/cm/cloudera-manager.repo
wget https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/cloudera-cdh5.repo
修改yum源, 配置cdh的版本(默认是最新的)
vim /etc/yum.repos.d/cloudera-cdh5.repo
vim /etc/yum.repos.d/cloudera-manager.repo
[cloudera-manager]
# Packages for Cloudera Manager, Version 5, on RedHat or CentOS 7 x86_64
name=Cloudera Manager
baseurl=https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/5.14/
gpgkey =https://archive.cloudera.com/cm5/redhat/7/x86_64/cm/RPM-GPG-KEY-cloudera
gpgcheck = 1
[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat or CentOS 7 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.14/
gpgkey =https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1
安装 cm server
yum install cloudera-manager-daemons cloudera-manager-server -y
修改配置文件
vim /etc/cloudera-scm-server/db.properties
com.cloudera.cmf.db.type=mysql
# The database host
# If a non standard port is needed, use 'hostname:port'
com.cloudera.cmf.db.host=192.168.184.11:3306
# The database name
com.cloudera.cmf.db.name=cm
# The database user
com.cloudera.cmf.db.user=cm
# The database user's password
com.cloudera.cmf.db.password=admin23@XY
尝试启动
/etc/init.d/cloudera-scm-server restart
查看日志是否报错
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log
如果启动失败, 可能报错jdbc驱动找不到
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.47.tar.gz
解压后,将其jar包复制到cdh公共lib目录下
mv mysql-connector-java-5.1.47-bin.jar mysql-connector-java.jar
cp mysql-connector-java.jar /usr/share/cm/lib/
cp mysql-connector-java.jar /usr/share/cm/common_jars/
cp mysql-connector-java.jar $JAVA_HOME/lib/
若启动成功则:
在页面访问
192.168.184.11:7180
可能出现的问题:
待续>>>>>>>>>>>>>>
