参考：https://www.cloudera.com/documentation/enterprise/6/latest/topics/installation.html
简介
cm下载地址
https://archive.cloudera.com/cm6/6.2.0/redhat7/yum/RPMS/x86_64/
把该目录下的文件全部下载下来
CDH下载地址
https://archive.cloudera.com/cdh6/6.2.0/parcels/
安装前准备
修改hadoop用户的sudo权限
在 root 账号下，命令终端输入：vi /etc/sudoers
找到 root ALL=(ALL) ALL 这一行，
然后在他下面添加一行：
hadoop ALL=(ALL) ALL
配置ssh免密码登录
- 修改master 上的 etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.100.21 master
192.168.100.22 slaver1
192.168.100.23 slaver2
- scp到其他节点
scp /etc/hosts root@slaver2:/etc/hosts
- 在master机器通过hadoop登录，生成密钥
ssh-keygen -t rsa
[root@localhost /]# su hadoop
[hadoop@localhost /]$ ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/hadoop/.ssh/id_rsa):
Created directory '/home/hadoop/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/hadoop/.ssh/id_rsa.
Your public key has been saved in /home/hadoop/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:k6hBfjUWLqDJZue8bhXGprld7d8b3If8p7B9M6TFJrs hadoop@localhost.localdomain
The key's randomart image is:
+---[RSA 2048]----+
| . . |
| . o . . . |
| * o.. = |
| o * == o |
| ==o.S. . |
| o=. ... + B |
| o+ . . . % o|
| .o . . B =+|
| .. .oE*+=|
+----[SHA256]-----+
- 把公钥拷贝到其他节点
ssh-copy-id -i ~/.ssh/id_rsa.pub slave1
[hadoop@localhost /]$ ssh-copy-id -i ~/.ssh/id_rsa.pub slaver2
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/hadoop/.ssh/id_rsa.pub"
The authenticity of host 'slaver2 (192.168.100.23)' can't be established.
ECDSA key fingerprint is SHA256:ZiyMwPpiMB9MQAhzXWqqHX8m8rH8d7TRAI/mYiVWP74.
ECDSA key fingerprint is MD5:8a:6f:7c:04:f8:9e:8c:e9:28:50:f1:50:33:8c:2c:1e.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
hadoop@slaver2's password:
Number of key(s) added: 1
Now try logging into the machine, with: "ssh 'slaver2'"
and check to make sure that only the key(s) you wanted were added.
- 这样之后就可以免密码登录到其他节点
[hadoop@localhost /]$ ssh slaver1
Last login: Sat Mar 30 19:49:52 2019
[hadoop@localhost ~]$ exit
登出
Connection to slaver1 closed.
[hadoop@localhost /]$ ssh slaver2
Last login: Sun Mar 31 19:50:59 201
安装jdk
yum search jdk
yum install -y java-1.8.0-openjdk-devel.x86_64
安装配置mysql
https://www.cloudera.com/documentation/enterprise/6/latest/topics/cm_ig_mysql.html#cmig_topic_5_5
- 安装mysql
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update
sudo yum install mysql-server
sudo systemctl start mysqld
- 配置并启动mysql
1 如果mysql已经启动，则关闭mysql
sudo systemctl stop mysqld
2 配置mysql
找到配置文件
/etc/my.cnf
推荐配置如下
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
transaction-isolation = READ-COMMITTED
# Disabling symbolic-links is recommended to prevent assorted security risks;
# to do so, uncomment this line:
symbolic-links = 0
key_buffer_size = 32M
max_allowed_packet = 32M
thread_stack = 256K
thread_cache_size = 64
query_cache_limit = 8M
query_cache_size = 64M
query_cache_type = 1
max_connections = 550
#expire_logs_days = 10
#max_binlog_size = 100M
#log_bin should be on a disk with enough free space.
#Replace '/var/lib/mysql/mysql_binary_log' with an appropriate path for your
#system and chown the specified folder to the mysql user.
log_bin=/var/lib/mysql/mysql_binary_log
#In later versions of MySQL, if you enable the binary log and do not set
#a server_id, MySQL will not start. The server_id must be unique within
#the replicating group.
server_id=1
binlog_format = mixed
read_buffer_size = 2M
read_rnd_buffer_size = 16M
sort_buffer_size = 8M
join_buffer_size = 8M
# InnoDB settings
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 64M
innodb_buffer_pool_size = 4G
innodb_thread_concurrency = 8
innodb_flush_method = O_DIRECT
innodb_log_file_size = 512M
[mysqld_safe]
log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid
sql_mode=STRICT_ALL_TABLES
- 开机自启动
sudo systemctl enable mysqld
- 启动mysql
sudo systemctl start mysqld
- 设置root密码和其他安全设置
sudo /usr/bin/mysql_secure_installation
Enter current password for root (enter for none):
OK, successfully used password, moving on...
[...]
Set root password? [Y/n] **Y**
New password:
Re-enter new password:
Remove anonymous users? [Y/n] **Y**
[...]
Disallow root login remotely? [Y/n] **N**
[...]
Remove test database and access to it [Y/n] **Y**
[...]
Reload privilege tables now? [Y/n] **Y**
All done!
安装mysql客户端
- 下载
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
Download the MySQL JDBC driver from http://www.mysql.com/downloads/connector/j/5.1.html (in .tar.gz format). As of the time of writing, you can download version 5.1.46 using wget as follows:
wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz
Extract the JDBC driver JAR file from the downloaded file. For example:
tar zxvf mysql-connector-java-5.1.46.tar.gz
Copy the JDBC driver, renamed, to /usr/share/java/. If the target directory does not yet exist, create it. For example:
sudo mkdir -p /usr/share/java/
cd mysql-connector-java-5.1.46
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar
为CDH软件安装数据库
- 登录
mysql -u root -p
hadoop
CREATE DATABASE <database> DEFAULT CHARACTER SET <character set> DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
GRANT ALL ON <database>.* TO '<user>'@'%' IDENTIFIED BY '<password>';
Query OK, 0 rows affected (0.00 sec)
Databases for Cloudera Software
Confirm that you have created all of the databases:
SHOW DATABASES;
You can also confirm the privilege grants for a given user by running:
SHOW GRANTS FOR '<user>'@'%';
image.png
e.g
mysql> CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci;
Query OK, 1 row affected (0.00 sec)
mysql> GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'hadoop';
Query OK, 0 rows affected (0.05 sec)
关闭防火墙
service iptables stop
chkconfig iptables off
CM-server安装
采用离线方式安装
cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
依次安装上面三个工具，在server端需要安装上面三个，slaver端需要安装agent和daemons即可
安装agent时需要依赖其他包
所以提前运行以下命令
yum install -y bind-utils psmisc cyrus-sasl-plain cyrus-sasl-gssapi portmap /lib/lsb/init-functions httpd mod_ssl openssl-devel python-psycopg2 MySQL-python libpg.so.5
rpm -ivh cloudera-manager-daemons-6.2.0-968826.el7.x86_64.rpm
rpm -ivh cloudera-manager-agent-6.2.0-968826.el7.x86_64.rpm
rpm -ivh cloudera-manager-server-6.2.0-968826.el7.x86_64.rpm
安装之后的目录为
CDH安装前准备
在mysql创建用户
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav
sudo /opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms
安装CDH
- 启动scm-server
sudo systemctl start cloudera-scm-server
image.png
通过admin/admin登录 依次进行操作，在“Select Edition”中选择免费版本，目前的免费版本不限制节点个数，核心功能都具备，不包含navigator功能
image.png
image.png
目录
相关目录 /var/log/cloudera-scm-installer : 安装日志目录。
/var/log/* : 相关日志文件（相关服务的及CM的）。
/usr/share/cmf/ : 程序安装目录。
/usr/lib64/cmf/ : Agent程序代码。
/var/lib/cloudera-scm-server-db/data : 内嵌数据库目录。
/usr/bin/postgres : 内嵌数据库程序。
/etc/cloudera-scm-agent/ : agent的配置目录。
/etc/cloudera-scm-server/ : server的配置目录。
/opt/cloudera/parcels/ : Hadoop相关服务安装目录。
/opt/cloudera/parcel-repo/ : 下载的服务软件包数据，数据格式为parcels。
/opt/cloudera/parcel-cache/ : 下载的服务软件包缓存数据。
/etc/hadoop/* : 客户端配置文件目录。
