本节主要内容：

YARN环境搭建

1.系统环境：

OS：CentOS Linux release 7.5.1804 (Core)

CPU：2核心

Memory：1GB

运行用户：root

JDK版本：1.8.0_252

Hadoop版本：cdh5.16.2

2.集群各节点角色规划为：

172.26.37.245 node1.hadoop.com---->namenode，zookeeper，journalnode，hadoop-hdfs-zkfc，resourcenode，historyserver

172.26.37.246 node2.hadoop.com---->datanode，zookeeper，journalnode，nodemanager，hadoop-client，mapreduce

172.26.37.247 node3.hadoop.com---->datanode，nodemanager，hadoop-client，mapreduce

172.26.37.248 node4.hadoop.com---->namenode，zookeeper，journalnode，hadoop-hdfs-zkfc

3.环境说明：

本次追加部署

172.26.37.245 node1.hadoop.com---->resourcenode，historyserver

172.26.37.246 node2.hadoop.com---->nodemanager，hadoop-client，mapreduce

172.26.37.246 node2.hadoop.com---->nodemanager，hadoop-client，mapreduce

node1节点：负责任务调度，资源管理，和任务记录，部署resourcenode，historyserver

node2、node3节点：负责运行任务，部署nodemanager，hadoop-client，mapreduce

一.安装

node1节点

# yum -y install hadoop-yarn-resourcemanager hadoop-mapreduce-historyserver

node2、node3节点

# yum -y install hadoop-yarn-nodemanager hadoop-mapreduce hadoop-client

二.配置

1./etc/hadoop/conf/core-site.xml变更(node1、node2、node3节点)

# cp -p /etc/hadoop/conf/core-site.xml /etc/hadoop/conf/core-site.xml.20200623

# vi /etc/hadoop/conf/core-site.xml

添加如下

<property>

<name>hadoop.proxyuser.mapred.groups</name>

<value>*</value>

</property>



<property>

<name>hadoop.proxyuser.mapred.hosts</name>

<value>*</value>

</property>

2.配置YARN集群的属性及配置JobHistory服务器(node1节点)

# cp -p /etc/hadoop/conf/mapred-site.xml /etc/hadoop/conf/mapred-site.xml.20200623

# vi /etc/hadoop/conf/mapred-site.xml

<configuration>

<property>

<name>mapreduce.framework.name</name>

<value>yarn</value>

</property>



<property>

<name>mapreduce.jobhistory.address</name>

<value>node1.hadoop.com:10020</value>

</property>



<property>

<name>mapreduce.jobhistory.webapp.address</name>

<value>node1.hadoop.com:19888</value>

</property>



<property>

<name>yarn.app.mapreduce.am.staging-dir</name>

<value>/user</value>

</property>



</configuration>

3.配置YARN守护进程(node1、node2、node3节点)

配置以下服务：ResourceManager（在专用主机上）和NodeManager（在计划运行MapReduce v2作业的每台主机上）。

# cp /etc/hadoop/conf/yarn-site.xml /etc/hadoop/conf/yarn-site.xml.20200623

# vi /etc/hadoop/conf/yarn-site.xml

变更后为以下内容

<configuration>

<property>

<name>yarn.nodemanager.aux-services</name>

<value>mapreduce_shuffle</value>

</property>



<property>

<name>yarn.nodemanager.aux-services.mapreduce_shuffle.class</name>

<value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

<property>

<name>yarn.log-aggregation-enable</name>

<value>true</value>

</property>

<property>

<description>List of directories to store localized files in.</description>

<name>yarn.nodemanager.local-dirs</name>

<value>file:///var/lib/hadoop-yarn/cache/${user.name}/nm-local-dir</value>

</property>



<property>

<description>Where to store container logs.</description>

<name>yarn.nodemanager.log-dirs</name>

<value>file:///var/log/hadoop-yarn/containers</value>

</property>



<property>

<description>Where to aggregate logs to.</description>

<name>yarn.nodemanager.remote-app-log-dir</name>

<value>hdfs://cluster1/var/log/hadoop-yarn/apps</value>

</property>





<property>

<description>Classpath for typical applications.</description>

<name>yarn.application.classpath</name>

<value>

$HADOOP_CONF_DIR,

$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,

$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,

$HADOOP_MAPRED_HOME/*,$HADOOP_MAPRED_HOME/lib/*,

$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*

</value>

</property>



<property>

<name>yarn.resourcemanager.hostname</name>

<value>node1.hadoop.com</value>

</property>



</configuration>

4.建立yarn在hdfs上的日志文件夹

# sudo -u hdfs hadoop fs -mkdir -p /var/log/hadoop-yarn

# sudo -u hdfs hadoop fs -chown yarn:mapred /var/log/hadoop-yarn

5.创建实验环境

创建/user目录和及其history子目录；设置权限

# sudo -u hdfs hadoop fs -mkdir /user

# sudo -u hdfs hadoop fs -mkdir -p /user/cloudera

# sudo -u hdfs hadoop fs -chmod -R 1777 /user/cloudera

# sudo -u hdfs hadoop fs -chown mapred:hadoop /user/cloudera

# sudo -u hdfs hadoop fs -mkdir -p /user/hdfs

# sudo -u hdfs hadoop fs -chmod -R 1777 /user/hdfs

# sudo -u hdfs hadoop fs -chown mapred:hadoop /user/hdfs

# sudo -u hdfs hadoop fs -mkdir -p /user/history

# sudo -u hdfs hadoop fs -chmod -R 1777 /user/history

# sudo -u hdfs hadoop fs -chown mapred:hadoop /user/history

# sudo -u hdfs hdfs dfs -ls /

Found 3 items

drwxrwxrwt - hdfs supergroup 0 2020-06-25 11:47 /tmp

drwxr-xr-x - hdfs supergroup 0 2020-06-25 11:57 /user

drwxr-xr-x - hdfs supergroup 0 2020-06-25 11:49 /var

# sudo -u hdfs hadoop dfs -ls /user

DEPRECATED: Use of this script to execute hdfs command is deprecated.

Instead use the hdfs command for it.

Found 3 items

drwxrwxrwt - mapred hadoop 0 2020-06-25 11:57 /user/cloudera

drwxrwxrwt - mapred hadoop 0 2020-06-25 11:56 /user/hdfs

drwxrwxrwt - mapred hadoop 0 2020-06-25 11:59 /user/history

6.启动YARN和MapReduce JobHistory服务器

要启动YARN，必须先启动ResourceManager和NodeManager服务：

同时确保在启动NodeManager服务之前先启动ResourceManager。

在ResourceManager系统上(node1节点)：

# service hadoop-yarn-resourcemanager start

# service hadoop-yarn-resourcemanager status

在每个NodeManager系统上（node2/3节点）：

# service hadoop-yarn-nodemanager start

# service hadoop-yarn-nodemanager status

启动MapReduce JobHistory服务器(node1节点)

# service hadoop-mapreduce-historyserver start

# service hadoop-mapreduce-historyserver status

7.验证history服务

http://172.26.37.245:19888/jobhistory

06.Hadoop:YARN环境搭建