欢迎Follow我的GitHub, 关注我的简书.

配置Hadoop，单机伪分布式模式，用于学习Hadoop的原理，实际工作仍需要Hadoop集群。设备为Mac操作系统。

Hadoop

SSH

开启本机的远程登录，位于 系统偏好设置 -> 共享

SSH

登录本机

➜  ~ ssh localhost
Last login: Sun Sep  3 08:02:53 2017

默认需要键入密码，可以复制已有的SSH密匙至本机的验证密匙，避免每次都需要输入密码。

cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

Homebrew

使用Homebrew安装Hadoop，默认会设置一些系统参数，避免手动修改，Homebrew官网。

执行安装命令

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

安装失败的卸载命令

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/uninstall)"

/Library/Caches/权限不足，导致错误

==> Cleaning up /Library/Caches/Homebrew...
==> Migrating /Library/Caches/Homebrew to /Users/wangchenlong/Library/Caches/Homebrew...
==> Deleting /Library/Caches/Homebrew...
Warning: Failed to delete /Library/Caches/Homebrew.

修改文件夹归属到当前用户

sudo chown -R $USER /usr/local/*
sudo chown -R $USER /Users/wangchenlong/Library/*

安装成功

➜  ~ brew --version
Homebrew 1.3.1
Homebrew/homebrew-core (git revision 1278; last commit 2017-09-02)

在修改用户权限时，导致pip的python实效，需要重新安装pip，报错

pip installation /usr/local/opt/python/bin/python2.7: 
bad interpreter: No such file or directory

修复pip，参考

重新安装pip

curl https://bootstrap.pypa.io/ez_setup.py -o - | sudo python
sudo easy_install pip

Hadoop安装

使用brew安装hadoop

➜  ~ brew install hadoop
==> Downloading https://www.apache.org/dyn/closer.cgi?path=hadoop/common/hadoop-2.8.1/hadoop-2.8.1.tar.gz
==> Best Mirror http://mirrors.tuna.tsinghua.edu.cn/apache/hadoop/common/hadoop-2.8.1/hadoop-2.8.1.tar.gz
######################################################################## 100.0%
==> Caveats
In Hadoop's config file:
  /usr/local/opt/hadoop/libexec/etc/hadoop/hadoop-env.sh,
  /usr/local/opt/hadoop/libexec/etc/hadoop/mapred-env.sh and
  /usr/local/opt/hadoop/libexec/etc/hadoop/yarn-env.sh
$JAVA_HOME has been set to be the output of:
  /usr/libexec/java_home
==> Summary
🍺  /usr/local/Cellar/hadoop/2.8.1: 25,233 files, 2.1GB, built in 1 minute 1 second

Hadoop的安装位置：/usr/local/Cellar/hadoop/2.8.1

显示目录归属的用户，确保为当前用户

ls -lad /usr/local /usr/local/Cellar

为了使用方便，设置Hadoop的环境变量HADOOP_HOME，我使用的Shell是oh-my-zsh，默认的配置是.zshrc，在末尾添加环境变量即可，参考

#Hadoop Settings
export HADOOP_HOME=/usr/local/Cellar/hadoop/2.8.1/libexec

测试

➜  ~ echo $HADOOP_HOME
/usr/local/Cellar/hadoop/2.8.1/libexec
➜  ~

vi /etc/hosts，添加本机的地址映射

127.0.0.1 bd01.wangchenlong.org

HDFS配置

重启电脑后，Hadoop服务全部关闭，需要重新启动。

进入Hadoop的配置目录

➜  ~ cd ${HADOOP_HOME}/etc/hadoop
➜  hadoop git:(master) ✗ pwd
/usr/local/Cellar/hadoop/2.8.1/libexec/etc/hadoop
➜  hadoop git:(master) ✗ ls
capacity-scheduler.xml     kms-env.sh
configuration.xsl          kms-log4j.properties
container-executor.cfg     kms-site.xml
core-site.xml              log4j.properties
hadoop-env.sh              mapred-env.sh
hadoop-metrics.properties  mapred-queues.xml.template
hadoop-metrics2.properties mapred-site.xml
hadoop-policy.xml          mapred-site.xml.template
hdfs-site.xml              slaves
httpfs-env.sh              ssl-client.xml.example
httpfs-log4j.properties    ssl-server.xml.example
httpfs-signature.secret    yarn-env.sh
httpfs-site.xml            yarn-site.xml
kms-acls.xml
➜  hadoop git:(master) ✗

配置HDFS，编辑core-site.xml文件

➜  hadoop git:(master) ✗ vi core-site.xml
➜  hadoop git:(master) ✗ open core-site.xml

添加两个属性，hadoop.tmp.dir用于存储HDFS的临时文件，fs.default.name用于指定HDFS的访问端口。临时文件，如果使用/tmp文件夹，每次重启电脑都会清空，因此，我们创建永久性质的临时文件夹./Hadoop/opt/data/tmp，提前使用mkdir创建。

<configuration>
    <property>
        <name>hadoop.tmp.dir</name>
        <!-- 永久存储的tmp文件. -->
        <value>/Users/wangchenlong/Hadoop/opt/data/tmp/hadoop-${user.name}</value>
        <description>A base for other temporary directories.</description>
    </property> 
    <property>
        <name>fs.default.name</name>
        <value>hdfs://bd01.wangchenlong.org:8020</value>
    </property>
</configuration>

编辑hdfs-site.xml文件，HDFS在存储时的复制数量dfs.replication，我们是单机，只需要存储1份。

<configuration>
    <property>
        <name>dfs.replication</name>
        <value>1</value>
    </property>
</configuration>

格式化HDFS

hdfs namenode -format

如果在临时文件夹./Hadoop/opt/data/tmp/hadoop-wangchenlong/dfs/中，创建了dfs文件夹，则表示格式化成功。检查dfs/name/current文件夹

-rw-r--r--  1 wangchenlong  staff  218 Sep  3 14:36 VERSION
-rw-r--r--  1 wangchenlong  staff  329 Sep  3 14:36 fsimage_0000000000000000000
-rw-r--r--  1 wangchenlong  staff   62 Sep  3 14:36 fsimage_0000000000000000000.md5
-rw-r--r--  1 wangchenlong  staff    2 Sep  3 14:36 seen_txid

fsimage是持久化存储的文件，fsimage.md5是文件的校验，seen_txid是HDFS的版本

VERSION的信息，namespaceID是NameNode的唯一ID；clusterID是集群ID，NameNode与DataNode的集群ID一致，表示同一个集群

namespaceID=742879587
clusterID=CID-679809f6-9f1f-4d6c-a35f-ac519ad543c2
cTime=1504420584572
storageType=NAME_NODE
blockpoolID=BP-1330604992-169.254.59.178-1504420584572
layoutVersion=-63

JPS(Java Virtual Machine Process Status Tool)是JDK提供的一个显示当前所有Java进程pid的命令，也可以用于显示Hadoop的进程信息。

➜  ~ jps
767 Jps
➜  ~

Hadoop的命令，都位于sbin中，使用hadoop-daemon.sh启动NameNode

➜  ~ cd ${HADOOP_HOME}/sbin
➜  sbin git:(master) ✗ ls
distribute-exclude.sh   refresh-namenodes.sh    stop-all.sh
hadoop-daemon.sh        slaves.sh               stop-balancer.sh
hadoop-daemons.sh       start-all.sh            stop-dfs.sh
hdfs-config.sh          start-balancer.sh       stop-secure-dns.sh
httpfs.sh               start-dfs.sh            stop-yarn.sh
kms.sh                  start-secure-dns.sh     yarn-daemon.sh
mr-jobhistory-daemon.sh start-yarn.sh           yarn-daemons.sh
➜  sbin git:(master) ✗ ./hadoop-daemon.sh start namenode
starting namenode, logging to /usr/local/Cellar/hadoop/2.8.1/libexec/logs/hadoop-wangchenlong-namenode-wangchenlong.local.out
➜  sbin git:(master) ✗ jps
840 NameNode
907 Jps
➜  sbin git:(master) ✗

启动DataNode。如果无法启动DataNode，因为多次格式化NameNode，可能会导致VERSION中的namespaceID不一致，删除hadoop-${user.name}文件夹，重新格式化即可。

➜  sbin git:(master) ✗ ./hadoop-daemon.sh start datanode
starting datanode, logging to /usr/local/Cellar/hadoop/2.8.1/libexec/logs/hadoop-wangchenlong-datanode-wangchenlong.local.out
➜  sbin git:(master) ✗ jps
840 NameNode
984 Jps

启动SecondaryNameNode

➜  sbin git:(master) ✗ ./hadoop-daemon.sh start secondarynamenode
starting secondarynamenode, logging to /usr/local/Cellar/hadoop/2.8.1/libexec/logs/hadoop-wangchenlong-secondarynamenode-wangchenlong.local.out
➜  sbin git:(master) ✗ jps
1105 SecondaryNameNode
1139 Jps
840 NameNode

最终情况的Java进程，需要NameNode、DataNode、SecondaryNameNode都存在。

➜  sbin git:(master) ✗ jps
1105 SecondaryNameNode
2484 DataNode
840 NameNode
2522 Jps

在HDFS上，创建文件夹demo-test，上传一个任意文件wcl.txt。

➜  sbin git:(master) ✗ hdfs dfs -mkdir /demo-test
➜  sbin git:(master) ✗ hdfs dfs -ls /
Found 1 items
drwxr-xr-x   - wangchenlong supergroup          0 2017-09-03 15:19 /demo-test
➜  ~ hdfs dfs -put wcl.txt /demo-test
➜  ~ hdfs dfs -ls /demo-test
Found 1 items
-rw-r--r--   1 wangchenlong supergroup         23 2017-09-03 15:21 /demo-test/wcl.txt
➜  ~

关闭HDFS的命令汇总

./hadoop-daemon.sh stop namenode
./hadoop-daemon.sh stop secondarynamenode
./hadoop-daemon.sh stop datanode

启动HDFS的命令汇总

cd $HADOOP_HOME/sbin
./hadoop-daemon.sh start namenode
./hadoop-daemon.sh start secondarynamenode
./hadoop-daemon.sh start datanode

配置Yarn

进入配置目录，复制MapReduce的模板

cd ${HADOOP_HOME}/etc/hadoop
cp mapred-site.xml.template mapred-site.xml

编辑mapred-site.xml，MapReduce的框架设置为Yarn。

<configuration>
    <property>
        <name>mapreduce.framework.name</name>
        <value>yarn</value>
    </property>
</configuration>

配置Yarn的属性，yarn-site.xml。yarn.nodemanager.aux-services是Yarn的NodeManager，用于管理node，使用MR的随机洗牌模式；yarn.resourcemanager.hostname是Yarn的资源管理器（ResourceManager），使用本机地址。

<configuration>
<!-- Site specific YARN configuration properties -->
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname</name>
        <value>bd01.wangchenlong.org</value>
    </property>
</configuration>

启动NodeManager和ResourceManager，先进入命令文件夹，再启动yarn-daemon脚本，最后使用jps检查是否成功。

➜  sbin git:(master) ✗ ./yarn-daemon.sh start nodemanager
starting nodemanager, logging to /usr/local/Cellar/hadoop/2.8.1/libexec/logs/yarn-wangchenlong-nodemanager-wangchenlong.local.out
➜  sbin git:(master) ✗ ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /usr/local/Cellar/hadoop/2.8.1/libexec/logs/yarn-wangchenlong-resourcemanager-wangchenlong.local.out
➜  sbin git:(master) ✗ jps
3028 SecondaryNameNode
2966 DataNode
3736 ResourceManager
2905 NameNode
3769 Jps
3661 NodeManager
➜  sbin git:(master) ✗

Yarn的端口是8088，访问 http://bd01.wangchenlong.org:8088/

Yarn

MR作业

MR的数据存储使用HDFS，MR的框架使用Yarn，执行MR作业 - 单词统计。

在./Hadoop/opt/data中，创建文本wc.input

hadoop mapreduce hive
hbase spark storm
sqoop hadoop hive
spark hadoop

将文件上传至HDFS的demo-test文件夹

➜  data hdfs dfs -put wc.input /demo-test
➜  data hdfs dfs -ls /demo-test
Found 2 items
-rw-r--r--   1 wangchenlong supergroup         71 2017-09-03 16:05 /demo-test/wc.input
-rw-r--r--   1 wangchenlong supergroup         23 2017-09-03 15:21 /demo-test/wcl.txt
➜  data

进入Hadoop的示例文件夹share

➜  data cd ${HADOOP_HOME}/share/hadoop/mapreduce
➜  mapreduce git:(master) ✗ ls
hadoop-mapreduce-client-app-2.8.1.jar             hadoop-mapreduce-client-jobclient-2.8.1-tests.jar lib
hadoop-mapreduce-client-common-2.8.1.jar          hadoop-mapreduce-client-jobclient-2.8.1.jar       lib-examples
hadoop-mapreduce-client-core-2.8.1.jar            hadoop-mapreduce-client-shuffle-2.8.1.jar         sources
hadoop-mapreduce-client-hs-2.8.1.jar              hadoop-mapreduce-examples-2.8.1.jar
hadoop-mapreduce-client-hs-plugins-2.8.1.jar      jdiff
➜  mapreduce git:(master) ✗

执行MR作业

yarn jar hadoop-mapreduce-examples-2.8.1.jar wordcount /demo-test/wc.input /demo-output/

查看输出结果

➜  mapreduce git:(master) ✗ hdfs dfs -cat /demo-output/part-r-00000
17/09/03 16:12:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
hadoop  3
hbase   1
hive    2
mapreduce   1
spark   2
sqoop   1
storm   1
➜  mapreduce git:(master) ✗

开启Yarn的历史服务

cd ${HADOOP_HOME}/sbin
./mr-jobhistory-daemon.sh start historyserver

停止命令汇总

./mr-jobhistory-daemon.sh stop historyserver
./yarn-daemon.sh stop nodemanager
./yarn-daemon.sh stop resourcemanager

启动命令汇总

./yarn-daemon.sh start nodemanager
./yarn-daemon.sh start resourcemanager
./mr-jobhistory-daemon.sh start historyserver

MR作业的执行日志，The url to track the job可能无法访问，替换为相应的host即可。

INFO client.RMProxy: Connecting to ResourceManager at bd01.wangchenlong.org/127.0.0.1:8032
INFO input.FileInputFormat: Total input files to process : 1
INFO mapreduce.JobSubmitter: number of splits:1
INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504434568655_0001
INFO impl.YarnClientImpl: Submitted application application_1504434568655_0001
INFO mapreduce.Job: The url to track the job: http://localhost:8088/proxy/application_1504434568655_0001/
INFO mapreduce.Job: Running job: job_1504434568655_0001
17/09/03 18:30:13 INFO mapreduce.Job: Job job_1504434568655_0001 running in uber mode : false
17/09/03 18:30:13 INFO mapreduce.Job:  map 0% reduce 0%
17/09/03 18:30:32 INFO mapreduce.Job:  map 100% reduce 0%
17/09/03 18:30:52 INFO mapreduce.Job:  map 100% reduce 100%
17/09/03 18:30:53 INFO mapreduce.Job: Job job_1504434568655_0001 completed successfully

OK, that's all! Enjoy it!