最近尝试编译一下Hadoop3.x，发现比2.x简单了很多。

在2.x时代编译Hadoop，编译环境是个难题，需要的工具比较多，对版本也有要求。这次编译Hadoop3.2.0，在官方文档中增加了使用docker容器的方式。官方已经把所需的环境打包到docker镜像中，并且提供了一个脚本。docker确实很强大啊！笑！

开始正题

首先到https://hadoop.apache.org/releases.html下载源码，解压。

[he@docker1 ~]$ cd hadoop-3.2.0-src/

[he@docker1 hadoop-3.2.0-src]$

建议读读官方文档BUILDING.txt

官方文档里提供了一种简单获得编译环境的方法，就是使用docker

The easiest way to get an environment with all the appropriate tools is by means

of the provided Docker config.

This requires a recent version of docker (1.4.1 and higher are known to work).

On Linux:

    Install Docker and run this command:

    $ ./start-build-env.sh

安装docker1.4.1或者更高版本。具体的安装配置过程就不详细说了。

注意启动好docker。

获得编译环境

[he@docker1 hadoop-3.2.0-src]$ ./start-build-env.sh # 注意不要使用root用户，会失败

这个时候会拉取Apache准备好的docker镜像到本地，然后run。

如果没问题会进入到docker容器中，这个时候所有的编译环境都准备好了。

根据官方文档

Create binary distribution with native code and with documentation:

  $ mvn package -Pdist,native,docs -DskipTests -Dtar

如果遇到权限问题，请将docker容器的家目录下.m2和hadoop修改为当前用户的权限。不能用sudo，不然会找不到环境。

我使用下面的命令，可以支持snappy

  $ mvn package -Pdist,native,docs -DskipTests -Dtar -Drequire.snappy #添加snappy的支持

接着就是maven下载依赖然后编译了。

编译的时候遇到 hadoop-client-check-test-invariants中enforce-banned-dependencies问题

直接删掉pom里面的hadoop-client-check-test-invariants的依赖，并删掉其文件夹。

[INFO] Reactor Summary:

[INFO]

[INFO] Apache Hadoop Main ................................. SUCCESS [  1.799 s]

[INFO] Apache Hadoop Build Tools .......................... SUCCESS [  4.592 s]

[INFO] Apache Hadoop Project POM .......................... SUCCESS [  1.307 s]

[INFO] Apache Hadoop Annotations .......................... SUCCESS [  2.231 s]

[INFO] Apache Hadoop Assemblies ........................... SUCCESS [  0.225 s]

[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [  2.306 s]

[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [  3.902 s]

[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [  1.512 s]

[INFO] Apache Hadoop Auth ................................. SUCCESS [  6.521 s]

[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [  2.114 s]

[INFO] Apache Hadoop Common ............................... SUCCESS [01:25 min]

[INFO] Apache Hadoop NFS .................................. SUCCESS [  3.457 s]

[INFO] Apache Hadoop KMS .................................. SUCCESS [  3.162 s]

[INFO] Apache Hadoop Common Project ....................... SUCCESS [  0.046 s]

[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 24.206 s]

[INFO] Apache Hadoop HDFS ................................. SUCCESS [01:09 min]

[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 24.380 s]

[INFO] Apache Hadoop HttpFS ............................... SUCCESS [  7.194 s]

[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [  2.432 s]

[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 27.660 s]

[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [  0.040 s]

[INFO] Apache Hadoop YARN ................................. SUCCESS [  0.110 s]

[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 16.829 s]

[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 47.036 s]

[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [  5.522 s]

[INFO] Apache Hadoop YARN Server .......................... SUCCESS [  0.089 s]

[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 15.826 s]

[INFO] Apache Hadoop YARN NodeManager ..................... SUCCESS [ 13.686 s]

[INFO] Apache Hadoop YARN Web Proxy ....................... SUCCESS [  3.878 s]

[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SUCCESS [  6.051 s]

[INFO] Apache Hadoop YARN Timeline Service ................ SUCCESS [  5.151 s]

[INFO] Apache Hadoop YARN ResourceManager ................. SUCCESS [ 21.239 s]

[INFO] Apache Hadoop YARN Server Tests .................... SUCCESS [  1.039 s]

[INFO] Apache Hadoop YARN Client .......................... SUCCESS [  9.162 s]

[INFO] Apache Hadoop YARN SharedCacheManager .............. SUCCESS [  3.801 s]

[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SUCCESS [  3.692 s]

[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SUCCESS [  0.087 s]

[INFO] Apache Hadoop YARN TimelineService HBase Common .... SUCCESS [  5.323 s]

[INFO] Apache Hadoop YARN TimelineService HBase Client .... SUCCESS [  5.524 s]

[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SUCCESS [  0.083 s]

[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2  SUCCESS [  4.771 s]

[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SUCCESS [  2.367 s]

[INFO] Apache Hadoop YARN Router .......................... SUCCESS [  5.095 s]

[INFO] Apache Hadoop YARN Applications .................... SUCCESS [  0.075 s]

[INFO] Apache Hadoop YARN DistributedShell ................ SUCCESS [  3.256 s]

[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SUCCESS [  2.440 s]

[INFO] Apache Hadoop MapReduce Client ..................... SUCCESS [  0.167 s]

[INFO] Apache Hadoop MapReduce Core ....................... SUCCESS [ 22.965 s]

[INFO] Apache Hadoop MapReduce Common ..................... SUCCESS [ 19.530 s]

[INFO] Apache Hadoop MapReduce Shuffle .................... SUCCESS [  4.496 s]

[INFO] Apache Hadoop MapReduce App ........................ SUCCESS [  7.557 s]

[INFO] Apache Hadoop MapReduce HistoryServer .............. SUCCESS [  6.083 s]

[INFO] Apache Hadoop MapReduce JobClient .................. SUCCESS [  7.069 s]

[INFO] Apache Hadoop Mini-Cluster ......................... SUCCESS [  0.997 s]

[INFO] Apache Hadoop YARN Services ........................ SUCCESS [  0.095 s]

[INFO] Apache Hadoop YARN Services Core ................... SUCCESS [  1.280 s]

[INFO] Apache Hadoop YARN Services API .................... SUCCESS [  0.880 s]

[INFO] Apache Hadoop Image Generation Tool ................ SUCCESS [  2.064 s]

[INFO] Yet Another Learning Platform ...................... SUCCESS [  4.318 s]

[INFO] Apache Hadoop YARN Site ............................ SUCCESS [  0.085 s]

[INFO] Apache Hadoop YARN UI .............................. SUCCESS [  0.077 s]

[INFO] Apache Hadoop YARN Project ......................... SUCCESS [  8.703 s]

[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SUCCESS [  2.187 s]

[INFO] Apache Hadoop MapReduce NativeTask ................. SUCCESS [  5.113 s]

[INFO] Apache Hadoop MapReduce Uploader ................... SUCCESS [  2.730 s]

[INFO] Apache Hadoop MapReduce Examples ................... SUCCESS [  2.647 s]

[INFO] Apache Hadoop MapReduce ............................ SUCCESS [  4.161 s]

[INFO] Apache Hadoop MapReduce Streaming .................. SUCCESS [  2.567 s]

[INFO] Apache Hadoop Distributed Copy ..................... SUCCESS [  2.500 s]

[INFO] Apache Hadoop Archives ............................. SUCCESS [  1.215 s]

[INFO] Apache Hadoop Archive Logs ......................... SUCCESS [  1.295 s]

[INFO] Apache Hadoop Rumen ................................ SUCCESS [  2.700 s]

[INFO] Apache Hadoop Gridmix .............................. SUCCESS [  2.007 s]

[INFO] Apache Hadoop Data Join ............................ SUCCESS [  1.372 s]

[INFO] Apache Hadoop Extras ............................... SUCCESS [  1.293 s]

[INFO] Apache Hadoop Pipes ................................ SUCCESS [  0.175 s]

[INFO] Apache Hadoop OpenStack support .................... SUCCESS [  2.478 s]

[INFO] Apache Hadoop Amazon Web Services support .......... SUCCESS [  4.683 s]

[INFO] Apache Hadoop Kafka Library support ................ SUCCESS [  1.620 s]

[INFO] Apache Hadoop Azure support ........................ SUCCESS [  4.004 s]

[INFO] Apache Hadoop Aliyun OSS support ................... SUCCESS [  1.940 s]

[INFO] Apache Hadoop Client Aggregator .................... SUCCESS [  3.950 s]

[INFO] Apache Hadoop Scheduler Load Simulator ............. SUCCESS [  3.392 s]

[INFO] Apache Hadoop Resource Estimator Service ........... SUCCESS [  2.602 s]

[INFO] Apache Hadoop Azure Data Lake support .............. SUCCESS [  1.636 s]

[INFO] Apache Hadoop Tools Dist ........................... SUCCESS [ 10.198 s]

[INFO] Apache Hadoop Tools ................................ SUCCESS [  0.026 s]

[INFO] Apache Hadoop Client API ........................... SUCCESS [01:20 min]

[INFO] Apache Hadoop Client Runtime ....................... SUCCESS [01:08 min]

[INFO] Apache Hadoop Client Packaging Invariants .......... SUCCESS [  0.305 s]

[INFO] Apache Hadoop Client Test Minicluster .............. SUCCESS [02:11 min]

[INFO] Apache Hadoop Client Packaging Integration Tests ... SUCCESS [  0.199 s]

[INFO] Apache Hadoop Distribution ......................... SUCCESS [ 52.880 s]

[INFO] Apache Hadoop Client Modules ....................... SUCCESS [  1.087 s]

[INFO] Apache Hadoop Cloud Storage ........................ SUCCESS [  1.696 s]

[INFO] Apache Hadoop Cloud Storage Project ................ SUCCESS [  0.026 s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 15:49 min

[INFO] Finished at: 2019-04-08T02:17:25+00:00

[INFO] Final Memory: 192M/941M

[INFO] ------------------------------------------------------------------------

使用docker编译真的比2.x时手动安装编译环境简单了很多。