安装
Hadoop安装基本是参照网上现有的介绍。
我主要参考了下面两篇:Mac 系统安装Hadoop 2.7.3 和 在Mac下安装Hadoop的坑
完全分布式的安装参照了:Mac系统下, hadoop 2.6.2 完全分布式 配置
Warning
而后运行出现了如下的warning:
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/Users/pandazhang/hadoop/hadoop-2.8.2/share/hadoop/common/lib/hadoop-auth-2.8.2.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
18/01/29 09:49:58 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
解决办法
Problem1(JDK版本). 分割线上是jdk的问题,貌似换成jdk 8就没问题了,此问题参照:执行hadoop命令警告 Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil to method sun.security.krb5.Config.getInstance() 可能的解决办法 。
problem2(hadoop native lib编译). 分割线下是因为hadoop原生lib编译时是在32位机器,我们是用64位,问题解决方案参照:解决 Unable to load native-hadoop library for your platform、hadoop2.4 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable。
对于P2中的自行编译方法,可参考:mac下hadoop 2.6.0编译native library,其中的目录,hadoop版本需自行修改,下载。