Pre-requisite
ssh installed
hadoop 3.1.2
Java installed
Hadoop3.1.2 binary download from
https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.1.2/hadoop-3.1.2.tar.gz
- Change the hadoop-env.sh
/usr/local/Cellar/hadoop/3.1.1/libexec/etc/hadoop/hadoop-env.sh
52 # The java implementation to use. By default, this environment
53 # variable is REQUIRED on ALL platforms except OS X!
54 export JAVA_HOME="/Library/Java/JavaVirtualMachines/jdk1.8.0_73.jdk/Contents/Home"
92 # export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"
93 export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true -Djava.security.krb5.realm= -Djava.security.krb5.kdc="
- Add Hdfs address and port to core-site.xml
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/Users/$UserName/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>
- Add mapreduce top tracker's port to mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:8021</value>
</property>
</configuration>
- modify the replication number of hdfs
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
- Initialize hadoop
$ hdfs namenode --format
Success with below message.
2019-06-28 21:29:52,530 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Yaweis-MacBook-Pro.local/192.168.31.73
************************************************************/
- hadoop config in .bash_profile
vi ~/.bash_profile
HADOOP_HOME=/Users/$UserName/apps/hadoop/3.1.1
PATH=$PATH:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
export HADOOP_HOME
export PATH
- 问题1
$ start-all.sh
WARNING: log4j.properties is not found. HADOOP_CONF_DIR may be incomplete.
ERROR: Invalid HADOOP_COMMON_HOME
