一篇漫画让你了解Hadoop HDFS！
默认的3副本存放规则：
- 如果写入Client是HDFS集群中的DN，则1st副本存放在本Client所在主机；
- 如果写入Client不是HDFS集群中DN，则1st副本随机存放在集群中某个DN;
- 2nd副本存放于和1st副本不同机架的某DN节点上；
- 3rd副本存放于和2nd副本相同机架的另外一个DN上。
/**
* The class is responsible for choosing the desired number of targets
* for placing block replicas.
* The replica placement strategy is that if the writer is on a datanode,
* the 1st replica is placed on the local machine,
* otherwise a random datanode. The 2nd replica is placed on a datanode
* that is on a different rack. The 3rd replica is placed on a datanode
* which is on a different node of the rack as the second replica.
*/
@InterfaceAudience.Private
public class BlockPlacementPolicyDefault extends BlockPlacementPolicy {
private static final String enableDebugLogging =
"For more information, please enable DEBUG log level on "
+ BlockPlacementPolicy.class.getName();
private static final ThreadLocal<StringBuilder> debugLoggingBuilder
= new ThreadLocal<StringBuilder>() {
@Override
protected StringBuilder initialValue() {
return new StringBuilder();
}
};
关于block副本数需要注意的地方：
- 设置的副本数不能超过集群中DataNode的数量
- 每个DataNode只能存放某block的一个副本
- 每个机架最多只能存放某个block的2个副本
