小结一下Hadoop/Hive的文件格式和压缩算法，目录如下，
0. Overview
1. 文件格式
2. 压缩算法
3. Others
4. Reference
Overview
文件格式和压缩算法在大数据系统里面是一个高关注的优化点，双方常常是配合着一起调优使用。
1. 文件格式
A file format is the way in which information is stored or encoded in a computer file. In Hive it refers to how records are stored inside the file. As we are dealing with structured data, each record has to be its own structure. How records are encoded in a file defines a file format.
| file format | characteristics | hive storage option |
|---|---|---|
| TextFile | plain text, default format | STORED AS TEXTFILE |
| SequenceFile | row-based, binary key-value, splittable | STORED AS SEQUENCEFILE |
| Avro | row-based, binary or JSON, splittable | STORED AS AVRO |
| RCFile | columnar, RLE | STORED AS RCFILE |
| ORCFile | Optimized RC, Flatten | STORED AS ORC |
| Parquet | column-oriented binary file, Nested | STORED AS PARQUET |
2. 压缩算法
To balance the processing capacity required to compress and uncompress the data, the CPU required to processing compress or uncompress data, the disk IO required to read and write the data, and the network bandwidth required to send the data across the network.
Compression is not recommended if your data is already compressed (such as images in JPEG format). In fact, the resulting file can actually be larger than the original.
| compression format | characteristics | splittable |
|---|---|---|
| DEFLATE | DefaultCodec | no |
| GZip | uses more CPU resources than Snappy or LZO; provides a higher compression ratio; A good choice for cold data | no |
| BZip2 | more compression than GZip | yes |
| LZO | better choice for hot data | yes if indexed |
| LZ4 | significantly faster than LZO | no |
| Snappy | performs better than LZO, better choice for hot data | yes? |
Others
- 游程编码，Run Length Encoding，RLE，常用于列式存储，4A3B2C1D4E
- 纠删码，Erasure Coding，EC，hadoop 3.0.0的replica，但由于其带宽和cpu高消耗，常用于冷数据，ｋ块原始+ｍ块校验
- Doc Values，最大公约数压缩，偏移量进行编码，按照docid排序的，利用内存映射文件mmap，预读取机制
- skipList
- bitSet [1,3,4,7,10]->[1,0,1,1,0,0,1,0,0,1]
- Roaring Bitmap (bitset improvement)，类似RLE，4A3B
- Frame Of Reference编码
- 数值差分[73,300,302,332,343,372]->[73,227,2,30,11,29]
- term index，tire树
- term dictionary
-
finite state transducers
FST - 维度字段上移到父文档里，而不用在每个子文档里重复存储，从而减少索引的尺寸
- segment一个int就可以存储
- Hyperloglog
- 聚合之后再做聚合，Pipeline Aggregation
Reference
- Format Wars
- Data Storage and Modelling in Hadoop
- Apache Hive Different File Formats
- Hive 列存储简介
- hadoop 压缩 gzip biz2 lzo snappy
- Choosing a Data Compression Format
- Data Compression in Hadoop
- Hadoop: The Definitive Guide
- An Overview of File and Serialization Formats in Hadoop
- 深入理解 ElasticSearch Doc Values
