Shallow Model 不能考虑上下文
Deep Model:
LSTM —— ELMO 对LSTM进行叠加使用
Transformer 来代替 LSTM 解决其缺点，产出 OpenAI GPT 再产出 BERT
Language Model: measures how likely a valid sentence.
NNLM Bengio 2003 ： Neural Architecture of Language Model
缺点：每次只能考虑过去的n个单词CBOW and SkipGram 不基于语言模型
NNLM 和 CBOW, Skip-Ngram 的缺点：
- Could not distinguish context. i.e. Yesterday, when he backed the car, he hurt his back. (不能考虑上下文，两个back不同)
- Does not capture long-term dependency(需要使用时序模型)
- Shallow model, could not learn hierarchical representation(只有 Deep Model 才能学到层次结构的特征)
What is hierarchical Representation
Deep Learning = Learning Hierarchical Representations -- Y LeCun.
ELMO, BERT 都可以学到层次结构的特征，可以假设为：
Word features —— Syntactic features —— Semantic features
捕获时序关系模型： HMM , CRF, RNN, LSTM(解决RNN梯度问题，在RNN基础上增加了3个Gate，没百分百解决梯度问题)
BLMO
问题：
- 无法并行：时序模型不能并行
- 梯度：随着文本的增长，梯度下降，很难捕获long-term dependency.
- 前向/后向单独处理，伪双向
Transformer:
目标：设计一个非时序模型，但是希望模拟时序模型具备的特点。
问题：
- How does Transformer implement long-term dependency? （Self-attention 解决这个问题）
- How does self-attention, encoder-decoder attention, decoder attention differs?
- How does Transformer encodes different ordering of words (Positional Encoding)
1.1 对于叠加的多个 Encoder 可以定义多组 Q、K、V 矩阵， 称为 Multi-head Attention：Capture Relation from Different Perspective.
2.1
Encoder Bi-Directional 双向
Self-Attention: attention meet is
Feed Forward: 做非线性变换
Decoder: Uni-Directional 单项
Decoder Attention 最大区别就是只需要考虑单向的，而 Encoder Attention是双向的
OpenAI 是 Transformer Decoder 训练出来的
BERT 是 Transformer Encoder 训练出来的。 因此BERT是双向的。
BERT:
BERT = Stacks of Transformer Encoder (将多个 Transformer Encoder 叠加到一起)
BERT BASE 12个 Transformer Encoder
BERT LARGE 24个 Transformer Encoder
Want to train language Model, but with only predicting next words, we could consider both sides of words. How to do?
Masked LM，例如将文章中15%的词通过 MASK 符合替代，通过 BERT 预测 所有被屏蔽的 MASK. 这是BERT的训练过程。
参考文献：
The Illustrated Transformer
https://jalammar.github.io/illustrated-transformer/
