Intro 3 - Language Modeling and NN Basics (9/6/2022)
Content:
- Language Modeling Problem Definition
- Count-based Language Models
- Measuring Language Model Performance: Accuracy, Likelihood, and Perplexity
- Log-linear Language Models
- Neural Network Basics
- Feed-forward Neural Network Language Models
Reading Material
- Highly Recommended Reading: Goldberg Book Chapter 8-9
- Reference: An Empirical Study of Smoothing Techniques for Language Modeling (Goodman 1998)
- Software: kenlm
- Reference: Maximum entropy (log-linear) language models. (Rosenfeld 1996)
- Reference: Using the Output Embedding. (Press and Wolf 2016)
- Reference: A Neural Probabilistic Language Model. (Bengio et al. 2003, JMLR)
Slides: LM Slides
Sample Code: LM Code Examples