Large Language Models - A Curated Reading List

While I am working on the blog series on the LLaMA family of models, I have also put together a curated reading list of papers that chart the evolution of large language models. These papers provide crucial context for understanding the foundations of large language models and landscape of LLMs that are the backbone of systems like meta.ai, ChatGPT, and Claude, among others.

The LLaMA Family of Models, Model Architecture, Size, and Scaling Laws

In this blog, my attempt is to create a comprehensive reading list, including some that are featured in my blog above.

Key research papers on deep learning architectures

1. Sequence to Sequence Learning with Neural Networks. Sutskever et al., 2014, Google

2. Attention is All You Need. Vaswani et al., 2017, Google

3. Improving Language Understanding by Generative Pre-Training. Radford et al., 2018. OpenAI - GPT-1

4. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al., 2018. Google

5. BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. Lewis et al., 2019. Facebook

6. Language Models are Unsupervised Multitask Learners. Radford et al., 2019. OpenAI - GPT-2

7. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Raffel et al., 2019. Google - T5

8. Language Models are Few-Shot Learners. Brown et al., 2020. Open AI - GPT-3

9. On Layer Normalization in the Transformer Architecture. Xiong et al., 2020.

Survey Papers

1. A Survey of Large Language Models. Zhao et al., 2023. [Github]

2. Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontiers. Qin et al., 2024.

3. Large Language Models: A Survey. Minaee et al., 2024

Efficient pre-training and scaling laws

1. Scaling Laws for Neural Language Models. Kaplan et al., 2020. Open AI

2. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. Rae et al., 2021. DeepMind

3. Training Compute-Optimal Large Language Models. Hoffmann et al., 2022 DeepMind - Chinchilla

4. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. Dao et al., 2022. Stanford University

5. PaLM: Scaling Language Modeling with Pathways. Chowdhery et al., 2022. Google

6. Cramming: Training a Language Model on a Single GPU in One Day. Geiping et al., 2022. University of Maryland, College Park

Survey Papers

1. Efficient Transformers: A Survey. Tay et al., 2020 (Revised 2022). Google

2. A Survey on Efficient Training of Transformers. Zhuang et al., 2023

Fine-tuning and parameter-efficient transfer learning

1. Parameter-Efficient Transfer Learning for NLP. Houlsby et al., 2019. Google [Github]

2. Finetuned Language Models are Zero-Shot Learners. Wei et al., 2021. Google

3. LoRA: Low-Rank Adaptation of Large Language Models. Hu et al., 2021. Microsoft [Github] [Video]

4. QLoRA: Efficient Finetuning of Quantized LLMs. Dettmers et al., 2023. University of Washington [Github]

Survey Papers

1. A Survey of Quantization Methods for Efficient Neural Network Inference. Gholami et al., 2021. UC Berkeley

2. Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning. Lialin et al., 2022. UMass Lowell

3. Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond. Yang et al., 2023 [ Github]

4. The Efficiency Spectrum of Large Language Models: An Algorithmic Survey. Ding et al., 2024, Microsoft

Aligning LLMs

1. Deep Reinforcement Learning from Human Preferences. Christiano et al., 2017 (Revised 2023). Google

2. Fine-Tuning Language Models from Human Preferences. Ziegler et al., 2019 (Revised 2020)

3. Training Language Models to Follow Instructions with Human Feedback. Ouyang et al., 2022. OpenAI - InstructGPT

4. Training a helpful and harmless assistant with reinforcement learning from human feedback. Bia et al., 2022. Anthropic

5. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. Ganguli et al., 2022. Anthropic

6. Constitutional AI: Harmlessness from AI Feedback. Bia et al., 2022. Anthropic

Survey Papers

1. Aligning Large Language Models with Human: A Survey. Wang et al., 2023. Huawei Noah’s Ark Lab

2. Large Language Model Alignment: A Survey. Shen et al., 2023. Tianjin University

Note, this is not an exhaustive reading list, and I will be updating it as I come across any new paper while I work on #icodeformyभाषा! There will be more of such reading lists in the future for different areas within natural language processing.