LLM (Large Language Model)

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.04	OpenELM: An Efficient Language Model Family with Open-source Training and Inference Framework	layer-wise scaling	Apple	Makrdown
2024.04	Rho-1: Not All Tokens Are What You Need	Roh-1, Selective Language Modeling	Tsinghua University, Microsoft	Makrdown
2024.03	Jamba: A Hybrid Transformer-Mamba Language Model	Jamaba, Transformer, Mamba, MoE	AI21labs	Makrdown
2024.02	Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models	Griffin, Hawk	Google DeepMind	Blog
2024.02	OLMo: Accelerating the Science of Language Models	Open Source, OLMo	AI2	Makrdown
2024.01	Orion-14B: Open-source Multilingual Large Language Models	Open Source, MLLM	OrionStar Inc.	Markdown
2023.05	PaLM2 Technical Report	PaLM2	Google	Blog
2023.04	Scaling Transformer to 1M tokens and beyond with RMT	Attention, RMT	Neural Networks and Deep Learning Lab, MIPT, Dolgoprudny, Russia	Blog
2023.03	GPT-4 Technical Report	GPT-4	OpenAI	Blog
2019.05	BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding	BERT	Google AI Language	Blog	NAACL-HLT 2019

---

Reasoning

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.04	Self-Explore to Avoid the Pit: Improving the Reasoning Capabilities of Language Models with Fine-grained Rewards	Self-Explore, Granular Learning Signals	KAIST	Makrdown
2024.02	Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models	Boosting of Thoughts (BoT)	University of Toronto	Markdown	ICLR 2024
2024.02	Chain-of-Thought Reasoning Without Prompting	CoT-decoding	Google DeepMind	Blog
2024.02	Self-Discover: Large Language Models Self-Compose Reasoning Structures	Decompose, Reasoning Structure	University of Southern California	Blog
2024.02	A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chains	Benchmark, REVEAL	Google	Blog
2024.01	LLMs cannot find reasoning errors, but can correct them!	CoT	Google Research	Markdown
2023.10	Small Language Models Fine-tuned to Coordinate Larger Language Models improve Complex Reasoning	Prompt Decomposition	IIT Delhi, India	Markdown	EMNLP 2023
2023.05	Learning to Reason and Memorize with Self-Notes	Self-Notes	Meta AI, NVIDIA	Blog	NeurIPS 2023 Poster
2023.04	Unleashing Infinite-Length Input Capacity for Large-scale Language Models with Self-Controlled Memory System	Self-Controlled Memory (SCM)	ByteDance AI Lab	Blog
2023.04	Least-to-Most Prompting Enables Complex Reasoning in Large Language Models	least-to-most prompting	Google Research	Blog	ICLR 2023
2023.04	PINTO: Faithful Language Reasoning Using Prompt-Generated Rationales	Rationale	University of Southern California	Blog	ICLR 2023
2023.04	Answering Questions by Meta-Reasoning over Multiple Chains of Thought	CoT, Meta-Reasoning	AI2	Blog	EMNLP 2023
2023.01	Specializing Smaller Language Models towards Multi-Step Reasoning	Knowledge Distillation, Reasoning	AI2	Makrdown

---

RAG (Retrieval Augmented Generation)

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.04	How faithful are RAG models? Quantifying the tug-of-war between RAG and LLMs' internal prior	Prior Probability	Stanford University	Makrdown
2024.03	Harnessing Multi-Role Capabilities of Large Language Models for Open-Domain Question Answering	LLMQA	Renmin University	Markdown	WWW 2024
2024.01	RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval	tree, summarization	Stanford University	Markdown	ICLR 2024
2024.01	A Survey on Evaluation of Large Language Models	Evaluation, Survey	...	Markdown	ACM
2024.01	The Power of Noise: Redefining Retrieval for RAG Systems	Noise, RAG	...	Makrdown	ACM
2024.01	Corrective Retrieval Augmented Generation	RAG, CRAG	University of Science and Technology of Chian, Google Research	Blog
2023.12	RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!	Reranking	University of Waterloo	Markdown
2023.10	Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents	Re-Ranking	Baidu	Markdown
2023.10	Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!	Reranking	Nanyang Technological University	Markdown
2023.10	Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection	Self, RAG	University of Washington, AI2, IBM Research AI	Markdown
2023.02	Augmented Language Models: a Survey	RAG, Survey	Meta	Markdown
2021.10	BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models	IR Benchmark	UKP-TUDA	Markdown	NeurIPS 2021

---

Training

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.04	The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructions	Instruction Hierarchy	OpenAI	Markdown
2024.01	Beyond Sparse Rewards: Enhancing Reinforcement Learning with Language Model Critique in Text Generation	RELC, RL, PPO	McGill, Toronto, Mila, Google Research	Blog	ACL 2024 under review
2024.01	Can AI Assistants Know What They Don't Know?	Hallucination	Fudan University	Blog
2024.01	Tuning Language Models by Proxy	Proxy-tuning	Allen Institue of AI	Blog
2024.01	Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models	Self-Play	...	Markdown
2024.01	Knowledge Fusion of Large Language Models	Knowledge Fusion	Tencent AI Lab	Blog
2023.12	Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision	superhuman model	OpenAI	Blog
2023.12	Select, Prompt, Filter: Distilling Large Language Models for Summarizing Conversations	Knowledge Distillation	Zoom Video Communications	Markdown	EMNLP 2023
2023.12	Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models	Self-Training, ReST	Google DeepMind	Markdown
2023.10	SELF: Self-Evolution with Language Feedback	SELF	The University of Hong Kong	Blog
2023.03	oBERTa: Improving Sparse Transfer Learning via improved initialization, distillation, and pruning regimes	RoBERTa, Knowledge Distillation	Department of Computer Science, the University of Illinois Urbana-Champaign	Blog

---

Prompting

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.03	Dynamic Contexts for Generating Suggestion Questions in RAG Based Conversational Systems	Dynamic Contexts	University of Illinois, Chicago	Makrdown	WWW`2024 workshop PromptEng
2024.02	Hint-before-Solving Prompting: Guiding LLMs to Effectively Utilize Encoded Knowledge	HSP, Summarization (not mentioned though)	National University of Singapore	Markdown
2024.02	A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts	Gist Memory, ReadAgent	Google DeepMind	Markdown
2024.01	Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding	Meta-Prompting	Stanford University, OpenAI	Markdown

---

PEFT

Date	Title(arxiv)	Keyword	Affiliation	Note	Conference
2024.03	PERL: Parameter Efficient Reinforcement Learning from Human Feedback	PERL, LoRA, Reward Model	Google Research	Markdown
2024.02	ResLoRA: Identity Residual Mapping in Low-Rank Adaption	ResLoRA, Merge	Microsoft	Blog
2024.02	LoRA+: Efficient Low Rank Adaptation of Large Models	LoRA+	UC Berkeley	Markdown
2023.04	LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model	LLaMA, Visual Instruction	Shanghai Artificial Intelligence Laboratory	Blog

---

Compression

Date	Title(arxiv)	Keyword	Affiliation	Note	Conference
2024.01	SliceGPT: Compress Large Language Models by Deleting Rows and Columns	Model Compression	Microsoft Research	Markdown
2023.04	Learning to Compress Prompts with Gist Tokens	Gist Token, Compression	Stanford University	Blog	NeurIPS 2023
2022.05	Matryoshka Representation Learning	Matryoshka, Adaptive	Google Research	Markdown	NeurIPS 2022

---

Tool, Agent

Date	Title(arxiv)	Keyword	Affiliation	Note	Conference
2023.04	Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models	Tools, Chameleon	University of California, Los Angeles, Microsoft Research	Blog	NeurIPS 2023

---

Evaluation / Benchmark / Dataset

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.05	Observational Scaling Laws and the Predictability of Language Model Performance	Observational Scaling Law	Stanford, Toronto	Makrdown
2024.03	RewardBench: Evaluating Reward Models for Language Modeling	Reward Model, DPO	AI2	Markdown
2024.02	Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation	Bonito, Instruction Tuning Dataset	Brown University	Markdown	ACL 2024 submission
2024.02	LLM-based NLG Evaluation: Current Status and Challenges	NLG, Evaluation	Peking University	Markdown
2024.02	Can Large Language Models Understand Context?	Benchmark, In-Context Learning	Apple	Markdown	Findings of EACL 2024
2024.01	Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research	Dolma, Dataset, Pretraining	AI2	Markdown
2023.05	Do Large Language Models Know What They Don't Know?	Self-Knowledge, SelfAware (Dataset)	School of Computer Science, Fudan University	Markdown
2023.04	Are Emergent Abilities of Large Language Models a Mirage?	Emergent Ability	Stanford University	Blog	NeurIPS 2023 oral
2023.04	Evaluating Verifiability in Generative Search Engines	Generative Search Engine	Stanford University	Blog	Findings of EMNLP 2023
2016.10	SQuAD: 100,000+ Questions for Machine Comprehension of Text	SQUAD, QA, Benchmark	Stanford University	Blog

---

Others

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2024.02	The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits	1-bit, BitNet b1.58	Microsoft Research	Blog
2024.02	LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens	LongRoPE, 2048k tokens	Microsoft Research	Makrdown
2023.04	ResiDual: Transformer with Dual Residual Connections	Layer Normalization	Microsoft Research	Blog	ICLR 2024 rejected

---

Multimodal

Date	Title (arxiv)	Keyword	Affiliation	Note	Conference
2023.03	Visual Instruction Tuning	LLaVA, Visual Instruction Tuning	Microsoft Research	Blog	NeurIPS 2023 oral
2021.08	BROS: A Pre-trained Language Model Focusing on Text and Layout for Better Key Information Extraction from Documents	BROS, AMLM, TMLM	NAVER CLOVA	Markdown	AAAI 2022