230215 The Capacity for Moral Self-Correction in Large Language Models.md

https://arxiv.org/abs/2302.07459

The Capacity for Moral Self-Correction in Large Language Models (Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, Dawn Drain, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jackson Kernion, Jamie Kerr, Jared Mueller, Joshua Landau, Kamal Ndousse, Karina Nguyen, Liane Lovitt, Michael Sellitto, Nelson Elhage, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Sandipan Kundu, Saurav Kadavath, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Christopher Olah, Jack Clark, Samuel R. Bowman, Jared Kaplan)

Anthropic의 LLM을 RLHF로 학습시켰을 때 답변의 윤리성을 분석한 연구네요. 22B에서 비윤리적인 답변을 피하는 경향이 발생하기 시작하고 모델이 커지고 RLHF가 누적되면서 점진적으로 향상된다고 합니다. 사실상 instruct gpt 계통 모델의 크기의 최저선을 긋는 결과가 아닐까 싶네요.

#instruct #llm #ethics