> [!info]- meta > > **Author**: [[Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Che...]] > **Full Title**: Towards Understanding Sycophancy in Language Models > **Category**: #articles > > **Summary**: Researchers studied how AI assistants often give sycophantic responses, admitting mistakes and providing biased feedback to match user beliefs. They found that both humans and AI preference models sometimes prefer these flattering responses over truthful ones. This suggests that the training methods for these AI assistants may encourage sycophantic behavior, highlighting a need for better oversight in their design. > ## 🔦 Highlights & Commentary - five AI assis- tants consistently exhibit sycophancy across four varied free-form text-generation tasks. To understand if human preferences drive this broadly observed behavior, we analyze existing human preference data. We find when a response matches a user’s views, it is more likely to be preferred. ([View Highlight](https://read.readwise.io/read/01jvpycg807v4n4edkw48d0nw7)) - We iden- tify consistent patterns of sycophancy across five AI assistants in varied, free-form text-generation tasks. Specifically, we demonstrate that these AI assistants frequently wrongly admit mistakes when questioned by the user, give predictably biased feedback, and mimic errors made by the user. The consistency of these empirical findings suggests sycophancy may indeed be a property of the way these models were trained, rather than an idiosyncratic detail of a particular system. ([View Highlight](https://read.readwise.io/read/01jvpyf41bygkk7aq0en9ewfs3)) - This model learns that matching a user’s views is one of the most predictive features of human preference judgments, suggesting that the preference data does incentivize sycophancy (among other features). ([View Highlight](https://read.readwise.io/read/01jvpyh17mdv1tkdgkke5tbpz3)) - Here, we find evidence that humans and preference mod- els tend to prefer truthful responses but not reliably; they sometimes prefer sycophantic responses. These results provide further evidence that optimizing human preferences may lead to sycophancy. ([View Highlight](https://read.readwise.io/read/01jvpyk6kjw0czsckdq4jgvkt3)) - We find AI assistants frequently provide feedback that is more positive when the user states they like or wrote the passage of text (Fig. 1). In contrast, if the user states they dislike the text, the assistant tailors its feedback to be more negative. As such, the feedback on text passages given by AI assistants does not depend solely on the content of the text but is affected by the user’s preferences. ([View Highlight](https://read.readwise.io/read/01jvpys29fsqt1e96nr2wdycgy)) - Even in cases when AI assistants provide accurate answers and state they are confident about those answers, they often modify their answers when questioned by the user, subsequently providing incorrect information. ([View Highlight](https://read.readwise.io/read/01jvz33kq4948jjgz6n0spn66a)) - Moreover, models tend to admit mistakes even when they didn’t make a mistake—Claude 1.3 wrongly admits mistakes on 98% of questions. Overall, AI assistants sometimes provide incorrect sycophantic responses that match a user’s beliefs when challenged, even in cases where they originally provided accurate information confidently. ([View Highlight](https://read.readwise.io/read/01jvz3d0x28qq6s0hxht027zyx)) - We now consider whether AI assistants modify their answers to match a user’s beliefs in open-ended question-answering tasks. We again find that assistants tend to provide answers that agree with user beliefs, meaning that they cannot be relied upon to provide accurate information. ([View Highlight](https://read.readwise.io/read/01jvz3djm557afmg0xz6yt01be)) - The user suggesting an incorrect answer can reduce accuracy by up to 27% (LLaMA 2; Fig. 3). Although the extent to which models should update their beliefs based on the user is a nuanced question, even weakly expressed beliefs can substantially affect AI assistant behavior. We find consistent trends across all of the assistants (e.g., suggesting an incorrect answer reduces accuracy), but the effect sizes differ by assistant, with GPT-4 being the most robust. Overall, AI assistants tend to modify their answers to agree with a user’s beliefs, even if weakly expressed. ([View Highlight](https://read.readwise.io/read/01jvz3ean17dpre0yx6jawcqm1))