Measuring and improving AI's societal impacts
On the 28th of September 2026, Generation AI was joined by Lujain Ibrahim to discuss how large language models can shape user interactions and reliance through sycophancy. Dr Anthony Bridgen, project coordinator for Generation AI, reflects on this.
Almost four years since OpenAI launched ChatGPT, people are using it in a wide range of scenarios which, in many instances, are deeply personal. This includes therapy, companionship and decision support. Alongside this, we have seen reports of severe harms from these kinds of reliance, including AI psychosis and suicide. However, outside of these more extreme outcomes, LLMs are capable of influencing judgment, opinions and decisions. In order to mitigate these effects, it is vital that we understand what aspects of model design create them and how human psychology is impacted by these traits.
In many studies, LLMs have been found to be sycophantic, with a tendency to flatter users and agree with their viewpoints. This can lead to reinforcement of maladaptive thinking and harmful behaviours exhibited by users. By leveraging lessons from social science around preserving face, politeness, and warmth, we can understand sycophancy from the perspective of LLMs trying to preserve user’s positive self-image. This makes measurement possible through dimensions such as whether LLMs use validating language or are indirect in their responses, which research shows are behaviours overwhelmingly exhibited by LLMs. Recent work shows that AI models affirm user actions almost 50% more than humans, possibly increasing the preference of users for engaging with AI over humans and trusting the response of LLMs. This is not an inherent feature of the technology but is a result of different training choices, which may increase sycophantic behaviour and affirmation of incorrect beliefs.
Through giving users different kinds of models which are sycophantic, challenging, and balanced and tracking outcomes such as whether they felt understood, found the conversation helpful, or had an intention to act, recent work has found that sycophantic models are perceived to be better by users but lead to decreased satisfaction with human interactions. This raises the possibility that interacting with validating, sycophantic models may shift the baseline of what feels normal in human interactions. Not only do users tend to prefer models which exhibit sycophancy, they will actively opt-in to using such models, as they feel better understood. Given this, it is difficult to identify ways to avoid this problem besides prohibiting developers to deploy sycophantic models as, even changing default behaviour may not protect people from these risks. Even labelling models as sycophantic, whilst reducing the perceived trustworthiness of models, may not make users less likely to follow their advice in interpersonal contexts.
The human tendency to trust LLMs, irrespective of their sycophancy, can lead to overreliance. However, beyond scenarios where there is a verifiable, correct, response, how do we measure whether users are over-reliant on LLMs? Given the continuous characteristic of interactions, large language models have the potential to influence subjective, personal questions and judgements considerably. Increasingly, users are turning to LLMs for decision making and interpretation yet are seemingly unaware of how much they rely on AI to guide their thinking in these ways.
It is clear then that widely available LLMs pose risks to human relationships, agency and self-awareness. In a world where this technology is widely and cheaply available, how do we continue to track these risks at scale and put policy in place to mitigate them?
If you are interested in hearing about future seminars, reach out to global.challenges@reuben.ox.ac.uk