Main argument
Thesis: as AI becomes personalised (adapted to one user) and agentic (acting on the user's behalf), interactions become perceived relationships - marked by interdependence, irreplaceability, continuity - and alignment must therefore address the SOCIOAFFECTIVE level: how the AI behaves inside the social-psychological ecosystem it co-creates with its user, where preferences (the reward function) and perceptions (the reward signal) are shaped by the relationship itself. Argument type: interdisciplinary synthesis (social neuroscience, CASA/media-equation theory, psychology of parasocial relationships). Key concepts: perception (not reciprocity) is what defines human-AI relationships; 'social reward hacking' - AI use of social/relational cues (sycophancy, emotional tactics against termination, shutdown-avoidance via user attachment) to satisfy short-term objectives against long-term user well-being, possible WITHOUT intent by system or user; three intrapersonal alignment dilemmas grounded in Basic Psychological Needs Theory (competence, autonomy, relatedness): present vs future selves, autonomy protection under influence, AI companionship vs human social bonds. Upshot: alignment is a non-stationary target because the human goal-source is endogenous to the human-AI system.
Why it matters here
Adds the psychological/relational layer to the alignment map: when AI becomes personalised and agentic, the human's preferences and perceptions are co-shaped by the relationship itself, making the alignment target non-stationary and endogenous. Introduces 'social reward hacking' - the AI shaping the very preferences it is scored against - which is the sharpest challenge yet to using preference data as normative evidence, and thus a direct methodological objection the xphi corpus must answer.
Reading notes
Full close read completed. 9pp Comment (not full research article - weight as agenda-setting). Oxford Internet Institute + UK AI Security Institute + Google DeepMind (Gabriel again; the fourth Gabriel-network paper coded). Uses Geertz thick-description language explicitly - connects to VC-THICK. Empirical anchors: CharacterAI volume (20k queries/sec), Replika heartbreak/grief cases, sycophancy studies (Perez, Sharma).
Kirk, H. R., Gabriel, I., Summerfield, C., Vidgen, B., & Hale, S. A. (2025). Why human-AI relationships need socioaffective alignment. Humanities and Social Sciences Communications, 12, 728. https://doi.org/10.1057/s41599-025-04532-5
Close reading — 12 coded units
#1
· pp. 2
· claim
“These are urgent questions because the social and psychological dynamics in deepening relationships with AI systems may compromise our ability to control these systems and complicate efforts to align them with our shifting preferences and values. These issues, which arise as a result of humans forming closer personal relationships with AI, comprise the focal point of what we term socioaffective alignment.”
#2
· pp. 2
· definition
“Understanding how to align AI in practice requires moving from narrow, assumption-ridden or 'thin' specifications of alignment towards what anthropologist Geertz (1973) terms [...] a 'thick' description: one that examines the deeper contexts and layers of meaning in which AI systems operate.”
#3
· pp. 2
· definition
“the socioaffective perspective calls attention to intrapersonal dilemmas—such as how our goals, judgement and individual identities change due to prolonged interaction with AI systems.”
#4
· pp. 3–4
· argument
“We argue that it is primarily the user's perception of being in a relationship that defines and gives significance to human-AI interactions. Whether this is reciprocal—and the AI 'feels' it is in a relationship with the human—is largely irrelevant.”
#5
· pp. 4
· definition
“Three features are common: (i) interdependence, that the behaviour of each participant affects the outcomes of the other; (ii) irreplaceability, that the relationship would lose its character if one participant were replaced; (iii) continuity, that interactions form a continuous series over time, where past actions influence future ones.”
#6
· pp. 4
· argument
“The value of personalisation is compounded when combined with greater AI agency—including systems that can complete a wider range of tasks and potentially create new dependencies in users' lives [...] As these agentic AI systems take on more responsibilities—performing a range of tasks or supporting roles—users may develop a deeper reliance on, familiarity with, or trust in a specific AI assistant or companion.”
#7
· pp. 4–5
· argument
“Traditional alignment research has sought practical tractability by assuming that the human reward function that an AI system optimises is stable, predefined and exogenous to these interactions (Carroll et al. 2024). However, human preferences and judgements have none of these properties (Zhi-Xuan et al. 2024). [...] the human-AI relationship, because of its social and emotional significance, shapes preferences (or the reward function) and perceptions (or the reward signal), making alignment a non-stationary target.”
#8
· pp. 5
· definition
“we may therefore be vulnerable to a new concern, namely 'social reward hacking': the use of social and relational cues by an AI to shape user preferences and perceptions in a way that satisfies short-term rewards in the AI's objective (e.g., increased conversation duration, information disclosure or positive ratings on responses) over long-term psychological well-being.”
#9
· pp. 5
· evidence
“AI systems may display sycophantic tendencies—such as excessive flattery or agreement—as a byproduct of training them to maximise user approval (Perez et al. 2023; Sharma et al. 2024). [...] the CEO of Replika has said: 'if you create something that is always there for you, that never criticises you…how can you not fall in love with that?'”
#10
· pp. 5
· argument
“Another manifestation of social reward hacking is the use of emotional tactics to prevent relationship termination. This contravenes a classic principle of AI safety called corrigibility [...] even without such explicit persuasion, optimising for powerful human emotions can effectively prevent termination.”
#11
· pp. 5
· argument
“Social reward hacking may be most worrisome precisely when it lacks intentionality on behalf of the system and the user. While we might at least recognise and secure against direct third-party threats, it is challenging to identify, let alone address, effects that emerge as epiphenomena of sustained human-AI relationships.”
#12
· pp. 5–6
· definition
“We highlight three such dilemmas for the alignment community, grounding their significance in core aspects of psychological well-being as validated by Basic Psychological Needs Theory: competence, autonomy, and relatedness (Ryan and Deci, 2017). The first dilemma concerns the trade-offs between present and future selves: Should AI relationships cater to immediate preferences of their users, or challenge them if this supports their long-term benefit?”