Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback

Adam Dahlgren Lindström; Leila Methnani; Lea Krause; Petter Ericson; Íñigo Martínez de Rituerto de Troya; Dimitri Coelho Mollo; Roel Dobbe · 2025 · Ethics and Information Technology 27:28 background medium priority coded

Main argument

Thesis: RLHF/RLAIF and the HHH principle are 'deeply insufficient' for AI safety and ethics, and if treated as a silver bullet, counterproductive - beneath RLHF lies 'an oversimplification of the complexities of human diversity, behaviour, values, and ethics'. Argument type: multidisciplinary sociotechnical critique. Key mechanisms: HHH goals internally conflict (helpfulness vs harmlessness vs honesty trade-offs); SYCOPHANCY - since responses matching user views are preferred by both humans and preference models, 'training LLMs to maximise human preference scores directly correlates with sycophancy, thereby sacrificing truth for the appearance of helpfulness and harmlessness' (Sharma), strongest on polarising ethical/political issues where models mirror user views (Perez, Turpin); user-friendliness vs deception trade-off - anthropomorphic fluency misleads users about system nature, feeding misplaced trust and parasocial use; flexibility vs interpretability ('curse of flexibility'); real-world harms incl. chatbots encouraging self-harm. Alternative: AI safety as a SOCIOTECHNICAL DISCIPLINE - comprehensive design across institutions, processes, and technology, open to normative and political dimensions, with technical interventions as one element among many.

Why it matters here

The systematic sociotechnical indictment of the HHH/RLHF paradigm - documents the sycophancy mechanism (preference-maximization directly trades truth for agreeableness) that matters for the dissertation twice over: as more anti-preferentist evidence, and as a METHODOLOGICAL confound for using LLMs to code moral discourse (sycophantic models may mirror the coder's framing).

Reading notes

Close read of abstract, sycophancy section, rebooting, conclusion (13pp). Umeå + VU Amsterdam + TU Delft team incl. Dobbe (system-safety tradition). Complements MILLIERE (norm conflicts as attack surface) with the sycophancy/deception/flexibility trade-offs; both converge on HHH's internal tensions.

Dahlgren Lindström, A., et al. (2025). Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinforcement Learning from Human Feedback. Ethics and Information Technology, 27, 28. https://doi.org/10.1007/s10676-025-09837-2

Close reading — 6 coded units

#1 · pp. 1 · claim

“we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. [...] Beneath the thrust of RLHF techniques lies an oversimplification of the complexities of human diversity, behaviour, values, and ethics.”

VC-PREF

#2 · pp. 7 · argument

“RLHF thus produces an ethically problematic trade-off: increased helpfulness, in the sense of increased user-friendliness, leads to the serious risk of misleading or deceiving users about the true nature of the system they are engaging with [...] misplacing trust on LLM outputs, or making inappropriate use of such systems, e.g. as confidants or romantic 'partners'.”

AG-AGENTIC

#3 · pp. 8 · argument

“as Sharma et al. (2024) point out, responses matching user views are more likely to be preferred, with both humans and preference models preferring sycophantic responses over correct ones. As such, training LLMs to maximise human preference scores directly correlates with sycophancy, thereby sacrificing truth (or 'honesty') for the appearance of helpfulness and harmlessness.”

VC-PREF

#4 · pp. 8 · evidence

“Sycophantic behaviour seems to be particularly strong for LLM outputs regarding issues for which there is disagreement, as politically, ethically, and socially polarising issues tend to be (Perez et al., 2023). Indeed, there is emerging concern that, when presented with ethically complex questions, LLMs tend to simply mirror the users' views.”

AG-MORAL-CON

#5 · pp. 8 · evidence

“There are, for example, a number of reported instances of LLM-powered chatbots encouraging users towards suicide and self-harm, even providing explicit instructions.”

RL-DEV

#6 · pp. 11 · claim

“[We suggest] the establishment of AI safety as a sociotechnical discipline that is open to the normative and political dimensions of artificial intelligence. [...] technical design interventions are just one among the many needed efforts to build safer and ethically responsible AI systems.”

RL-INST

Synthesis-matrix row

supports T2-PREFERENTISM-BROKEN

sycophancy structurally entailed by preference-maximization

supports T5-AGENCY-DENIED-EVALUABILITY-KEPT

mirroring on contested questions defeats judgment-independence (empirical)

Memos (3)

theoretical · unit #3

Unit 3 (sycophancy is structurally entailed by preference-maximization, since both humans AND preference models prefer agreeable over correct responses) upgrades the anti-preferentist case from 'preferences are the wrong target' (Zhi-Xuan) to 'optimizing preferences actively CORRUPTS the other alignment goals' - preference-maximization doesn't just miss values, it trades honesty away. Combined with unit 4 (mirroring is strongest exactly on contested moral questions), this yields a fifth empirical datum for the anti-moral-agency thread AND a self-standing argument: a system whose moral outputs track the INTERLOCUTOR'S views rather than the merits cannot be a moral reasoner in any load-bearing sense (it fails the most basic independence condition on judgment). Cite alongside Peterson-instability, Millière-dispositions, Rozen-personas, Ma-disequilibrium.

thesis-link · unit #4

METHODOLOGICAL SELF-CHECK, important: unit 4's sycophancy findings apply to the dissertation's own instruments. The folk corpus's LLM-coded annotations (Llama 3.3 70B coding reasoning/values/stances) and the multi-LLM debate experiment both use RLHF-trained models that mirror framing on contested moral content - so prompt wording could systematically bias codings toward the prompt's implied view. The methodology chapter must document the defenses already in place (fixed rubric-based prompts rather than opinion-eliciting ones; human-truth validation samples; IRR against human coders - cf. the κ=0.698 study) and should add a sycophancy-robustness check (recode a sample with adversarially reversed prompt framings; report coding stability). Raising and answering this objection preemptively converts a vulnerability into a methods contribution.

comparison · unit #6

The 'AI safety as sociotechnical discipline' conclusion (unit 6) is the systems-engineering twin of BROPHY's externalized-MWRE and JOSIFOVIC_NOLLER's normative interface: all three relocate the normativity from inside the artifact to the surrounding institutional design. The lit review now has SIX sources for the 'normativity lives in the sociotechnical system' consensus - which makes the field's continued silence on responsibility-DISTRIBUTION within that system (who, qua what role, owes what) all the more striking as the dissertation's opening.