Main argument
Thesis: the default assumption of value-alignment practice - that preference-aligned systems are thereby value-aligned - is philosophically naive, because preferences are descriptive of how things are while values are normative claims about how things should be; the field should move toward context-rich, collectively grounded, and relationally structured alignment targets. Argument type: literature synthesis. Assembles four critiques of preferentism: (1) individual preferences misrepresent collective/professional values (healthcare example: nursing vs medical professional norms both under bioethical principles); (2) empirically, LLMs trained on preference methods lack coherent/stable value systems (Khan 2025: measured 'cultural alignment' is often an artefact of evaluation design; Rozen 2025: values contingent on prompt persona); (3) the attitude-behavior gap makes both revealed and stated preferences unreliable evidence of values; (4) thin-value alignment provides no justification for the norms it encodes (thick/thin concepts, Williams/Väyrynen). New directions surveyed: thick values (Edelman), strong vs weak alignment (Khamassi - understanding intentions/causation vs statistical mimicry), bidirectional alignment (Shen). Ethical upshots: participatory alignment carries distributive-justice risks (well-resourced groups dominate); responsibility must be distributed across the alignment process and continuously re-evaluated as agents gain autonomy; governance requires domain-specific normative standards beyond abstract principle convergence (Jobin).
Why it matters here
Institutional research brief synthesizing the anti-preferentist turn in value alignment (the Zhi-Xuan et al. 2024 line). Valuable less for original argument than as a map of the current critique landscape and its empirical evidence base - and as institutional confirmation that 'preferences are descriptive, values are normative' is now a recognized field-level problem, not just the dissertation's private worry.
Reading notes
Full close read completed. 9pp brief, synthesis genre (research brief, not peer-reviewed article - weight accordingly). Key sources it maps, with library status: Zhi-Xuan et al. 2024 Beyond Preferences (Phil Studies, DOI s11098-024-02249-w - WAS IN THE ZIP as s11098-024-02249-w.pdf but not kept by triage; RECOVER); Khamassi et al. 2024 strong/weak alignment (Sci Reports - acquisition target); Khan/Casper/Hadfield-Menell 2025 FAccT cultural-alignment unreliability; Rozen et al. 2025 ICLR LLM value consistency; Shen et al. 2024 bidirectional alignment; Edelman et al. 2025 thick values; Bergman et al. 2024 STELA (IN LIBRARY, queued).
Li, J. (2026). Beyond Preference-based Value-alignment. IEAI Research Brief Q2 2026. Institute for Ethics in Artificial Intelligence, Technical University of Munich. https://www.ieai.sot.tum.de/research-brief-on-beyond-preference-based-value-alignment/
Close reading — 13 coded units
#1
· pp. 1
· claim
“Value alignment strategies in AI tend to assume that preferences are concrete manifestations of values, and preference-aligned machines are also value-aligned machines. However, scholars in philosophy increasingly note that holding this assumption may lead to shortcomings with alignment efforts.”
#2
· pp. 2
· definition
“From these early theories emerges the distinction between revealed preferences and ideal preferences (Samuelson, 1948). Revealed preferences refer to how people's desires are acted upon in real life through the choices they make [...] Ideal preferences refer to people's desires that would manifest if they were rational and informed in an ideal world [...] This distinction importantly points to the idea that people's actions may or may not reflect their actual desires.”
#3
· pp. 3
· definition
“Zhi-Xuan et al. (2024) label current value-alignment methods as falling under the preferentist account of value alignment. [...] the implicit claim behind these methods is that since preferences are assumed to be instantiations of values, aligning models to preferences would also amount to aligning with human values.”
#4
· pp. 3
· evidence
“AI used in healthcare settings may need to abide by not just individual hospital standards, but the ethical norms in the healthcare profession in which AI is being used. Nurses may be trained to focus on holistic care and patient advocacy, while physicians could focus more on accurate diagnosis and treatment. However, both professions operate according to bioethical principles such as autonomy, justice, beneficence and non-maleficence (Beauchamp & Childress, 1979). These differences may not be clear if the two groups are aggregated and, therefore, result in misalignment.”
#5
· pp. 3–4
· argument
“Therefore, AIs should move beyond individual preference satisfaction and toward contractualist and situated ways of alignment that account for the preferences of multiple stakeholders. Rather than committing energy to aligning AI systems to what Zhi-Xuan et al. (2024) describe as commitments to universal preference-based standards, AI systems could be more like purpose-built tools.”
#6
· pp. 4
· evidence
“A 2025 paper on cultural alignment found that survey methods which assess LLMs' cultural alignment fail to satisfy stability, extrapolability and steerability assumptions (Khan et al., 2025). The findings suggest that in some cases, alignment is often an artefact of evaluation design rather than a genuine property of models. [...] Another paper found that LLMs only displayed coherent value structures consistent with empirically backed theories of human values when they were given person descriptions and prompted to have a 'personality' (Rozen et al., 2024).”
#7
· pp. 4
· evidence
“This theory refers to the observable difference between what people say they prefer and what they do (LaPiere, 1934; Wicker, 1969). [...] A study on two decision-making experiments showed that approximately half of the participants displayed choice patterns inconsistent with standard models of rational preference (Cettolin & Riedl, 2019).”
#8
· pp. 4–5
· definition
“In philosophy, thin concepts such as 'bad' or 'good' carry evaluative force with minimal descriptive content, while thick concepts such as 'cruel' or 'honest' track features of the world and evaluate them [...] (Väyrynen, 2013; Williams, 1985). [...] According to Edelman et al. (2025), thin values are descriptively superficial, easily personalizable, and provide little to no justification for why they should exist as norms. [...] Alignment to thin values runs the risk of setting norms that arise from manipulation, addiction and power imbalances.”
#9
· pp. 5
· evidence
“they showed that common LLMs such as ChatGPT, Gemini and Copilot succeeded in identifying complex human principles, such as human dignity, when asked to conduct statistical pattern-matching. When the same LLMs were asked to identify human values in ambiguous situations that required understanding of causation and intentionality, they failed. [...] Strong value alignment follows three principles: an understanding of human values, the ability to reason about agents' intentions and the ability to represent the causal effects of actions (Khamassi et al., 2024).”
#10
· pp. 6
· argument
“participatory forms of alignment carry their own distributive risks, as the onus falls on societies and groups of individuals to produce and communicate their values. [...] Participatory and deliberative value alignment methods could exacerbate existing societal inequalities between those with the capacity to participate and those without.”
#11
· pp. 7
· argument
“If AI systems become more tightly embedded in society and agents are given greater autonomy, it could become more difficult to audit their behavior by inspecting individual points of value alignment. [...] Therefore, responsibility for AI systems needs to be distributed appropriately throughout the alignment process and continuously re-evaluated as systems and their uses evolve.”
#12
· pp. 7
· argument
“Jobin et al.'s (2019) survey of AI ethics guidelines finds broad convergence on abstract principles but significant divergence in how those principles are operationalized. This suggests that abstract ethical commitments are insufficient without domain-specific guidance.”
#13
· pp. 7
· claim
“The positioning of preferences as normative, or how things should be, when they are descriptive of how things are, within value alignment techniques, can be a source of technical and ethical issues. Alignment research should not continue advancing down a path that is philosophically naive about the concepts it is trying to operationalize.”