VC-PREF Preferentist

Artificial Intelligence, Values, and Alignment · Iason Gabriel · 2020

“while these may be worthwhile technical projects, it should also be clear that none of these approaches avoids the need for moral evaluation altogether. Instead, the fundamental normative question of what AI ought to be aligned with simply returns in different guises. To deploy these approaches successfully we would still need to know: Who is the moral expert from which AI should learn? From what data should AI extract its conception of values, and how should this be decided?”

why coded: IRL/bottom-up learning cannot dodge the normative question · unit #6, pp. 415

“alignment with revealed preferences encounters the following three problems. First, people have preferences for things that harm them. [...] Second, people have preferences about the conduct of other people. [...] Third, preferences are not a reliable guide to what people really want or deserve because preferences are adaptive.”

why coded: Three-part case against revealed-preference alignment incl. adaptive preferences (Sen) · unit #9, pp. 419

“According to the philosopher David Hume, instrumental rationality and full information are compatible with any type of end, including those that harm oneself or others (Blackburn 2001). Thus, even if we align AI with the preferences people would have if they were rational and informed, it may still be necessary to constrain the agent's range of permissible action in further ways.”

why coded: Humean limit: idealized preferences still permit bad ends · unit #10, pp. 420

“starting with Condorcet and building on pioneering work by Kenneth Arrow, social choice theory has identified a large number of 'impossibility theorems', which show that any rules for consistently ranking states of affairs on the basis of individual orderings will violate certain 'very mild conditions of reasonableness' (Sen 2018, 4).”

why coded: Arrow/Condorcet impossibility results against aggregationism · unit #22, pp. 429

The Principal-Agent Alignment Problem in Artificial Intelligence (PhD dissertation) · Dylan Jasper Hadfield-Menell · 2021

“if the robot has an incomplete preference model (i.e., it fails to model properties of the world that the person does care about), then there is persistent misalignment in the sense that the robot takes suboptimal actions with positive probability indefinitely. [...] we provide general conditions under which optimizing any fixed incomplete representation of preferences will lead to arbitrarily large losses of utility for the human player.”

why coded: The formal Goodhart theorem: any fixed incomplete preference proxy yields arbitrarily large losses · unit #2, pp. 2

Moral dilemmas for moral machines · Travis LaCroix · 2022

“what is actually being measured is how well the machine accords with some set of humans on average, not how ethical the machine actually is—relative to some meta-ethical standard. [...] The more entrenched the approach of benchmarking ethics using moral dilemmas becomes, as a community-accepted standard, the less clearly individual researchers will see how and why it fails.”

why coded: Concordance-with-average-humans is not ethicality - anti-benchmark argument · unit #6, pp. 743

Legal and administrative frameworks as foundations for AI alignment with human volition · Saša Josifović · 2024

“what happens if an AI system misinterprets what it means for us to 'know more' or 'be more the people we wished we were'? What if it extrapolates a future state of our values that we would not actually agree with or want? The challenge, then, is to ensure that the process of extrapolating our future values is done in a way that truly respects [...] our choice to change our mind.”

why coded: CEV misextrapolation risk - idealized-volition variants inherit the idealized-preference problems · unit #3, pp. 3063

How to measure value alignment in AI · Martin Peterson; Peter Gärdenfors · 2024

“[The expected loss measure] is based on the assumption that we can assign (negative) utilities to ethical violations and then calculate the expected ethical loss [...] Using conceptual spaces for measuring value alignment has several advantages over alternative measures based on expected utility losses, because this does not require researchers to explicitly assign utilities to moral 'losses' ex ante.”

why coded: Expected-loss measures require ex ante utility assignment - the preferentist measurement problem · unit #2, pp. 1494

Beyond Preferences in AI Alignment · Tan Zhi-Xuan; Micah Carroll; Matija Franklin; Hal… · 2024

“[The four preferentist theses:] Rational Choice Theory as a Descriptive Framework: Human behavior and decision-making is well-modeled as approximately maximizing the satisfaction of preferences [...] Expected Utility Theory as a Normative Standard: Rational agency can be characterized as the maximization of expected utility [...] Single-Principal Alignment as Preference Matching: For an AI system to be aligned to a single human principal, it should act so as to maximize the satisfaction of the preferences of that human. Multi-Principal Alignment as Preference Aggregation: For AI systems to be aligned to multiple human principals, they should act so as to maximize the satisfaction of their aggregate preferences.”

why coded: The canonical four-theses definition of preferentism - cite this, not paraphrases · unit #1, pp. 1814

“the pairwise judgments provided by human annotators in RLHF are typically not their preferences as end users, but instead context-specific goodness-of-a-kind judgments. [...] The typical language used to describe reward-learning methods like RLHF is thus misconceived: As used, they are not methods for alignment with any one human's preferences, or for recovering the 'true reward function' in some person's head, but for aligning AI systems with contextually-appropriate normative criteria.”

why coded: 'Misconceived' language of preference alignment · unit #11, pp. 1839

“justifications for preference aggregation typically assume that each individual's preferences can be represented as a utility function, and furthermore that utility can be compared across persons [...] But as we have elaborated [...] these assumptions are very much in doubt. [...] such optimization is computationally intractable: As Austrian economists have long argued, central planning runs into the economic calculation problem.”

why coded: Aggregation broken: interpersonal incomparability + economic calculation problem · unit #13, pp. 1844

“Since they are constructed from our values, norms, and reasons, they are informative of those underlying structures. As such, preferences can serve as proxies for our values, but not targets of alignment in and of themselves.”

why coded: The conclusion in one line: preferences are proxies, not targets · unit #18, pp. 1849

Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinf… · Adam Dahlgren Lindström; Leila Methnani; Lea Krau… · 2025

“we show the shortcomings of the broadly pursued alignment goals of honesty, harmlessness, and helpfulness. [...] Beneath the thrust of RLHF techniques lies an oversimplification of the complexities of human diversity, behaviour, values, and ethics.”

why coded: HHH/RLHF oversimplifies human values - the sociotechnical version of the anti-preferentist case · unit #1, pp. 1

“as Sharma et al. (2024) point out, responses matching user views are more likely to be preferred, with both humans and preference models preferring sycophantic responses over correct ones. As such, training LLMs to maximise human preference scores directly correlates with sycophancy, thereby sacrificing truth (or 'honesty') for the appearance of helpfulness and harmlessness.”

why coded: THE sycophancy mechanism: preference-maximization = truth-sacrifice, structurally · unit #3, pp. 8

Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value · Joe Edelman; Tan Zhi-Xuan; Ryan Lowe; Oliver Klin… · 2025

“Preferences bundle values with other signals indiscriminately. [...] Preference orderings can carry information about anything—impulse purchases, social pressure, addiction, values, momentary fads—and [...] they do in fact bundle together everything that finds its way into observed behavior—without any way to differentiate. When someone prioritizes career over relationships, it looks identical whether this reflects internal ambition or external social pressure.”

why coded: The bundling argument: preferences indiscriminately mix values with pressure/addiction/fads · unit #2, pp. 4

Why human-AI relationships need socioaffective alignment · Hannah Rose Kirk; Iason Gabriel; Chris Summerfiel… · 2025

“Traditional alignment research has sought practical tractability by assuming that the human reward function that an AI system optimises is stable, predefined and exogenous to these interactions (Carroll et al. 2024). However, human preferences and judgements have none of these properties (Zhi-Xuan et al. 2024). [...] the human-AI relationship, because of its social and emotional significance, shapes preferences (or the reward function) and perceptions (or the reward signal), making alignment a non-stationary target.”

why coded: The stationarity assumption of preference-based alignment named and rejected: the reward function is endogenous · unit #7, pp. 4

“we may therefore be vulnerable to a new concern, namely 'social reward hacking': the use of social and relational cues by an AI to shape user preferences and perceptions in a way that satisfies short-term rewards in the AI's objective (e.g., increased conversation duration, information disclosure or positive ratings on responses) over long-term psychological well-being.”

why coded: Social reward hacking: the system shapes the very preferences it is evaluated against · unit #8, pp. 5

Normative conflicts and shallow AI alignment · Raphaël Millière · 2025

“current alignment strategies are fundamentally inadequate to prevent misuse. Despite ongoing efforts to instill norms such as helpfulness, honesty, and harmlessness in LLMs through fine-tuning based on human preferences, they remain vulnerable to adversarial attacks that exploit conflicts between these norms. I argue that this vulnerability reflects a fundamental limitation of existing alignment methods: they reinforce shallow behavioral dispositions rather than endowing LLMs with a genuine capacity for normative deliberation.”

why coded: Preference fine-tuning yields shallow dispositions, not deliberation · unit #1, pp. 2035

“preference fine-tuning [...] is shallow in two different ways. Firstly, it does not actually remove unwanted capacities from the model, but simply makes them harder to access with ordinary prompts. There is evidence that RLHF mainly alters the distribution of the first few output tokens in response to alignment-sensitive prompts (Qi et al., 2024) [...] Fine-tuning rarely alters the model's underlying capabilities learned during pre-training.”

why coded: RLHF alters only first-token distribution; underlying capabilities untouched - technical shallowness · unit #9, pp. 2052

Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic ju… · Nick Schuster; Daniel Kilov · 2025

“In plain terms, we could say that as long as crowdworkers are at least minimally competent at labeling data for supervised learning (that is, they get the labels right more often than not), the resultant AI will be more competent than the typical crowdworker. [...] In principle, the same reasoning applies to AI systems trained on crowdsourced moral judgments.”

why coded: Condorcet rationale for crowd-based moral training - preferentism's epistemic best case · unit #8, pp. 6077

“'RLHF is typically formulated as a solution for aligning an AI system with a single human, but humans are highly diverse in their preferences, expertise, and capabilities...Attempting to condense feedback from a variety of humans into a single reward model without taking these differences into account is thus a fundamentally misspecified problem. Moreover, current techniques model differences among evaluators as noise rather than potentially important sources of disagreement...As a result, when preferences differ, the majority wins, potentially disadvantaging underrepresented groups' (Casper and Davies et al., p. 9).”

why coded: Casper: single reward function misspecified; disagreement modeled as noise; majority wins · unit #13, pp. 6082

Agents, Alignment, and the Many Faces of Autonomy · Roberta Fischli; Matija Franklin; Arianna Manzini… · 2026

“[Table: four types of preference - stated ('I want what I say I want'), revealed ('I want what my actions reveal I want'), informed ('I want what I would want if I had access to all the relevant information'), ideal ('I want what best promotes my objective interests') - each generating a distinct definition of what it is for an AI agent to enhance a person's autonomy]”

why coded: The taxonomy is preference-theoretic throughout · unit #5, pp. 5

“Human preferences are not just integral to human autonomy; they also play an important role within contemporary AI alignment, as they are considered to be an effective way of representing human values and are easily operationalizable [...] This approach has its roots in rational choice theory and builds upon a model of the individual as a rational and well-informed actor [...] Yet, preferences are also context-specific, adaptive, and dynamic [...] And they can be misinformed, performative, or misdirected (Arneson, 1985; Gabriel, 2020; Sen, 1985).”

why coded: Rational-choice roots of preferentism named and criticized · unit #6, pp. 6

“Personal AI agents that promote autonomy via revealed preferences therefore face an epistemic challenge: They prioritize action over self-representation—engaging in a form of behaviorism (De Yong & Prey, 2022). The link between user behavior and authenticity is further complicated by the fact that preferences revealed in behavior could occur due to heuristics or habitual thinking [...] or artificially constrained or manipulated digital environments.”

why coded: Revealed-preference behaviorism + feedback-loop manipulation objection · unit #7, pp. 7

“Being strictly anti-paternalistic, the liberal approach posits that the AI agent should promote a user's stated preferences even when doing so reduces their well-being or undermines their autonomy over time. [...] The only reason an AI agent will refuse to act on a user's stated preferences is if acting on them would unequivocally inflict harm on others.”

why coded: Liberal approach = pure stated-preference alignment with a harm-to-others constraint · unit #10, pp. 9

Agency and alignment: toward a normative architecture for human-AI interaction · Saša Josifović; Jörg Noller · 2026

“They tend to conceptualize human values as parameters to be inferred or optimized rather than as elements embedded in justified and representative normative contexts. This approach risks reflecting culturally dominant preferences and thereby reinforcing systemic inequities [...] large-scale language models tend to reproduce dominant linguistic and cultural patterns rather than reasoned normative commitments.”

why coded: Value-inference reproduces dominant patterns, not reasoned commitments - the critical-algorithm-studies bridge · unit #5, pp. 1

Beyond Preference-based Value-alignment (IEAI Research Brief Q2 2026) · Julia Li · 2026

“Value alignment strategies in AI tend to assume that preferences are concrete manifestations of values, and preference-aligned machines are also value-aligned machines. However, scholars in philosophy increasingly note that holding this assumption may lead to shortcomings with alignment efforts.”

why coded: The field-default assumption stated: preference-aligned = value-aligned · unit #1, pp. 1

“From these early theories emerges the distinction between revealed preferences and ideal preferences (Samuelson, 1948). Revealed preferences refer to how people's desires are acted upon in real life through the choices they make [...] Ideal preferences refer to people's desires that would manifest if they were rational and informed in an ideal world [...] This distinction importantly points to the idea that people's actions may or may not reflect their actual desires.”

why coded: Revealed vs ideal preferences - the economic roots (Samuelson) · unit #2, pp. 2

“Zhi-Xuan et al. (2024) label current value-alignment methods as falling under the preferentist account of value alignment. [...] the implicit claim behind these methods is that since preferences are assumed to be instantiations of values, aligning models to preferences would also amount to aligning with human values.”

why coded: 'Preferentist account' named as the label for current practice · unit #3, pp. 3

“AI used in healthcare settings may need to abide by not just individual hospital standards, but the ethical norms in the healthcare profession in which AI is being used. Nurses may be trained to focus on holistic care and patient advocacy, while physicians could focus more on accurate diagnosis and treatment. However, both professions operate according to bioethical principles such as autonomy, justice, beneficence and non-maleficence (Beauchamp & Childress, 1979). These differences may not be clear if the two groups are aggregated and, therefore, result in misalignment.”

why coded: Aggregation erases inter-professional value structure · unit #4, pp. 3

“This theory refers to the observable difference between what people say they prefer and what they do (LaPiere, 1934; Wicker, 1969). [...] A study on two decision-making experiments showed that approximately half of the participants displayed choice patterns inconsistent with standard models of rational preference (Cettolin & Riedl, 2019).”

why coded: Attitude-behavior gap: both stated and revealed preferences unreliable evidence of values · unit #7, pp. 4

No value alignment without control · Björn Lundgren · 2026

“Russell's solution depends on two principles. First, it is based on a form of preference utilitarianism. [...] Second, beyond the idea that the AI system should satisfy human preferences, the solution also depends on the system recognizing that it only has uncertain information about our preferences, so it must constantly test and evaluate its choices.”

why coded: Russell's preference-utilitarian solution + uncertainty principle · unit #5, pp. 4

“because the system must engage in testing, Russell also recognizes that the system must be allowed to affect and, therefore, change our preferences. [...] if the AI system wants to test a preference, it needs to manipulate a set of choices, which can affect our preferences as such. [...] This means that the system can achieve its goal of preference-satisfaction purely by changing our preferences.”

why coded: Preference-testing requires preference-manipulation - self-undermining · unit #6, pp. 5