Beyond Preferences in AI Alignment
Tan Zhi-Xuan; Micah Carroll; Matija Franklin; Hal Ashton · 2024 · Philosophical Studies 182:1813-1863 anchor high priority coded
Main argument
Thesis: the dominant 'preferentist' approach to alignment - four theses: (1) rational choice theory as descriptive model, (2) expected utility theory as normative standard, (3) single-principal alignment as preference matching, (4) multi-principal alignment as preference aggregation - fails on all four fronts, and alignment should be refounded on role-specific normative standards negotiated contractualistically. Argument type: critical review + research agenda (deliberately broad, brief arguments). Key moves: 'preference' is a THIN concept - a bare betterness ordering - while human values are THICK evaluative concepts constructed from reasons, so preferences are proxies for values, not targets (Evaluate-Commensurate-Decide model of decision); utility representations assume away incommensurability, which is empirically actual; EUT is instrumental rationality only and cannot say which preferences are normatively acceptable; RLHF's language is 'misconceived' - annotator judgments are already role-specific goodness-of-a-kind judgments (helpfulness/harmlessness), so existing practice covertly aligns to norms, not preferences; aggregate-preference optimization is theoretically broken (incomparability), computationally intractable (economic calculation problem), and politically infeasible ('tyranny of creator values'); the successor: align a MULTIPLICITY of AI systems each with normative standards appropriate to its social role, standards negotiated and justified to each stakeholder on grounds none can reasonably reject (broadly contractualist, Rawls/Scanlon), with legitimacy required where AI exercises power.
Why it matters here
The spine of the anti-preferentist turn - the paper the IEAI brief synthesizes and the Fischli paper cites as companion. Names and dismantles the four theses of 'preferentism' and proposes the successor framework (role-specific norms + contractualist negotiation) that the whole 2024-2026 literature now orbits. MIT/Berkeley/UCL/Cambridge; self-described interdisciplinary research agenda, deliberately broad.
Reading notes
Close read of intro, 2.3, 3.3, 4.3, 5.2 and conclusion (51pp; middle technical subsections skimmed via structure). Note two authors overlap with FISCHLI_2026 (Franklin) and the Carroll preference-change line. Cites Gabriel & Keeling 2024/2025 for the 'explicitly political conception of AI alignment' - now in library.
Zhi-Xuan, T., Carroll, M., Franklin, M., & Ashton, H. (2024). Beyond Preferences in AI Alignment. Philosophical Studies, 182, 1813-1863. https://doi.org/10.1007/s11098-024-02249-w
Close reading — 18 coded units
“[The four preferentist theses:] Rational Choice Theory as a Descriptive Framework: Human behavior and decision-making is well-modeled as approximately maximizing the satisfaction of preferences [...] Expected Utility Theory as a Normative Standard: Rational agency can be characterized as the maximization of expected utility [...] Single-Principal Alignment as Preference Matching: For an AI system to be aligned to a single human principal, it should act so as to maximize the satisfaction of the preferences of that human. Multi-Principal Alignment as Preference Aggregation: For AI systems to be aligned to multiple human principals, they should act so as to maximize the satisfaction of their aggregate preferences.”
“'preference' is a thin concept because it does not encode richer semantic information beyond the bare notion of 'betterness'. [...] But why exactly are some options preferred over others? In virtue of what reasons do people make these preference judgments? Without answering these questions, we are unlikely to model how someone's preferences generalize to novel options in ways they would endorse. To do so, we must go beyond preferences as the fundamental unit of analysis, and understand how preferences are computed and constructed from our reasons and values.”
“they are thick evaluative concepts—concepts that comprise both descriptive and normative elements—such as beauty, humor, or health. As Blili-Hamelin and Hancox (2023) point out, even the concept of intelligence so central to AI is thick in this way.”
“if utility functions are used to represent aggregate value judgments, this effectively assumes that distinct human values are always commensurable in some way, and that our resulting preferences are always complete. Yet, as value pluralists argue, there are contexts where it seems hard or impossible to commensurate our values (Anderson, 1995), resulting in choices where our reasons run short, and we cannot say if one option is ultimately better than another (Chang, 1997).”
“we propose that human decisions can be productively modeled as a three-stage process: Evaluate, Commensurate, then Decide (ECD). Given some choice options, a set of evaluation procedures compute valuations or rankings of the options under consideration, where each procedure corresponds to a distinct evaluative concept. These valuations serve as inputs to a commensuration procedure [...] which produces, where possible, a context-sensitive value assignment or preference ordering over the options (optionally with justifications for why certain trade-offs were made), while leaving certain preferences unspecified when some options are not comparable.”
“LLMs appear to learn the conceptual roles associated with particular words [...] and even perform rudimentary forms of moral reasoning (Jin et al., 2022). Still, LLMs remain limited in their ability to represent and reason with compositional concepts [...] and would function as poor models of humans on their own.”
“Within each class of trajectories with a fixed schedule of k contexts [...] there is a complete preference ordering over trajectories. Across these classes, trajectories are incomparable, leading to preferential gaps. Agents with such preferences would still optimize their behavior while within each context. At the same time, they would exhibit no reliable disposition towards being in some contexts more than others, or manipulating the schedule of contexts. At least in the sense we identified earlier, they would function as tools.”
“an arguably deeper problem with EUT is that it fails to ground the normativity of our preferences. EUT is a theory of instrumental rationality not value rationality: It tells us how to choose our actions in order to satisfy our preferences, and imposes constraints on what those preferences can be, but it does not say anything further about where those preferences can or should come from.”
“even without replacing human autonomy over normative affairs, we are already building AI systems that automate normative judgments [...] This unreliability suggests that we might want formal theories of normative reasoning after all. Without such theories, we would have no general way of evaluating whether an AI system reasons 'correctly', beyond comparison to often fallible human judgments.”
“Recognizing these issues, Gabriel (2020) argues for an explicitly moral conception of alignment [...] However, it is far from clear how to operationalize these abstract principles. To make progress, we suggest a conception of single-principal alignment that is significantly more constrained: When an AI system only serves an individual in performing a particular task or role, it should be aligned with the normative ideals or criteria that are appropriate for that role. [...] For general-purpose AI assistants, this implies alignment with the normative ideal of an assistant.”
“the pairwise judgments provided by human annotators in RLHF are typically not their preferences as end users, but instead context-specific goodness-of-a-kind judgments. [...] The typical language used to describe reward-learning methods like RLHF is thus misconceived: As used, they are not methods for alignment with any one human's preferences, or for recovering the 'true reward function' in some person's head, but for aligning AI systems with contextually-appropriate normative criteria.”
“a good assistant is aware that some choices are hard, and some options may seem incomparable (Chang, 1997). When helping someone with such a choice, the assistant does not pretend to know which option is better, or try to optimize that person's life; instead, the assistant respects their autonomy, and empowers them to make the most informed choice possible [...] while ultimately remaining agnostic as to which choice is 'best'.”
“justifications for preference aggregation typically assume that each individual's preferences can be represented as a utility function, and furthermore that utility can be compared across persons [...] But as we have elaborated [...] these assumptions are very much in doubt. [...] such optimization is computationally intractable: As Austrian economists have long argued, central planning runs into the economic calculation problem.”
“the project of building AI that optimizes humanity's aggregate preferences is politically infeasible [...] Allowing the creation of such AI systems would also risk the centralization of immense power [...] Instead, we are more likely to see a tyranny of creator values, with potentially disastrous consequences for everyone with a contrary way of life.”
“Rather than learning humanity's preferences in order to maximally satisfy them, AI systems should be aligned with normative standards and criteria that we collectively forge and negotiate—standards exemplified by social, legal, and moral norms. [...] Just as AI assistants should avoid harmful language, self-driving cars should follow the rules of the road.”
“contractualist alignment aims to align AI systems with goals, standards, and principles that are mutually agreed upon by people despite our disparate preferences and values, deriving its normative force from the fair and impartial agreement of relevantly-situated rational actors. [...] AI goals and standards should be justified to each stakeholder, on grounds that none can reasonably reject. Insofar as these AI systems are used to exercise power over others, they should also act in accordance with standards that are not just fair, but legitimate.”
“if we take fair and impartially negotiated standards as the target of AI alignment, then technical advances will not be enough; we also need to foster the development of social, economic, and political orders that provide the conditions for free and fair agreement. [...] After all, if we are going to align AI systems with normative standards we would collectively endorse, then we had better make sure that a 'we' exists to endorse them.”
“Since they are constructed from our values, norms, and reasons, they are informative of those underlying structures. As such, preferences can serve as proxies for our values, but not targets of alignment in and of themselves.”
Synthesis-matrix row
supports
T2-PREFERENTISM-BROKEN
the four theses dismantled; preferences as proxies not targets
complicates
T3-PROCEDURALISM-INCOMPLETE
contractualist norm-negotiation; concedes Gabriel-2020 not operationalizable
supports
T4-ROSSIAN-DEMAND
incommensurability + incomplete-preference formalisms = implementable Ross
complicates
T7-AGENTIC-BREAKS-FRAMES
containment response: engineered tool-likeness
Memos (5)
theoretical · unit #5
The ECD model (unit 5) plus 'preferences are constructed from reasons and values' (units 2, 18) is the strongest available philosophical legitimation of the xphi corpus methodology: the folk_ai.db coding scheme captures exactly what Zhi-Xuan et al. say alignment data SHOULD capture - the reasoning behind judgments (reasoning field = their 'justifications for trade-offs'), the values invoked (value_primary = their 'distinct evaluative concepts'), and the stance (their commensuration output) - rather than bare preference orderings. The methodology chapter can claim: where RLHF collects thin preference data, the xphi corpus collects thick, reason-annotated evaluative data - i.e., it already implements the post-preferentist data model the field's leading agenda paper calls for.
theoretical · unit #4
Incommensurability (unit 4) is the decision-theoretic face of Rossian pluralism: Ross's plural prima facie duties with no fixed exchange rate = Chang/Anderson's incommensurable values = Zhi-Xuan's incomplete preferences and vector-valued rewards. This completes the answer to GABRIEL_2020 unit 3's challenge (RL architecture favors consequentialism): Zhi-Xuan supplies the technical machinery (multi-objective RL, lexicographic orderings, preferential gaps, unit 7's tool-like locality) by which a Rossian pluralist architecture could actually be implemented. The dissertation's metaethics chapter can cite this as proof that Rossian structure is computationally realizable, not just philosophically attractive.
comparison · unit #15
The field's convergence on contractualism is now fully documented across the coded set: Gabriel 2020 (procedural fairness), Zhi-Xuan 2024 (contractualist norm-negotiation, Scanlonian justifiability), IEAI 2026 (contractualist purpose-built tools), Schuster & Kilov 2025 (legitimacy conditions), and Gabriel & Keeling 2025 is titled 'the fair treatment of claims'. The dissertation MUST position Rossian convergentism relative to this consensus. Preferred line: complementarity with division of labor - contractualism answers the POLITICAL question (whose standards, by what authority) while Rossian pluralism answers the OBJECT-LEVEL question (what considerations bear on this case and how they weigh), and convergentism bridges them (cross-framework convergence supplies the reasons-none-can-reasonably-reject that Scanlonian agreement needs). Note Zhi-Xuan unit 16 derives normative force FROM agreement; the dissertation derives agreement FROM convergent normative force - a genuine, arguable difference.
thesis-link · unit #10
Role-specific normative standards (units 10-11, 15) map directly onto the three case studies: AI Scribe = the clinical-documentation role (norms from medical professional ethics - cf. LI_2026 unit 4's nursing/medicine example), Chibook = the immigration-advisory role (norms from administrative justice), AI Interviewer = the hiring role (norms from employment equity - cf. SCHUSTER_KILOV unit 12). The dissertation can test the role-norms framework empirically: does the stakeholder corpus show people actually reasoning in role-normative terms rather than preference terms? If yes, that is empirical support for Zhi-Xuan's framework from data they never had; if no, a documented gap between the philosophical proposal and folk practice. Either result is a contribution.
comparison · unit #7
Tension worth exploiting between ZHIXUAN unit 7 and FISCHLI_2026: Zhi-Xuan propose engineering preference INCOMPLETENESS so agents stay 'tool-like' (no context-manipulation incentives) - i.e., their answer to agentic risk is to prevent full agency. Fischli et al. (same research community, Franklin on both papers) accept increasingly agentic assistants and manage autonomy trade-offs instead. The field's two live responses to agentic AI are thus CONTAINMENT (Zhi-Xuan) vs GOVERNANCE-BY-DESIGN (Fischli) - and neither addresses responsibility attribution when containment fails or governance misfires. The dissertation's responsibility-first framing cuts across both.