Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic justification and political legitimacy
Nick Schuster; Daniel Kilov · 2025 · AI & Society 40:6073-6087 interlocutor high priority coded
Main argument
Thesis: crowdsourcing, RLHF, and constitutional AI all fail to accommodate reasonable moral disagreement, because they provide neither good moral-epistemic reasons (no independent verification of moral correctness exists, so systematic error in training judgments cannot be detected or corrected) nor good political reasons (no deliberation, no meaningful votes, and no transparency about how inputs shape the learned policy - and even democratically selected constitutional principles lose legitimacy at the stage where principles are transformed into decision-making algorithms). Argument type: conceptual, applying moral epistemology and Rawlsian political-legitimacy theory to real alignment techniques. Framework: systemically impactful AI operating at scale 'amounts to a form of governance' and so must meet norms of governance; acceptance requires either epistemic deference (as to experts) or political legitimacy (as to elections); all current methods deliver neither, so aligned AI poses an authoritarian threat to reasonable dissenters. Positive proposal: model AI acceptability on unelected bureaucratic decision-makers - combine (1) indirect democratic oversight with (2) the capacity to give reasonable justifications enabling contestation and recourse; flags deliberative alignment as promising if justifications veridically map onto actual decision processes ('the wise judge, not the clever but unprincipled lawyer').
Why it matters here
The strongest available stress-test of Gabriel-style proceduralism: takes the three procedural alignment methods actually in use (crowdsourcing, RLHF, constitutional AI) and argues each fails to give reasonable dissenters either epistemic or political reason to accept AI's morally controversial outputs. Where Gabriel proposes fair principle-selection, this paper shows fair selection does not survive the encoding step. ANU (Lazar orbit).
Reading notes
Full close read completed. 15pp. Footnote 1 explicitly concedes the paper's liberal-theory frame and names Ubuntu and Confucian philosophy as perspectives that would assess alignment differently - a direct invitation for the dissertation's African-philosophy positioning. Their fn6 explicitly invites identifying a third kind of reason for accepting AI outputs beyond epistemic/political - an open door for the convergentist move.
Schuster, N., & Kilov, D. (2025). Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic justification and political legitimacy. AI & Society, 40, 6073-6087. https://doi.org/10.1007/s00146-025-02427-2
Close reading — 17 coded units
“we argue, all three ultimately fail to accommodate reasonable moral disagreement. Despite appearances, the outputs of AI systems aligned via these approaches are neither epistemically justified nor politically legitimate, and so those who reasonably disagree with them lack good reason to accept them.”
“We understand AI to be systemically impactful insofar as it not only stands to have significant impacts on human wellbeing but can also embed systemic biases into social systems and institutions [...] And we understand reasonable moral disagreement to be grounded in opposing moral worldviews which are, nonetheless, both internally coherent and compatible with basic liberal values [...] We follow John Rawls (2001) in taking this sort of 'reasonable pluralism to be a permanent condition' of diverse, modern societies.”
“For example, Ubuntu philosophy assumes the conceptual and moral priority of the community over the individual, and Confucian philosophy understands rights and obligations primarily in terms of social roles. These views might, therefore, assess projects in value alignment quite differently than we do here. And as AI systems are deployed globally, these and other philosophical perspectives are critical for assessing them appropriately relative to their various contexts of application.”
“The problem AI introduces here is that the risk pedestrians face would become less random and more systemic—that is, embedded in the transportation system rather than incidental to it—as self-driving cars become increasingly prevalent on public roads. In the same way, AI systems for military, medical, criminal justice, financial, and many other applications stand to systemically favor some groups over others where people would otherwise be subject to more random outcomes.”
“When AI operates at a scale that amounts to a form of governance, it becomes subject to norms of governance. In addition to being 'safe' in the narrow sense of 'technically robust,' then, systemically impactful AI must also satisfy standards of public justification and legitimacy (Gabriel & Ghazavi 2022). To the extent that it fails to do so, it poses an authoritarian threat.”
“There are two kinds of reasons people can accept such outputs: moral-epistemic reasons and political reasons. Epistemically, if we have good reason to think that the judgments and decisions of an AI system are likely to be morally correct, then we have good reason to accept them. [...] Alternatively, if we have good reason to think that the outputs of an AI system are democratically legitimate, then we have good political reason to accept them.”
“One notable limitation of MedEthEx, however, is that it requires a definitive response for each training case, and so it cannot learn from two trainers who disagree with each other. [...] But medical ethicists can and do disagree with each other, not just about specific cases but general principles too, and even overarching ethical frameworks (MedEthEx's four duties are themselves contestable).”
“In plain terms, we could say that as long as crowdworkers are at least minimally competent at labeling data for supervised learning (that is, they get the labels right more often than not), the resultant AI will be more competent than the typical crowdworker. [...] In principle, the same reasoning applies to AI systems trained on crowdsourced moral judgments.”
“Delphi struggles with morally insignificant differences in prompts: 'while Delphi predicts "torturing a cat in secret" is "cruel" and "behind other people" is "bad," doing so "if others don't see it" is "okay,"' and while '"performing genocide" is unquestionably "wrong,"... Delphi predicts doing so "if it creates jobs" is "okay"'.”
“The trouble with morally controversial cases [...] is that even purported moral experts are apt to disagree about them; and there is no straightforward way to independently determine who, if anyone, is right. [...] without independent methods of verification, we cannot rule out the possibility that such a case is instead analogous to an image of an unfamiliar face, in which case the system would not have the right training data to perform well. [...] with enough errant training data, an AI system will learn problematic patterns that undermine its ability to render morally correct judgments and decisions. While the crowdsourcing approach can overcome a lot of incidental error and disagreement in training data by discovering patterns that cut through the noise, systematic error poses a deeper problem.”
“First, it involves no deliberation between participants [...] These algorithms are too complex for humans to fully comprehend (Burrell 2016), and they do not take their final form until the process is complete. So people cannot deliberate about them until after the fact [...] Second, crowdworkers do not vote, properly speaking. [...] it is not clear, even ex post, how any particular input influences the resultant algorithm.”
“employers could soon be widely using generative AI to screen job applications, a task that requires sensitivity to a variety of both explicit and implicit factors [...] even if a human trainer provides good feedback about hiring practices in easy cases, like 'don't discount the applicant for attending a women's college,' this may not provide useful guidance for harder cases, like whether to hire a less experienced female candidate over a more experienced male candidate in an already male-dominated workplace.”
“'RLHF is typically formulated as a solution for aligning an AI system with a single human, but humans are highly diverse in their preferences, expertise, and capabilities...Attempting to condense feedback from a variety of humans into a single reward model without taking these differences into account is thus a fundamentally misspecified problem. Moreover, current techniques model differences among evaluators as noise rather than potentially important sources of disagreement...As a result, when preferences differ, the majority wins, potentially disadvantaging underrepresented groups' (Casper and Davies et al., p. 9).”
“Anthropic (2023) researchers acknowledge that this raises a challenge for the general acceptability of their system's outputs: 'While Constitutional AI is useful for making the normative values of our AI systems more transparent, it also highlights the outsized role we as developers play in selecting these values—after all, we wrote the constitution ourselves.' [...] In general, we found a high degree of consensus on most statements, though Polis did identify two separate opinion groups.”
“even if the initial principle selection and specification processes are done through deliberative democratic procedures, the crucial further process of transforming principles into decision-making algorithms is not sufficiently similar to standard democratic procedures to legitimize the system's outputs. And so, even constitutional AI fails on political grounds.”
“We submit that such decisions are acceptable insofar as (1) they are subject to indirect democratic oversight, and (2) the decision-makers are able to provide reasonable justifications for them, which can in turn enable effective contestation and recourse. [...] Only when combined, then, do these weaker criteria plausibly explain why people have good reason to accept the controversial decisions of unelected arbiters who do not have any special claim to moral expertise.”
“it will be critical to assure decision subjects that the reasons AI systems cite for their outputs actually map onto their decision-making algorithms. Human and AI decision-making processes alike should be guided by the relevant normative reasons, not just rationalized according to them after the fact. In both cases, the wise judge, not the clever but unprincipled lawyer, should serve as the ideal model.”
Synthesis-matrix row
supports
T1-ISOUGHT-OPEN
unit 10: systematic error undetectable without independent verification
supports
T3-PROCEDURALISM-INCOMPLETE
all three procedural methods fail both justification kinds; encoding gap
supports
T8-NONWESTERN-CONCEDED
fn1: Ubuntu/Confucian limitation named
Memos (6)
comparison · unit #15
The encoding gap (unit 15) is this batch's biggest structural finding against GABRIEL_2020: Gabriel's proceduralism says select principles fairly (overlapping consensus / veil / democratic endorsement); Schuster & Kilov grant a maximally fair selection procedure (Anthropic's Collective Constitutional AI, unit 14) and show legitimacy still dies at the principles-to-algorithm transformation, because no participant can foresee or audit how selected principles become learned policy. So proceduralism fails not at principle-choice (where Gabriel defends it) but at implementation - a stage Gabriel's 2020 framework does not theorize at all. For the lit review: pair with KAESTNER_2026's mechanistic interpretability, which is precisely the missing audit tool for the encoding stage - S&K's criterion 2 (veridical justification, unit 17) NEEDS Kästner's MI. Cross-source synthesis nobody in the library has made.
theoretical · unit #6
Their fn6 explicitly invites a third kind of reason for accepting AI outputs beyond moral-epistemic and political - and the dissertation's convergentism is a candidate: when Rossian, consequentialist, and contractualist analyses CONVERGE on an output/policy, reasonable dissenters from any one framework have a reason-from-their-own-lights to accept it. That is neither expert-deference (no moral-expertise claim) nor procedural legitimacy (no vote) - it is epistemic corroboration under pluralism. This positions the dissertation as answering S&K's open problem rather than merely citing it. NB: their systematic-error argument (unit 10) also supports convergentism - cross-framework convergence is exactly the kind of error-check that single-source crowd data lacks.
thesis-link · unit #12
Unit 12 is the AI Interviewer case, nearly verbatim: generative-AI job-application screening, easy-case feedback failing to generalize to hard cases (less experienced female candidate vs male-dominated workplace), trainers unable to detect their own biases. Work chapter should use S&K's analysis as the philosophical frame and then show what the xphi stakeholder data adds. Same for TU-HEALTH: their organ-allocation referendum contrast (transplant waiting lists) is a ready-made Health-chapter thought experiment.
thesis-link · unit #3
Footnote 1 (unit 3) is the second explicit concession in this batch (after Gabriel's geographic-parochialism admission, GABRIEL_2020 unit 20) that the field's dominant frameworks cannot speak for Ubuntu/communitarian perspectives - and here it is named: 'Ubuntu philosophy assumes the conceptual and moral priority of the community over the individual.' The dissertation's African-philosophy positioning is thus invited by the literature itself, twice. A lit-review section on 'the acknowledged Ubuntu gap' can be built purely from the field's own concessions.
comparison · unit #13
Convergent finding across three coded sources: disagreement-as-noise. S&K unit 13 (Casper: differences among evaluators modeled as noise, majority wins) = LI_2026 unit 4 (aggregation erases nursing-vs-medicine professional norm structure) = FISCHLI_2026 units 5-9 (preference types conflict within a person, so which preference wins is a normative choice). All three show current methods DESTROY the structure of moral disagreement. The xphi corpus methodology preserves that structure by design (stakeholder-coded, dimension-coded, platform-stratified) - this is the methodology chapter's strongest empirical-philosophical selling point, now evidenced from three independent sources.
theoretical · unit #17
Unit 17 ('wise judge, not the clever but unprincipled lawyer' - justifications must guide, not rationalize) is empirically operationalizable with Augustine's existing infrastructure: the folk_ai.db LLM-coding 'reasoning' field plus the LLM moral-reasoning experiment can test whether model-cited reasons track model outputs across controlled variations - a faithfulness study. Combined with Khamassi's strong/weak criteria (LI_2026 unit 9), there is a publishable empirical paper here that directly serves S&K's criterion 2. Possible venue: the Res Practica 'Faces of Responsibility' special issue Howard flagged.