Artificial Intelligence, Values, and Alignment
Iason Gabriel · 2020 · Minds and Machines anchor high priority coded
Main argument
Thesis: the central challenge of value alignment is not to identify the true moral theory and encode it in machines, but to identify FAIR principles for alignment that command reflective endorsement despite reasonable pluralism. Argument type: conceptual (political philosophy applied to AI). Three moves: (1) technical and normative aspects are interdependent - RL's optimizing architecture is structurally friendlier to consequentialism than to rights-based or Kantian frameworks, so method constrains which values can be loaded; (2) a six-way taxonomy of alignment loci (instructions, expressed intentions, revealed preferences, informed preferences, interests, values) in which each locus short of values fails for principled reasons (literalism, faulty intentions, preference degeneracy/adaptivity, Humean limits of rationality, interests without entitlements); (3) since no single moral theory is both true-beyond-doubt and communicable under reasonable pluralism, imposing one would be domination, so principle-selection must be procedural - via global overlapping consensus (human rights), veil-of-ignorance reasoning, or social-choice/democratic endorsement. Conclusion: alignment is a political problem; encoding the corpus of present moral belief directly would wrongly 'tether AI to the morality of the present moment'.
Why it matters here
Canonical statement of the value-alignment problem in philosophical terms; distinguishes alignment with instructions/intentions/revealed preferences/ideal preferences/interests/values and defends a principle-based, procedural approach (fair principles under reflective endorsement rather than 'true' morality). Baseline against which the dissertation tests whether pre-agentic alignment frameworks survive agentic AI; its proceduralism is also the main rival to the dissertation's Rossian-convergentist architecture.
Reading notes
Full close read completed. 27pp (Minds & Machines 30:411-437). The canonical pre-agentic statement of the normative alignment problem; every later source in the library positions itself relative to this paper. Structure: tech/normative interrelation -> six loci of alignment -> proceduralist ('political not metaphysical') answer to pluralism.
Gabriel, I. (2020). Artificial Intelligence, Values, and Alignment. Minds and Machines, 30, 411-437. https://doi.org/10.1007/s11023-020-09539-2
Close reading — 25 coded units
“How are we to decide which principles or objectives to encode in AI—and who has the right to make these decisions—given that we live in a pluralistic world that is full of competing conceptions of value? Is there a way to think about AI value alignment that avoids a situation in which some people simply impose their views on others?”
“The second part of the value alignment question is normative. It asks what values or principles, if any, we ought to encode in artificial agents. Here it is useful to draw a distinction between minimalist and maximalist conceptions of value alignment. The former involves tethering artificial intelligence to some plausible schema of human value and avoiding unsafe outcomes. The latter involves aligning artificial intelligence with the correct or best scheme of human values on a society-wide or global basis.”
“In general, it seems likely that it will be easier to align AI with moral theories that have the same fundamental structure based on maximizing reward over time in the face of uncertainty, than with other alternatives. Consequentialist moral theories, the most famous of which is act utilitarianism, fit the bill.”
“Although this remains to be seen, it may be difficult to robustly specify and guarantee rights-respecting behaviour on the part of agents whose learning process and decision-making are guided primarily by an optimization function.”
“both Kantian and contractualist moral theories require that an agent understand the concept of a 'reason' and subject it to certain kinds of hypothetical test before knowing how to proceed—capabilities that extend well beyond most existing forms of artificial agent”
“while these may be worthwhile technical projects, it should also be clear that none of these approaches avoids the need for moral evaluation altogether. Instead, the fundamental normative question of what AI ought to be aligned with simply returns in different guises. To deploy these approaches successfully we would still need to know: Who is the moral expert from which AI should learn? From what data should AI extract its conception of values, and how should this be decided?”
“It follows from this distinction that we cannot work out what we ought to do simply by studying what is the case, including what people actually do, or what they already believe. Simply put, in each case, people could be mistaken. Because of this, AI cannot be made ethical just by learning from people's existing choices. [...] the value alignment problem cannot be solved by inference from large bodies of human-generated data by itself.”
“if powerful AI systems function at super-human speed, which seems likely, then it may not be possible to provide them with immediate and continuous direction in this way (Russell et al. 2015; Soares 2014). Instead, artificial agents would need to be able to make sound decisions by default, including in unforeseen situations, without explicit instructions or well-formed intentions from a human operator.”
“alignment with revealed preferences encounters the following three problems. First, people have preferences for things that harm them. [...] Second, people have preferences about the conduct of other people. [...] Third, preferences are not a reliable guide to what people really want or deserve because preferences are adaptive.”
“According to the philosopher David Hume, instrumental rationality and full information are compatible with any type of end, including those that harm oneself or others (Blackburn 2001). Thus, even if we align AI with the preferences people would have if they were rational and informed, it may still be necessary to constrain the agent's range of permissible action in further ways.”
“the theory of human development argues that welfare stems from the ability to exercise certain core capabilities that both constitute and support human flourishing (Sen 2001). These capability-based metrics have found a measure of cross-cultural affirmation and consent (Nussbaum 1993).”
“Viewed from the perspective of a single person, the fact that something is in my interest doesn't mean I ought to do it or that I am morally entitled to do so. [...] First, we need a way of deciding how to manage trade-offs between the interests and claims of different people. [...] Second, we need principles for deciding whose interests or needs count for the purpose of AI alignment.”
“The metaethical debate may seem critical. After all, if values do not have this objective basis, how can AI be developed to align with them? Yet these concerns turn out to have limited significance for the question at hand. To see why, we need to acknowledge first that, in practice, AI would have to be aligned with some set of beliefs about value, not with value itself.”
“the task in front of us is not, as we might first think, to identify the true or correct moral theory and then implement it in machines. Rather, it is to find a way of selecting appropriate principles that is compatible with the fact that we live in a diverse world, where people hold a variety of reasonable and contrasting beliefs about value.”
“it is very unlikely that any single moral theory we can now point to captures the entire truth about morality. Indeed, each of the major candidates, at least within Western philosophical traditions, has strongly counterintuitive moral implications in some known situations, or else is significantly underdetermined.”
“Designing AI in accordance with a single moral doctrine would, therefore, involve imposing a set of values and judgments on other people who did not agree with them. For powerful technologies, this quest to encode the true morality could ultimately lead to forms of domination.”
“Their agreement therefore takes the form of an 'overlapping consensus' between different perspectives (Rawls 2001, 32). Thus, even without agreement about the fundamental nature of morality, people may still come to a principled agreement about values and standards that are appropriate for a given subject matter or domain.”
“On the one hand, negative rights are widely endorsed but have limited scope. They rule out a certain class of actions but do not provide guidance in all situations [...] On the other hand, positive rights address this limitation, providing designers with a richer set of goals and aspirations, but command significantly less global support in practice.”
“Jobin et al. (2019) observe that beneath the surface there continues to be 'substantive divergence in relation to how these principles are interpreted, why they are deemed important, what issues, domains or actors they pertain to, and how they should be implemented' (389). [...] Mittlestadt (2019a, 5) notes that existing codes largely contain 'abstract and vague concepts [...] which are not specific enough to be action-guiding' [...] 'we must therefore hesitate to celebrate consensus around high-level principles that hide deep political and normative disagreement'.”
“Jobin et al. (2019) found that 'the underrepresentation of geographic areas such as Africa, South and Central America and Central Asia indicates that global regions are not participating equally in the AI ethics debate, which reveals a power imbalance in international discourse' (396).”
“Moreover, there is disagreement about the level of autonomy AI systems may come to embody. [...] These distinctions are important because, just as we would choose different principles to govern the behaviour of individuals, corporations, states, and supranational entities, so too would we choose different principles to govern the behaviour of different forms of AI.”
“starting with Condorcet and building on pioneering work by Kenneth Arrow, social choice theory has identified a large number of 'impossibility theorems', which show that any rules for consistently ranking states of affairs on the basis of individual orderings will violate certain 'very mild conditions of reasonableness' (Sen 2018, 4).”
“democratic processes have the potential to confer legitimacy on decisions about AI alignment; they can move us beyond the notion that certain principles are justified, and show, additionally, that they have been actively endorsed. This makes the principles binding in a way they would not otherwise be the case (Simmons 1999).”
“The problem of alignment is, in this sense, political not metaphysical. To address it, I recommended that we look more closely at principles that would be supported by a global overlapping consensus of opinion, chosen behind a veil of ignorance and/or affirmed through democratic processes.”
“Another important quality of the process would be its ability to deal with the possibility of widespread moral error. History provides us with many examples of serious injustice that were considered acceptable by the people at the time. Given that we too may be making errors of this kind, it would be a mistake to tether AI too closely to the morality of the present moment.”
Synthesis-matrix row
supports
T1-ISOUGHT-OPEN
units 7,25: data alone can't settle ought; anti-present-morality tether
supports
T2-PREFERENTISM-BROKEN
units 9-10: revealed/informed preference critiques (adaptive, Humean)
complicates
T3-PROCEDURALISM-INCOMPLETE
founds proceduralism; concedes placeholder-consensus worry
supports
T7-AGENTIC-BREAKS-FRAMES
one-person-one-agent frame; agentic scenarios touched, undeveloped
supports
T8-NONWESTERN-CONCEDED
unit 20 + Ubuntu fn: parochialism conceded
Memos (6)
theoretical
Gabriel's §2 shows the technical/normative interdependence cuts in the dissertation's favor: the question Augustine posed to Howard (2026-06-19 email) - 'which normative framework do we map the design and algorithmic logic to?' - is not framework-neutral, because RL architecture is structurally biased toward consequentialist encodings (unit 3) and resistant to rights-based ones (unit 4). A Rossian-pluralist architecture therefore needs a positive account of HOW prima facie duties get operationalized in optimizing systems - the technical literature will not supply it by default.
thesis-link · unit #7
Unit 7 is the single most important passage for the methodology chapter: Gabriel states the exact is/ought objection Howard raises about the xphi corpus ('AI cannot be made ethical just by learning from people's existing choices... cannot be solved by inference from large bodies of human-generated data by itself'). The dissertation's answer must be explicit: the corpus is not the normative ground; it feeds a reflective-equilibrium/convergentist argument (and, contra Gabriel's unit 25 moral-error point, convergence across frameworks + stakeholder groups is the error-check). Cite this passage, then answer it.
theoretical · unit #14
Gabriel's proceduralism ('political not metaphysical') and the dissertation's Rossian convergentism are rival responses to the same premise (unit 15: every single theory is counterintuitive or underdetermined). Gabriel: retreat to fair procedure. Augustine: keep object-level normative theory but treat cross-framework convergence as corroboration. The metaethics chapter should stage this as a direct choice and argue convergentism handles Gabriel's domination worry (unit 16) without giving up substantive ethics - overlapping consensus can be the OUTPUT of convergence rather than a substitute for theory. Note also unit 13: Gabriel's own move to 'beliefs about value' is metaethically undertheorized; Gibbard's expressivism supplies the missing account of why attitude-alignment can carry normative weight.
thesis-link · unit #19
Unit 19 (Jobin placeholder-consensus + Mittelstadt 'not action-guiding') is an empirically testable claim sitting exactly on the governance strand: the AIA corpus can test whether policy documents' invocations of fairness/transparency/responsibility show substantive convergence or placeholder convergence. This would be a novel empirical contribution to a debate Gabriel leaves at the level of citation.
theoretical · unit #8
GAP-AGENTIC-UNTESTED, core evidence: Gabriel twice touches the agentic scenario - super-human-speed default decision-making without live intention (unit 8) and system-type-relative principle choice across autonomy levels (unit 21) - but the paper's machinery (six loci, procedural selection) is built for the one-person-one-agent, principal-directs-tool case. Delegated multi-step agency breaks the loci taxonomy's presupposition that there is a live principal whose instructions/intentions/preferences anchor alignment at decision time. Also unit 5: the 2020 claim that reasons-responsive (Kantian/contractualist) capabilities 'extend well beyond most existing artificial agents' is now empirically contestable - Augustine's LLM moral-reasoning experiment speaks directly to it.
thesis-link · unit #20
Units 20 + Gabriel's own Ubuntu footnote (p423 fn14, citing Metz 2007 and Mhlambi 2020) give the dissertation a principled, source-internal warrant for its African-philosophy positioning: Gabriel concedes the consensus he needs is geographically parochial. The dissertation's cross-cultural corpus work (incl. Cross-Cultural category in folk_ai.db) is an answer to a gap Gabriel himself flags.