TU-METAETH Metaethics chapter

Artificial Intelligence, Values, and Alignment · Iason Gabriel · 2020

“How are we to decide which principles or objectives to encode in AI—and who has the right to make these decisions—given that we live in a pluralistic world that is full of competing conceptions of value? Is there a way to think about AI value alignment that avoids a situation in which some people simply impose their views on others?”

why coded: Opening framing for the metaethics chapter's problem statement · unit #1, pp. 411

“In general, it seems likely that it will be easier to align AI with moral theories that have the same fundamental structure based on maximizing reward over time in the face of uncertainty, than with other alternatives. Consequentialist moral theories, the most famous of which is act utilitarianism, fit the bill.”

why coded: Directly answers 'which framework do we map algorithmic logic to' - the mapping is not architecture-neutral · unit #3, pp. 413

“The metaethical debate may seem critical. After all, if values do not have this objective basis, how can AI be developed to align with them? Yet these concerns turn out to have limited significance for the question at hand. To see why, we need to acknowledge first that, in practice, AI would have to be aligned with some set of beliefs about value, not with value itself.”

why coded: Claims metaethical debate has limited practical significance - alignment targets beliefs about value · unit #13, pp. 422

“the task in front of us is not, as we might first think, to identify the true or correct moral theory and then implement it in machines. Rather, it is to find a way of selecting appropriate principles that is compatible with the fact that we live in a diverse world, where people hold a variety of reasonable and contrasting beliefs about value.”

why coded: The rival thesis the dissertation's convergentism must engage · unit #14, pp. 424

Reinforcement Learning Under Moral Uncertainty · Adrien Ecoffet; Joel Lehman · 2021

“proposals in moral philosophy typically do not explicitly consider the sequential nature of decision making. [...] in RL, an agent cannot directly bring about possible worlds but rather takes (often very granular) actions, which have long term effects both in the consequences that they bring about and in how they shape the ethically-charged decision situations an agent may encounter in the future.”

why coded: Sequential decision-making as philosophy's blind spot - actions shape future moral situations · unit #2, pp. 2

“a final ambitious direction for future work is to explore mechanisms through which an agent can itself update its credences in moral theories (or derive new ones). That is, what might provide a principled foundation for machine meta-ethics?”

why coded: 'Machine meta-ethics': agents revising credences - the open frontier · unit #9, pp. 9

Moral dilemmas for moral machines · Travis LaCroix · 2022

“to say that we want an autonomous system to minimise the unethical outcomes under these circumstances presupposes that we already know what the unethical outcomes to be minimised are—i.e., that we have already sorted out the relevant metaethical questions.”

why coded: Benchmarking presupposes solved metaethics - the cart-before-horse objection · unit #4, pp. 743

Disagreement, AI alignment, and bargaining · Harry R. Lloyd · 2024

“MEC and MSEC are only applicable in cases where differences between the choiceworthinesses of the options available according to every moral theory in which the agent or group has credence can be measured on some common scale of value. Intertheoretic expected choiceworthiness is simply undefined in cases where unit comparisons are impossible. [...] imagine trying to compare absolutist deontology against scalar utilitarianism. These two different moral theories don't even use the same deontic categories.”

why coded: Intertheoretic unit comparison problem - the formal obstacle ANY cross-framework method must answer, incl. convergentism · unit #8, pp. 1767

Beyond Preferences in AI Alignment · Tan Zhi-Xuan; Micah Carroll; Matija Franklin; Hal… · 2024

“an arguably deeper problem with EUT is that it fails to ground the normativity of our preferences. EUT is a theory of instrumental rationality not value rationality: It tells us how to choose our actions in order to satisfy our preferences, and imposes constraints on what those preferences can be, but it does not say anything further about where those preferences can or should come from.”

why coded: The Weberian instrumental/value rationality distinction as the wedge against preferentism · unit #8, pp. 1831

Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value · Joe Edelman; Tan Zhi-Xuan; Ryan Lowe; Oliver Klin… · 2025

“TMV operationalizes these philosophical insights without directly advocating for the primacy of specific values, such as fairness or efficiency. [...] we now need a coordinated research program that transforms these early proofs of concept into robust, scalable approaches across the full stack of alignment.”

why coded: Claims to operationalize thick-value philosophy while staying neutral on first-order values · unit #4, pp. 5

Normative conflicts and shallow AI alignment · Raphaël Millière · 2025

“[fn1] Here, I deliberately frame the problem in strictly behavioral terms, to avoid taking a stance of what it would mean for a given AI system to have moral values. In particular, one might hold that having moral values requires various psychological capacities – including beliefs, desires, intentions, agency, or self-awareness – that are plausibly missing from current AI systems such as large language models.”

why coded: Deliberate bracketing of moral-values attribution - contrast with Augustine's substantive engagement · unit #2, pp. 2036

“it is useful to distinguish between prima facie and all-things-considered oughts (Ross, 1930; Hurley, 1989). Prima facie oughts are moral obligations that carry some weight or create a presumptive duty, but can be overridden by other, stronger moral considerations in a given situation. [...] Ross (1930) illustrates this distinction with the example of a conflict between keeping a promise and averting a serious accident. While there may be a prima facie duty to keep the promise, it can be overridden by the stronger prima facie duty to prevent harm, resulting in an all-things-considered duty to avert the accident.”

why coded: Direct evidence Rossian pluralism is doing live work in cutting-edge alignment theory · unit #6, pp. 2049

Kantian deontology for AI: alignment without moral agency · Oluwaseun Damilola Sanwoolu · 2025

“the lack of consensus on the correct ethical theory is not sufficient grounds to bypass ethical theories in this discussion. Concerning human morality, for example, one could adopt pluralist ethical approaches. Although ethical theories all have strengths and limitations, yet we continue to seek moral guidance despite the lack of agreement on a single 'correct' theory.”

why coded: Direct precedent: living with theoretical pluralism without abandoning first-order theory · unit #3, pp. 5426

“AI systems, however, lack both aspects [of freedom]. They are constrained by pre-programmed architectures, optimization objectives, and statistical learning from data. Consequently, they do not exhibit negative freedom, as they cannot truly deviate from causal determination, nor do they exhibit positive freedom, as they cannot will or legislate moral law from reason. [...] I argue that we refrain from ascribing moral agency to AI, particularly within the Kantian ethical framework.”

why coded: The explicit refusal to ascribe moral agency - Augustine's own conclusion, independently reached · unit #5, pp. 5428

Moral disagreement and the limits of AI value alignment: a dual challenge of epistemic ju… · Nick Schuster; Daniel Kilov · 2025

“There are two kinds of reasons people can accept such outputs: moral-epistemic reasons and political reasons. Epistemically, if we have good reason to think that the judgments and decisions of an AI system are likely to be morally correct, then we have good reason to accept them. [...] Alternatively, if we have good reason to think that the outputs of an AI system are democratically legitimate, then we have good political reason to accept them.”

why coded: The two-reasons framework (epistemic deference vs political legitimacy) - the space the convergentist third option must enter · unit #6, pp. 6075

Value Under Ignorance in Universal Artificial Intelligence · Cole Wyeth; Marcus Hutter · 2025

“Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the agent's belief distribution only predict a finite prefix of the history [...] We argue that it is equally natural to view the belief distributions as imprecise probability distributions, with the semimeasure loss as total ignorance.”

why coded: Even idealized universal agents face formally irreducible value-under-ignorance ambiguity · unit #1, pp. 338

Disentangling AI Alignment: A Structured Taxonomy Beyond Safety and Ethics · Kevin Baum · 2026

“[The proceduralist practical syllogism:] (P1) Moral pluralism (at least when understood descriptively) is a fact. (P2) There is deep moral uncertainty and persistent moral disagreement. (P3) Imposing values should be avoided. (P4) If P1-P3 and we must nevertheless align AIAs, we should aim for alignment relative to publicly justifiable norms rather than moral alignment. (P5) We must align AIAs. (P6) Publicly justifiable norms can (only? best?) be achieved through a fair and public process [...] (C) We should aim for alignment relative to norms that are the result of such a fair and public process.”

why coded: P4 is the contestable step: pluralism+uncertainty+anti-imposition do NOT force abandoning moral alignment if convergence supplies non-imposed moral warrant · unit #4, pp. 166

Wide reflective equilibrium in LLM alignment: bridging moral epistemology and AI safety · Matthew Brophy · 2026

“Critics argue that MWRE can lead to different, equally coherent moral belief sets, which would seem to risk relativism. [...] Daniels acknowledges the possibility of multiple equilibria but argues this doesn't necessitate a degeneration into relativism. Equilibria can be compared to competing scientific theories: employing criteria like explanatory power, scope, empirical fit, prediction, and parsimony.”

why coded: Multiple equilibria compared by theoretic virtues - relativism blocked without foundationalism · unit #6, pp. 5

Agents, Alignment, and the Many Faces of Autonomy · Roberta Fischli; Matija Franklin; Arianna Manzini… · 2026

“Implicit to this discussion is a subtle but important insight: even a single value like autonomy is not a straightforward alignment target; it is a multifaceted concept we need to elaborate on before operationalizing it. [...] value alignment needs to navigate trade-offs not just across, but also within, individual values.”

why coded: Within-value pluralism is a metaethical datum the dissertation's Rossian machinery can exploit · unit #9, pp. 8

Normative Ethics, Artificial Intelligence, and Value Alignment (Dynamic Normativity) · Nicholas Kluge Corrêa · 2026

“[Dynamic Normativity: the final three chapters] offer a minimal set of strategies and methodologies for tackling the value alignment problem, arguing that if a particular set of requirements are met, we can say that a minimal level of alignment has been achieved.”

why coded: Minimal-alignment sufficiency conditions - a rival normative account to engage · unit #1, pp. 1

“In Dynamic Normativity, we must satisfy three distinct requirements for a sufficient alignment condition. [...] (1) coherently aggregate human preferences, (2) learn from them, and (3) mitigate unwanted behavior. [...] Aligned AI systems should coherently aggregate human preferences in a way that resolves cases of uncertainty.”

why coded: The three-requirement sufficiency condition - the rival account's core · unit #10, pp. 85

Beyond Preference-based Value-alignment (IEAI Research Brief Q2 2026) · Julia Li · 2026

“The positioning of preferences as normative, or how things should be, when they are descriptive of how things are, within value alignment techniques, can be a source of technical and ethical issues. Alignment research should not continue advancing down a path that is philosophically naive about the concepts it is trying to operationalize.”

why coded: 'Alignment is, at its core, a normative project' - the brief ends where the dissertation begins · unit #13, pp. 7

No value alignment without control · Björn Lundgren · 2026

“value alignment does not necessarily have the same aims as normative ethics (where one tries to find the best ethical theories). That is, the goal of value alignment is not best understood as aiming to say which ethical theory is correct and how we implement it (since the latter may turn out to be impossible), but as a question of which principles best codify a logical praxis that ensures an ethically appropriate outcome while avoiding ethically bad or catastrophic outcomes.”

why coded: The alignment/normative-ethics gap the dissertation must position within · unit #4, pp. 3

“One possible interpretation is that the problem for the consequentialist theories considered so far is that they are monistic about values. The reasoning would be that having a singular intrinsic good makes a normative theory more sensitive to interpretations of that singular value [...] which then easily results in a simple-minded goal-satisfaction that is suboptimal.”

why coded: Monism-is-brittle argument = positive case for the dissertation's Rossian pluralism · unit #10, pp. 8

Moral Responsibility Without Moral Agency (Palgrave Handbook Ch 18) · Nikhil Mahant · 2026

“instead of a metaphysical, it takes a distinctive conceptual approach. While other approaches focus on whether morally responsible entities must (metaphysically) possess the features that they are assumed to possess (e.g., consciousness), I have made a case against Agential View on the ground that it would be the most optimal way to theorize about moral responsibility.”

why coded: Conceptual-methodological route distinguished from metaphysical routes · unit #3, pp. 283

Understanding the Process of Human-AI Value Alignment · Jack McKinlay; Marina De Vos; Janina A. Hoffmann;… · 2026

“As Sutrop cautions, AI developers risk assuming that a normative solution will naturally follow from sufficiently technically capable AI, even though we as a society still remain undecided about our value priorities.”

why coded: Sutrop: no normative solution follows from technical capability - society undecided on value priorities · unit #3, pp. 11

“in choosing (implicitly or explicitly) an ethical theory and the goals and values for a system to align to, a value-laden decision is itself being made about which values and goals are worth empowering through artificial intelligence systems. [...] these decisions must be made carefully if we want to avoid marginalising schools of ethical thoughts or particular values, particularly given the dominance some cultures currently enjoy in AI development.”

why coded: Theory-choice is itself value-laden; risks marginalising schools of thought given cultural dominance · unit #7, pp. 13

Justifications for Democratizing AI Alignment and Their Prospects · Andre Steingrüber; Kevin Baum · 2026

“The empirical fact of normative (and metanormative) disagreement makes us normatively (and metanormatively) uncertain [...] This uncertainty eliminates what would be a straightforward justification for any potentially illegitimate state of affairs. If we were normatively (and metanormatively) certain, we could simply show that the normative constraints we are implementing are (objectively) correct.”

why coded: The justificatory-gap argument: uncertainty defeats theoretical justification, opening space for political justification · unit #3, pp. 149

“We are metanormatively uncertain in at least three respects: We are uncertain whether there is a normative ground truth [...] whether this normative ground truth is unique [...] And we are uncertain whether and how we can have knowledge about this normative ground truth.”

why coded: Three-fold metanormative uncertainty (existence, uniqueness, knowability of ground truth) - precise metaethical taxonomy · unit #4, pp. 150

Re-aligning Value Alignment: A Metaethical Perspective on AI Ethics (Palgrave Handbook Ch… · Mehmet B. Unver · 2026

“efforts to align AI with human ethics are often constrained by what can be characterised as 'bounded moral contexts' within AI design [...] the prevailing emphasis on applied ethics, particularly within market-driven regulatory frameworks, leaves a critical gap in achieving genuinely holistic and robust value alignment.”

why coded: The bounded-moral-contexts diagnosis of applied-ethics alignment · unit #1, pp. 1

“Metaethics, by contrast, interrogates the very nature of morality, the meaning of human values and the conditions under which moral claims can be justified. Without this foundational reflection, AI systems risk implementing ethical prescriptions superficially, reflecting human input without true understanding or capacity for nuanced moral reasoning.”

why coded: Metaethics as the missing layer - the handbook's own framing premise · unit #2, pp. 2