Main argument
Thesis: among alignment targets designed to respect stakeholder disagreement, voting-theoretic 'moral parliaments' and decision-theoretic 'maximise socially expected choiceworthiness' (MSEC) both fail to select attractive compromise options, whereas a bargaining-theoretic target (Nash bargaining solution over simulated stakeholder bargaining, or an ordinal variant) succeeds. Argument type: formal-philosophical comparison via counterexamples. Key cases: Jackson (51/49 split - majority voting picks A though 49% think it terrible and everyone thinks B nearly best: tyranny of the majority); Biorisk (majority rule makes safety-compute allocation discontinuous in stakeholder opinion: hypersensitivity); fanaticism (MSEC lets a 0.01%-credence theory with huge choiceworthiness stakes dominate); intertheoretic unit comparisons (expected choiceworthiness undefined across theories with different deontic categories, e.g. absolutist deontology vs scalar utilitarianism). Bargaining avoids these: NBS's Nash-product favors equal gains and compromise options; simulated bargaining with latent-class stakeholder models is computationally feasible; rejects 'simulated folk bargaining' (predicting actual human bargaining) for interpretability and spite/heuristics reasons, preferring a normative solution concept. Leaves open who counts as a stakeholder.
Why it matters here
Completes the Phil Studies cluster's map of formal responses to normative disagreement: where Gabriel/G&K go deliberative and Zhi-Xuan goes contractualist, Lloyd formalizes the disagreement-handling problem and compares voting (parliaments), decision theory (MSEC), and bargaining (Nash) - endorsing bargaining. Supplies the technical social-choice machinery the dissertation's stakeholder-convergence design implicitly relies on, and its objections (tyranny of majority, hypersensitivity, fanaticism, intertheoretic comparison) are tests any convergentist proposal must pass.
Reading notes
Close read of secs 2-5.3, 5.4, conclusion (31pp; formal appendix skimmed). Yale + Center for AI Safety. Note: has a companion piece (Lloyd 2024, on MSEC) not in library. The intertheoretic-comparison objection to MSEC is highly relevant to any cross-framework convergence method - flag for the methodology chapter.
Lloyd, H. R. (2024). Disagreement, AI alignment, and bargaining. Philosophical Studies, 182, 1757-1787. https://doi.org/10.1007/s11098-024-02224-5
Close reading — 12 coded units
#1
· pp. 1760
· argument
“another arguably important requirement for fairness here is to ensure that these disadvantaged groups have a fair say in determining the criteria under which the effects of AI systems should be judged as 'fair' or 'unfair.' [...] fair AIs should optimise for objectives that at least partially reflect the values of all of the communities who those AIs will affect.”
#2
· pp. 1760
· evidence
“AI ethicists have proposed numerous prima facie plausible 'fairness criteria' for AI systems. Unfortunately, several impossibility theorems have recently demonstrated that no single AI system can jointly satisfy all of these criteria (Chouldechova, 2017; Corbett-Davies et al., 2017; Kleinberg et al., 2016 [...]). In the face of these impossibility theorems, it is inevitable that there will be social disagreement about what is required for fairness in AI systems.”
#3
· pp. 1761
· argument
“The project of aligning AIs with human values is arguably more likely to succeed if it can command a broad base of support. But the alignment project is unlikely to command a broad base of support if its intended alignment target only reflects the values of a certain subset of society. [...] It is just bad politics for AI safety proponents to advocate alignment with potentially controversial conceptions of desirable AI behaviour such as total welfare maximisation.”
#4
· pp. 1762
· evidence
“Jackson: some moral parliament (unsimulated or simulated) faces a choice between three options, A, B, and C. 51% of parliamentarians think that A is the best, B is almost as good, but C is terrible. 49% of parliamentarians think that C is the best, B is almost as good, but A is terrible. [...] a plurality- or majority-rule voting system will select option A. Yet many of us intuit, to the contrary, that it would be better for our alignment approach to select option B [...] plurality- and majority-rule voting can fail to select an attractive compromise option.”
#5
· pp. 1763
· evidence
“[Biorisk:] under plurality- or majority-rule voting, how much compute the biomedical AI should spend on safety testing depends discontinuously upon how many stakeholders endorse the less as opposed to the more permissive view [...] the parliament's verdict is extremely sensitive to small differences in stakeholder opinion, such as the difference between 49 and 51% endorsement. Moreover, this hypersensitivity strikes me as entirely unnecessary.”
#6
· pp. 1765–1766
· definition
“According to MSEC, some AI ought to be aligned with some alignment target X iff aligning that AI with X would maximise expected choiceworthiness according to the group agent that represents the AI's stakeholders.”
#7
· pp. 1767
· argument
“[Fanaticism:] imagine that the stakeholder group has 99.9% credence in the moral theory M1, and 0.01% credence in the moral theory M2. [...] MSEC implies that H is preferrable to G. Yet many of us intuit, to the contrary, that it would be better for our AI to select option G [...] it would be reckless and uncompromising for an AI to prefer H over G.”
#8
· pp. 1767
· argument
“MEC and MSEC are only applicable in cases where differences between the choiceworthinesses of the options available according to every moral theory in which the agent or group has credence can be measured on some common scale of value. Intertheoretic expected choiceworthiness is simply undefined in cases where unit comparisons are impossible. [...] imagine trying to compare absolutist deontology against scalar utilitarianism. These two different moral theories don't even use the same deontic categories.”
#9
· pp. 1777
· argument
“[Against simulated folk bargaining:] unless and until significant progress is made on the problem of 'interpretability,' it will be difficult to understand why the AI bargaining simulator selects particular outcomes. [...] real-world cases of bargaining over morally contentious issues might have undesirable features that we should not wish to emulate [...] a subgroup of stakeholders who strongly dislike some of the other stakeholders might be motivated by spite or schadenfreude.”
#10
· pp. 1778
· definition
“[Nash bargaining solution axioms:] 1. Scale invariance [...] 2. Pareto optimality [...] 3. Symmetry [...] 4. Independence of irrelevant alternatives. The NBS uniquely satisfies all four of these axioms. [...] One attractive feature of the NBS is that (all else being equal) it favours equal division of gains from trade between the bargainers.”
#11
· pp. 1780
· argument
“it should be sufficient for our AI to learn the preferences of some representative subsample of the stakeholder population. Second, our AI is also likely to be able to group the preference orderings of the stakeholders within this subsample into a manageable number of latent classes. The AI can then compute the solution for bargaining between the representative members of these latent classes.”
#12
· pp. 1780
· gap
“One important question that I have not discussed in this paper is that of who should count as a 'stakeholder' for any particular AI system [...] This is a particularly important topic for future alignment research, since it has the potential to significantly affect the verdicts of all three of the voting-, decision-, and bargaining-theoretic approaches to alignment.”