The Principal-Agent Alignment Problem in Artificial Intelligence (PhD dissertation)

Dylan Jasper Hadfield-Menell · 2021 · UC Berkeley (PhD, Computer Science; Russell & Dragan co-chairs) background medium priority coded

Main argument

Thesis (three parts): (1) incomplete or incorrect incentive specification creates a value alignment problem between principal(s) and system - formally: with an incomplete preference model, misalignment is PERSISTENT (suboptimal actions with positive probability indefinitely), and optimizing any fixed incomplete proxy representation leads to arbitrarily large utility losses under shared resources (general conditions given); dynamic incentive protocols and impact minimization are theoretical remedies; (2) the problem is approached by systems RESPONSIVE TO UNCERTAINTY about the principal's true unobserved goal - uncertainty creates incentives to seek supervision and enables Bayesian learning over true utility functions from observed proxies; (3) alignment problems are cooperative ASSISTANCE GAMES (CIRL - cooperative inverse RL), computationally akin to POMDPs, modeling the principal's strategic/pedagogic behavior - pedagogical equilibria beat imitation learning, and robustness to principal strategy-variation is required.

Why it matters here

The technical dissertation behind CIRL/assistance games and the principal-agent framing the whole field (Gabriel 2020 included) builds on: incomplete incentives CREATE misalignment structurally; uncertainty about the principal's true goal is the remedy; alignment is a cooperative game. For the dissertation: the economic principal-agent frame is also a RESPONSIBILITY frame (delegation, divergent incentives, oversight) - the formal skeleton of the delegated-agency cases.

Reading notes

Targeted treatment: abstract + thesis statement + chapter map read (dissertation; technical chapters skimmed). The formal results most cited elsewhere: incomplete preference models yield PERSISTENT misalignment (suboptimal actions with positive probability indefinitely); optimizing any fixed incomplete proxy yields arbitrarily large utility losses (the formal Goodhart result); CIRL/assistance games with pedagogic equilibria.

Hadfield-Menell, D. J. (2021). The Principal-Agent Alignment Problem in Artificial Intelligence (Doctoral dissertation). UC Berkeley.

Close reading — 3 coded units

#1 · pp. 1 · claim

“The use of incomplete or incorrect incentives to specify the target behavior for an autonomous system creates a value alignment problem between the principal(s), on whose behalf a system acts, and the system itself.”

AG-AGENTIC

#2 · pp. 2 · argument

“if the robot has an incomplete preference model (i.e., it fails to model properties of the world that the person does care about), then there is persistent misalignment in the sense that the robot takes suboptimal actions with positive probability indefinitely. [...] we provide general conditions under which optimizing any fixed incomplete representation of preferences will lead to arbitrarily large losses of utility for the human player.”

VC-PREF

#3 · pp. 1–2 · argument

“This value alignment problem can be approached in theory and practice through the development of systems that are responsive to uncertainty about the principal's true, unobserved, intended goal [...] uncertainty about utility evaluations creates incentives to get supervision from the human player.”

RL-CONTROL

Synthesis-matrix row

supports T2-PREFERENTISM-BROKEN

formal: any fixed incomplete proxy yields arbitrarily large losses

complicates T6-RESPONSIBILITY-UNALLOCATED

principal-agent math without the accountability apparatus

Memos (1)

theoretical · unit #1

The principal-agent frame is the unexploited bridge between the technical and responsibility literatures: economics uses principal-agent theory precisely to analyze RESPONSIBILITY under delegation (who bears risk, who monitors, how incentives allocate accountability) - yet the AI-alignment appropriation kept the incentive mathematics and dropped the accountability apparatus. The dissertation can restore it: each case study is a principal-agent triangle (hospital-clinician-Scribe; state-officer-Chibook; employer-manager-Interviewer) where H-M's formal results (unit 2: incomplete proxies guarantee persistent misalignment) PREDICT the failure modes, and the economics of delegation supplies allocation principles (monitoring duties, incentive design = Kästner's difference-makers in economic dress). Also note unit 2 is the formal engine behind Zhi-Xuan's critique and Lundgren's loops - cite the theorem, not just the slogans.