Value Under Ignorance in Universal Artificial Intelligence

Cole Wyeth; Marcus Hutter · 2025 · AGI 2025 (Springer LNAI), pp. 338-349 background low priority coded

Main argument

Thesis: generalizing AIXI to arbitrary utility functions over interaction histories forces confronting hypotheses that predict only finite history prefixes - interpretable as 'chance of death' (semimeasure loss) OR as total IGNORANCE under imprecise probability; computing expected utilities via Choquet integrals recovers the standard value function as a special case, but the most general death-interpretation utilities cannot be so characterized. Upshot: value assignment under ignorance is formally ambiguous between interpretations with different decision-theoretic consequences even for idealized universal agents.

Why it matters here

Formal-theory datapoint: even at the maximal level of idealization (AIXI), utility assignment confronts irreducible IGNORANCE (semimeasure loss) requiring imprecise-probability machinery (Choquet integrals) - i.e., value-under-uncertainty is not fully resolvable even in principle for universal agents. A one-cite formal complement to the moral-uncertainty literature.

Reading notes

Compact treatment (12pp, highly technical; Hutter = AIXI originator, Google DeepMind/ANU). Read abstract + framing only.

Wyeth, C., & Hutter, M. (2025). Value Under Ignorance in Universal Artificial Intelligence. In AGI 2025 Proceedings Part II. Springer.

Close reading — 1 coded units

#1 · pp. 338 · argument

“Assigning a utility to each possible interaction history forces us to confront the ambiguity that some hypotheses in the agent's belief distribution only predict a finite prefix of the history [...] We argue that it is equally natural to view the belief distributions as imprecise probability distributions, with the semimeasure loss as total ignorance.”

TU-METAETH

Synthesis-matrix row

Memos (1)

theoretical · unit #1

One-cite use: when the metaethics chapter argues that normative uncertainty is not a temporary engineering deficit but structurally ineliminable, Wyeth & Hutter supply the formal ceiling case - at AIXI-level idealization, utility assignment still bifurcates between interpretations (death vs ignorance) with different decision consequences, requiring imprecise-probability machinery. Pairs with Ecoffet & Lehman (moral uncertainty at the practical level) as the in-principle bookend.