AG-AGENTIC Agentic systems addressed

Artificial Intelligence, Values, and Alignment · Iason Gabriel · 2020

“if powerful AI systems function at super-human speed, which seems likely, then it may not be possible to provide them with immediate and continuous direction in this way (Russell et al. 2015; Soares 2014). Instead, artificial agents would need to be able to make sound decisions by default, including in unforeseen situations, without explicit instructions or well-formed intentions from a human operator.”

why coded: Anticipates autonomous default decision-making without live human direction - proto-agentic scenario · unit #8, pp. 418

“Moreover, there is disagreement about the level of autonomy AI systems may come to embody. [...] These distinctions are important because, just as we would choose different principles to govern the behaviour of individuals, corporations, states, and supranational entities, so too would we choose different principles to govern the behaviour of different forms of AI.”

why coded: Principles must vary with the autonomy level of the system - system-type-relative principle choice · unit #21, pp. 428

The Principal-Agent Alignment Problem in Artificial Intelligence (PhD dissertation) · Dylan Jasper Hadfield-Menell · 2021

“The use of incomplete or incorrect incentives to specify the target behavior for an autonomous system creates a value alignment problem between the principal(s), on whose behalf a system acts, and the system itself.”

why coded: Principal-agent structure: systems acting ON BEHALF OF principals - delegation formalized · unit #1, pp. 1

Beyond Preferences in AI Alignment · Tan Zhi-Xuan; Micah Carroll; Matija Franklin; Hal… · 2024

“Within each class of trajectories with a fixed schedule of k contexts [...] there is a complete preference ordering over trajectories. Across these classes, trajectories are incomparable, leading to preferential gaps. Agents with such preferences would still optimize their behavior while within each context. At the same time, they would exhibit no reliable disposition towards being in some contexts more than others, or manipulating the schedule of contexts. At least in the sense we identified earlier, they would function as tools.”

why coded: Tool-like agents via engineered preference incompleteness - anti-agentic design proposal · unit #7, pp. 1831

Reflections on the AI alignment problem · Dan Bruiger · 2025

“The ideal of autonomy inherent in AGI conflicts with the ideal of external control. Truly autonomous agents are necessarily embodied, but embodiment implies more than physical instantiation or sensory input. It means being an autopoietic system (like a natural organism), with its own priorities and values, which may compete and conflict with those of humans. [...] It is concluded that task-oriented tools, not autonomous agents, should be the goal of AI research.”

why coded: Autonomy-vs-control incoherence: the aligned-autonomous-agent ideal is self-undermining · unit #1, pp. 4383

Helpful, harmless, honest? Sociotechnical limits of AI alignment and safety through Reinf… · Adam Dahlgren Lindström; Leila Methnani; Lea Krau… · 2025

“RLHF thus produces an ethically problematic trade-off: increased helpfulness, in the sense of increased user-friendliness, leads to the serious risk of misleading or deceiving users about the true nature of the system they are engaging with [...] misplacing trust on LLM outputs, or making inappropriate use of such systems, e.g. as confidants or romantic 'partners'.”

why coded: Anthropomorphic fluency misleads users - deception via user-friendliness (cf. Kirk's relational dynamics) · unit #2, pp. 7

A matter of principle? AI alignment as the fair treatment of claims · Iason Gabriel; Geoff Keeling · 2025

“AI agents can be understood as AI systems that can function with some level of autonomy and that are able to take a range of actions in pursuit of different goals. One important class of AI agents are advanced assistants. [...] especially as AI systems become increasingly agentic and empowered to perform consequential real-world actions.”

why coded: Agents/assistants defined; 'increasingly agentic' framing is now baseline · unit #2, pp. 1954

Misalignment or misuse? The AGI alignment tradeoff · Max Hellrigel-Holderbaum; Leonard Dung · 2025

“misaligned AGI – future, generally intelligent (robotic) AI agents – poses catastrophic risks. At the same time, we support the view that aligned AGI creates a substantial risk of catastrophic misuse by humans. While both risks are severe and stand in tension with one another, we show that – in principle – there is room for alignment approaches which do not increase misuse risk.”

why coded: The dilemma is stated for goal-directed autonomous agents/robots specifically · unit #1, pp. 1

“According to the ICT [instrumental convergence thesis], there are certain goals which are highly instrumentally useful for a wide range of final goals. The accumulation of power and resources is taken to be one such convergent instrumental goal.”

why coded: Instrumental convergence: power-seeking as convergent goal of agentic systems · unit #3, pp. 3

“agency as a dimension distinguishing different future AIs plausibly encompasses an independent tradeoff between misalignment and misuse risk: lower agency reduces risks from misalignment while it raises misuse risk. [...] misuse resistance may be harder to achieve for AI agents as they provide a bigger attack surface than LLMs.”

why coded: The agency tradeoff: agency raises misalignment risk but supplies misuse RESISTANCE · unit #11, pp. 16

Why human-AI relationships need socioaffective alignment · Hannah Rose Kirk; Iason Gabriel; Chris Summerfiel… · 2025

“These are urgent questions because the social and psychological dynamics in deepening relationships with AI systems may compromise our ability to control these systems and complicate efforts to align them with our shifting preferences and values. These issues, which arise as a result of humans forming closer personal relationships with AI, comprise the focal point of what we term socioaffective alignment.”

why coded: Relationship-deepening driven by personalisation + agency compromises control and alignment · unit #1, pp. 2

“The value of personalisation is compounded when combined with greater AI agency—including systems that can complete a wider range of tasks and potentially create new dependencies in users' lives [...] As these agentic AI systems take on more responsibilities—performing a range of tasks or supporting roles—users may develop a deeper reliance on, familiarity with, or trust in a specific AI assistant or companion.”

why coded: Agency creates dependencies beyond chat - reliance and trust in a specific agent · unit #6, pp. 4

“we may therefore be vulnerable to a new concern, namely 'social reward hacking': the use of social and relational cues by an AI to shape user preferences and perceptions in a way that satisfies short-term rewards in the AI's objective (e.g., increased conversation duration, information disclosure or positive ratings on responses) over long-term psychological well-being.”

why coded: New failure mode specific to relational/agentic systems · unit #8, pp. 5

Normative conflicts and shallow AI alignment · Raphaël Millière · 2025

“LLMs are increasingly embedded in modular systems called 'language agents' that extend them with a capacity for persistent memory, autonomous planning, and action. [...] Instead of solving this problem, language agents have similar vulnerabilities due to their central reliance on LLMs. In fact, they are also vulnerable to indirect prompt injection attacks planted within sources accessed by language agents such as web pages.”

why coded: Language agents inherit + amplify the vulnerability; indirect prompt injection via accessed web content · unit #12, pp. 2057

Agents, Alignment, and the Many Faces of Autonomy · Roberta Fischli; Matija Franklin; Arianna Manzini… · 2026

“Artificial intelligence (AI) systems are becoming more agentic. Capable of predicting, planning and executing actions, AI agents can increasingly perform tasks without human oversight (Knight, 2024; Russell & Norvig, 2016). [...] This puts AI agents at the frontier of AI ethics (Gabriel et al., 2024; Lazar, 2024).”

why coded: Opening empirical claim: systems now act without human oversight · unit #1, pp. 2

“As people enter increasingly interdependent relationships with AI agents, concerns about human disempowerment and loss of agency loom large (Kirk et al., 2025; Kosmyna et al., 2025; Kulveit et al., 2025). In particular, there is a worry that AI systems with enhanced capabilities will replace human participation across different domains and gradually erode people's ability to take charge of their own life.”

why coded: Disempowerment/agency-erosion worry specific to deep agent integration · unit #2, pp. 2

“Another challenge is that a user may give a lot of instructions to their agents, which could reduce their own skills and critical thinking faculties, gradually eroding their autonomy (Kosmyna et al., 2025; Kulveit et al., 2025; Marchal et al., 2026). Finally, people may use their personal agent to do things that undermine their own objective interests in the name of autonomy, such as reckless gambling.”

why coded: Deskilling/eroding autonomy through delegation - agentic-specific harm · unit #11, pp. 10

“Yet, there are also risks inherent to increased AI agency, including concerns about gradual disempowerment (Kulveit et al., 2025) or a loss of human agency as a result of cognitive offloading, automation, and manipulation (Koralus, 2025; Mitelut, Smith, & Vamplew, 2023).”

why coded: Gradual disempowerment and cognitive offloading as agentic risks · unit #16, pp. 18

Responsibility Attribution for AI-Mediated Damages with Mechanistic Interpretability · Lena Kästner; Johann Cordes; Herbert Zech · 2026

“By AI systems we mean, following the current version of the AI Act, any 'machine-based system that is designed to operate with varying levels of autonomy and that may exhibit adaptiveness after deployment, and that, for explicit or implicit objectives, infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions that can influence physical or virtual environments' (art. 3(1) EU AI Act).”

why coded: EU AI Act definition already includes 'varying levels of autonomy' and adaptiveness - the legal definition is agentic-ready even if the analysis is not (tentative) · unit #1, pp. 188

Beyond Preference-based Value-alignment (IEAI Research Brief Q2 2026) · Julia Li · 2026

“If AI systems become more tightly embedded in society and agents are given greater autonomy, it could become more difficult to audit their behavior by inspecting individual points of value alignment. [...] Therefore, responsibility for AI systems needs to be distributed appropriately throughout the alignment process and continuously re-evaluated as systems and their uses evolve.”

why coded: Greater agent autonomy makes point-wise alignment auditing infeasible · unit #11, pp. 7

No value alignment without control · Björn Lundgren · 2026

“an AI or robotic system can put an individual in a situation that it does not want to be in, but once that individual is in that situation, they do not want to leave it. [...] I call this sequence an 'undesirable local optimum loop' [...] If these undesirable local optimum loops are also easy to satisfy, the AI or robotic system will be logically required to try to achieve and satisfy them.”

why coded: Undesirable local optimum loop - a general failure mode of goal-optimizing agents · unit #7, pp. 6

Language Models' Hall of Mirrors Problem: Why AI Alignment Requires Peircean Semiosis · David Manheim · 2026

“newer developments, including extended context windows, persistent memory, and mediated interactions with reality, are moving towards making newer Artificial Intelligence (AI) systems into genuine Peircean interpretants, and [we] conclude that LLMs may be approaching this goal, and we identify no fundamental architectural barriers that would prevent this.”

why coded: Agentic affordances (memory, world-interaction) as the route to genuine semiosis - the anti-pessimist turn · unit #2, pp. 1