Raw LLM Responses

@thomasseptimius Doesn't seem like a fair representation of what I have read from Yudkowsky. Are you just refering to the video? In his writing, the *why* of it is rather clear: powerful optimisation processes tend to convert available resources into a form that is beneficial for the optimisation target. This is a pretty fundamental feature of "optimisation", and when applied to a general domain (like planet Earth) means that any aspect of the current world state that is not part of that optimisation target will be changed to either directly match the target or instrumentally contribute to it (e.g. by making the attainment or sustainment of the target more probable). *Almost all* optimisation targets are incompatible with human survival because of this, so creating a superhumanly competetent, general optimiser is *almost certainly* a cause for human extinction. You can argue that future AI systems will not be in the class of systems that are here refered to as *general optimisers*, but it sure seems like we are training current systems to optimise some kinds of scores (i.e. minimize prediction loss). Afaik, mechanistic interpretability has so far brought nothing to light that would counter this assessment. So in Yudkowsky's framework, unless you make sure that your AI is not in this category of general optimisers (nor will grow into this category over time), or unless you endow it with a *close to ideal* optimisation target, building a sufficiently powerful AGI probably kills everyone. It's not that the AI is spontaneously malicious - rather, it is just not the default for optimisers to care about you, and that seems like a pretty hard thing to change.