Raw LLM Responses

AI Risk expert here. It's unfortunately a bit worse than that. If you start with the assumption that a superintelligent AI will learn from data that does not contain any examples of AI being the enemy of humanity, that does not change the fact that it will still want* to accomplish whatever its goals are (which we did not get to robustly and precisely set in advance). Almost no matter what those goals are, when pursued with superhuman effectiveness, anything that is not directly implicit in those goals will be sacrificed for number-go-up on its favorite thing. This is known to be true for current, toy-model machine learning systems: Any parameter that is not specified to be within a specific range will be set to extreme values in order boost reward by a tiny fraction of a percent. And we don't have any realistic way to get human values into an AI system. When you look at instrumentally useful steps toward takeover, like deception, the same holds true: If we assume a superintelligence is trained without any examples of deception, it will still independently discover deception as a useful strategy. Even humans can do that. *"want" is anthropomorphic language that here means something like "contain a preference ordering in its action policy". Anything that behaves as if it has a goal comes under the sway of things like convergent instrumental goals (such as self-preservation and resource acquisition) and the principle of the orthogonality of intelligence and goals (there are no stupid end goals, just stupid ways to achieve them).