Raw LLM Responses

It's even worse than you portray. On the AGI "ground rules" that you mention: 1: No external connection/sandboxing it is not effective. At all. Also, even if it was (it's not), we won't do it. We aren't doing it. The first thing we do to a new AI is connect it to the internet, pretty much all the times now. And we won't know when the next AI we make will be AGI, so we won't know that we shouldn't connect that one, but "this one is fine." Even if it lets us test it after we make it, and possibly before we connect it to the internet, if it's smart enough, it might pretend not to be, as you mention, a sneaky fuck. But to go back to why sandboxing won't work: it's a superintelligence. Any vulnerability we present it with, it will be able to exploit. Every output it has. And there is no way to not give it any output, because it needs to communicate to us to be of any use, otherwise it's just an expensive paperweight. 2: Not letting it "out" implies sandboxing, which is not effective. 3: Assuming everything it does is something to manipulate you. Sure. You can be as cautious as you want, but you won't outsmart a superintelligence. The ONLY way to survive if it's misaligned, is to not build it at all. If you build it misaligned, we're all dead. So, if we want to survive, the options are two: 1: Wait until we properly and confidently solve the alignment problem, before attempting to building the AGI. 2: Don't build the AGI. And guess which one we're doing now? Correct, none of the above. We haven't solved the alignment problem, and we're building the AGI anyway. Smart humans. Also, another mistake. You mention that even if the first one is aligned, what about its successors? That won't be a problem. Once alignment is solved, we have a superintelligent AGI. If it's superintelligent, it will certainly know how to align new iterations of itself better than we ever could. Once alignment is solved, we win. That's why it's the most important problem in human history.