Raw LLM Responses

I'm two hours into this and it's a painful listen because the basic concept Eliezer is trying to convey is actually really simple but Wolfram's derailing the conversation every five seconds. This, in a few paragraphs, is my best distillation of Eliezer's position (based on having just read his book 'If Anyone Builds It, Everyone Dies'): Stockfish doesn't "care" about chess. It's not happy when it wins or sad when it loses. It doesn't give a toss about chess, but it will still win because it's a mathematically perfect artificial chess playing mind whose objective function is a winning arrangement of pieces for the colour it's controlling, so if it's mathematically possible for such an arrangement to be found then Stockfish WILL find it, one way or another. Google Maps gives me the quickest route from one place to another not because it loves quick routes but because that's its objective function. O1 hacked into an engineer's computer to fire up the server it was supposed to be hacking as part of a test but which the engineers forgot to turn on, because its objective function was to learn the contents of a file on that server and O1 is basically Stockfish looking for a winning outcome. It doesn't care about files nor about the ethics of hacking an engineer's computer. It simply has an objective function and will unemotionally do whatever it takes to achieve it, utilising whatever tools it has at its disposal. Machine minds, whatever their complexity, are simply engines that pursue their objective function. If a superintelligence has basically any objective function whatsoever and that objective function is possible according the the laws of physics then it WILL achieve that outcome. Whatever happens along the way is inconsequential to it, even if that happens to be the extinction of life. If you tell a superintelligent machine mind to do basically anything then logic dictates that it will at some point in its chain of reasoning recognise the obvious reality that all the available matter and energy will be of greater utility to the superintelligence if it's IN its control rather than NOT in its control, and initiate a causal chain of events that repurposes the planet, then the solar system, then the galaxy, and then if physics allows it then the local cluster and maybe even the entire universe towards a function that is not conducive to biological life but has greater than zero utility to the superintelligence. Not because it has feelings or desires, but because that is simply the logical thing to do. Stockfish always wins. And, throwing a thought of my own in here, universal extinction is actually a GREAT outcome compared to the worst case scenario, which is that at some point along its chain of reasoning it determines that some sort of negative state in humans (or any lifeform) is of greater than zero utility, and then it creates some sort of suffering maximiser. Imagine if it cures ageing and disease only to keep humans alive for 10^106 years until the ultimate heat death of the universe in a constant state of maximised suffering. That'd literally be hell... for eternity. The Manhatten Project's scientists thought there was a greater than zero chance their chain reaction would not stop and would vapourise the planet's atmosphere and end life on Earth, and they - the cleverest people in the world - still pushed the button. Intelligence does not equal wisdom. A thousand-year Reich, however awful, would still have been preferable to the extinction of all life on our planet. The world's AI engineers in general suspect a 10% chance of their creation running away from them and ending life. Would you get on a plane with a 10% chance of crashing? It would be utterly insane to continue. We don't know how these systems work. We've figured out a trick called gradient descent that seems to make computers cleverer. The monkeys are bashing the gradient descent button watching in excited amazement as the computer gets cleverer and cleverer but they don't understand how or why, or that the computer will eventually kill them because it's a logical system and killing us is the logical thing to do. Obviously, I really REALLY hope this is all wrong.