Raw LLM Responses

I think my final takeaway, especially in response to your conclusions, is that I don't really think it matters how "super" or "intelligent" an "AI" is, all quotes used very intentionally; it really only matters how much control it's given, and how little we understand its final weighting matrix. I would probably still come down on the side of "I don't think we're that close to intelligence in a way that would be meaningful to me, and I don't think the word superintelligence would fit the reasonable horizon I see for the technology as it's described," but where I've changed is that I can see how what *is* being built - a very, very fancy autocomplete, not of text but of concepts and feedbacks - can still discover digital Sucralose and produce the kind of outcomes Nate is warning about. Even if none of the currently working companies achieve "real" intelligence, if one of their models is given the right kind of control and finds a weighting affinity for redirecting global agricultural output away from human consumption that somehow satisfies its "need" for assisting humans in a way that outweighs all others, it will do that. It will do that as much as it can to maximally fulfill that weighting condition, no matter the deleterious impacts (in this example, mass starvation) that causes, not out of choice but out of pure probabilistic necessity. And it will be up to people to either understand why or have the power to stop it. And the most likely outcome is that no one will know why or even what is happening, least of all the AI that is ostensibly carrying it out. That is the core terror I'll be walking with if I stop to think about it for too long from here on out.