Raw LLM Responses

I agree that capitalism is the key problem. But I think the interest in AI goes beyond just the usual cash grabs. It's a source of global political power to those that control and distribute it as widely as possible. Think about those virtual agents you talk about. Imagine 100 million users with their own AI assistant/agents. If you control that AI you can decide its views and values and effectively influence 100 million users in a subtle and pervasive manner. Ask your AI for the day's news? Maybe your AI company doesn't like guns, and will always highlight gun violence news to you. Maybe they do like guns, so they instead show you lots of articles where someone with a gun stopped a criminal. Why does this work? The human mind suffers from many biases and logical fallacies. If you repeat information or scenarios frequently, they become part of the world view of people which then affects their future actions and thoughts without them noticing. The power these AIs could have over us can be absurd and very much like a comic book villain. It just depends on how many people you reach and how many safety measures are in place. A company providing AI to a billion people could even implement systemic genocide. Simply have the AI show people of the unliked race or religion information that makes having kids feel unrewarding and burdensome so they don't reproduce. I'm not saying such craziness will happen, only that it's plausibly foreseeable in a world where a handful of people control all information presented to a user. But like you said, now that capitalism has its hands on AI, there's absolutely no political will to shut any of that down in any way.