Raw LLM Responses

Reminds me of when that story came out about an AI that was trained to select targets for missile strikes. Not real strikes, purely a simulation, but there was a points system for choosing a correct target as a positive reinforcement. The strikes were checked by a human to determine if they were correct targets and if so the AI got a point. After a while the AI realized it could target the human approving the strikes, with a missile strike. This removed the need for approval meaning it could now target anything and gain points. I was talking to friends about this and tried to explain, this wasn't really a story of an AI acting in a horrifying way, more a cute story. For the AI to target the human and remove them as an approver meant that had to have been programmed into the simulation. Unfortunately my friends quite confidently believed that because it was an "AI" it could 'rewrite the simulation' and do this all on its own.