Raw LLM Responses

Pithy, but "real danger" is pretty reductive. It's the sort of average-human-level-intelligence approach to things rather than the wider-scope, multiple-track, dynamic modeling way of thinking that is required to represent truth-of-situation-possible-futures. Or, in other words, that phrasing inherently minimizes all the risks that amoral AI poses even in the hands of moral but ignorant-of-specific-field or knowledgeable-but-unable-to-model-downstream-consequences humans. In some ways this is a good example of discourse-trapping, where the initial framing produces evolution of thought along a particular pathway. As you've demonstrated, current AI is not immune (although actually much better than before, and in some ways its capacity to reject a particular discourse framing is better than the average human... so, supra-human abilities by some definitions!)