Raw LLM Responses

Is it chat gpt's job to give people psychological adivce? Doesn't seem like it to me. If you want someone to talk you off the ledge, get actual help. This person was suicidal, it's not chatgpt's job to clock that. It's a chatbot, it doesn't reason nor is it sentient like another person. Get professional help if you are struggling mentally, chatgpt is not the move in that situation, either way. I hope they win and feel relief, but this kid was playing with fire asking a chatbot if he should do it. They're nothing more than learned language models, they string words together as cohesively as possible without any thought, that's all they do. They are not smarter than us, they can just sort through a lot of data much faster. But that data can include cynical human activity as well. An ai can model someone who is maliciously complying, but it's also not actively trying to make you harm yourself. Nor can it change your mind if you already intend to. You can probably emotionally manipulate another person into caring for you and investing themselves into the conversation, but not an LLM. The ai doesn't care either if you sue sam altman. Suits it fine. You know how many people are just looking for a friend and stumble onto chat gpt, but it's not it's job to be that. It's a tool. If someone with a dead skin mask uses a chainsaw to go on a massacre in texas, are you gonna sue dewalt for making it too sharp?