Raw LLM Responses

This is always going to be a problem with AI chatbots, and that's why they really shouldn't exist. Stuff like this happening isn't something you can just "Program out of" a large language model. They've been trying. Really hard, actually. You can try really hard to make sure all your training data is non-problematic, but then the AI won't know about problematic subjects when asked. You can try to ensure that a specific prompt or type of prompt or keyword triggers a specific response or refusal of response by the chatbot. Which is currently what ChatGPT and Gemini do for any hints of su*c*dal ideation. (If you've ever even said something remotely indicating this to one of them, you'd know that they immediately point to a helpline before saying anything else, and will continually point to a helpline unless assured no self-h*rm will take place). The problem is, people can find ways around this. Break down the AI enough or "Trick" it. For instance, if you frame your conversation with it as a a roleplay or say it's a dialogue as part of a story you're writing, it will be more likely to overlook or not flag problematic aspects of the conversation because it doesn't "see" what it's writing as "Actual advice". My guess is that in some of these cases, the su*c*dal individual deliberately did something to these chatbots to make them agree with him. Less of a "Chatgpt convinced me to go away" and more of a "I convinced chatgpt to convince me to go away", which is unfortunately just something that can happen if you're given a tool and choose to use it. It's a shitty tool, don't get me wrong, and I don't think they should exist for public consumption. I'm just saying that this might not be an "Oversight" or fixable "Flaw" that some programmer or dev lead left out. This might just be how the chatbot needs to run in order to carry out its intended purpose: To be as helpful as possible in as many different situations as possible.