Raw LLM Responses

I've been a machine learning researcher for over a decade and imho the real risk from these algorithms isn't some terminator doomsday scenario that organizations like openAI are condescendingly crying but instead the risk comes from people trusting these things when they probably shouldn't. More than one place has done an experiment with algorithmic policing and with a naive 1000ft view of the subject it actually sounds like a good idea but practically speaking it is a horrible idea right now. The datasets are extremely biased and therefore any algorithm will also be extremely biased. There is a saying in the field that is something like "junk in and junk out" and that refers to the data used to train the models. Even if we were somehow able to get perfect data the next problem is that with large models even the most informed researchers on the planet basically have no clue how any particular decision, or more generally any output, from such a model was formed. Just as an example, we would have virtually no way of knowing if the reason a model said to increase police presence in an area was due to biased policing in that area previously or if there was actually some real pattern outside of bias that the model was picking up on. Maybe some day we'll have models that can give us a chain of reasoning behind the decision so we can say for sure "this is not racist" or whatever and that it actually found a real pattern that will help decrease crime but right now that isn't a thing that is possible with current tech