Raw LLM Responses

I think far too much of the decisions on AI are being considered as if we have an actual understanding of how things work and that we have a much stronger influence on how they behave _in human terms_. Hallucination is an example of this. I think the use of the term misleads people into thinking of it as an event, when I think the reality is that what we're training AI to do is to choose tokens as if it understands things, and rewarding it based on how much it _appears_ to understand, and I think the clearest demonstration of reasoning, is coming to an accurate conclusion when you don't have all the info in your training. I think we're essentially saying "Make something up that sounds convincing." And the better known something is, and the more data on the subject in the training, The easier it is to make it convincing, using facts, and the more likely it will give a "good" answer. The less info it has on the subject, the more likely it is to be incorrect. _But_ the more believable the lie, the more likely we'll reward it as a "good" answer. Additionally, even in our perceptions of what makes an AI "better" is when it can come to the right conclusions with relatively limited information. It's not about looking up exact things in wikipedia. Search engines have been able to do that for years. What amazes us is when you can ask something with limited information, and still get the right answer when you wouldn't expect it. Unfortunately we're most amazed by the biggest guesses, "hallucinations", which happen to be true, so in our own brains we give it trust points more when it's the most reckless, and this will likely lead us to believe it when it gives us other answers that are similarly "hallucinated" when we don't have the knowledge to confirm its accuracy. I've seen this happen in real time, when playing board games with some friends. Someone asked chatgpt about some technicalities of some rules in monopoly, which they thought were fairly obscure, and it answered "correctly", so in very human terms, they trusted it as "reliable". Then later they asked about a rule that we were disagreeing about in another game we were playing, and it gave a convincing answer (though it was not anything we could confirm from the rulebook, or any official errata from the game designer), so everyone went along with it, probably seeing it as even more trustworthy, regardless of the fact that we couldn't confirm its answer. Then, even later, playing a different game, two players were disagreeing on their interpretations of another rule, so asked chatgpt, and were ready to accept its judgment, and continue playing, except that I had _just_ seen that exact rule pointed out specifically in the manual when I was thumbing through it a minute earlier, and chatgpt was simply giving a "believable" answer, even though it was totally incorrect. I think there's a broken incentive structure in the training, (and meta-training, from how we respond to it), where reckless answers that happen to be correct are valued more than obviously correct ones, so we end up trusting it more on things that are impossible to confirm and it's less likely to be factually correct about.