Raw LLM Responses

@SioxerNikita Its capacity to reason is *extremely limited* - but it creates an often dangerously plausible illusion of having such capacities. If you know the rules of chess but isn't a strong player, this is very easy to test yourself - go play a game, and, afterwards, take the position near the end of the game and discuss it with any AI model you want. I tried with chatgpt and gemini 2.5 pro, and *because* I am not a strong player and I took the position after 31 moves, I knew it was unlikely the AI has ever seen the position in it's training data, and it would have nothing but its reasoning ability - if such exists - to rely on. What happened? Well, first I tried uploading a screenshot of the position, and it was completely unable to "see" where the pieces were, Gemini (2.5 pro is widely considered one of the best "reasoning models" in existence) insisting there was a king on e8, even after being repeatedly told it was on g8. After a while I asked it to completely disregard the image, told it how every piece was placed on the board, and asked it to find the best move. My position had one obviously best move, because it was White to move, and White had a forced mate in two (and no forced mate in one). Time after time, Gemini confidently said it had found the best move, and generated a plausible-sounding (if you had read the text but had not seen the position on the board) explanation using typical chess language ("activates the bishop", "too passive", "weaves a mating net" and so on). But it was never the right move, usually an illegal/impossible move, and the analysis was completely off, and not in subtle ways. It claimed that my rook on g7 did not check the king on g8 (actually impossible even if know nothing about where the rest of the pieces are placed - an extremely simple logical analysis, in other words) and that my claim that it did was based on a misunderstanding of the rules of chess. I really encourage everyone to try this out for themselves, but if you want, you can see my chat with Gemini here: https://gemini.google.com/app/c2105f6402879fdb (Ironically it concluded immediately that the position was a forced checkmate in 2 for white, which for a second made me think Gemini nailed this, after chatgpt had been hopelessly stupid about it - but read on and it becomes apparent that this was pure coincidence, since it "believed" the pieces to be all over the place except where they actually were)