Raw LLM Responses

Given that the models predict the most likely next token based on the corpus (training text), and that each newer more up-to-date corpus includes more discussions with/about LLMs, this might not be as profound as it seems. For example, before GPT3 there were relatively few online discussions about the number of 'r's in strawberry. Since then there has obviously been alot more discussions about this, including the common mistake of 2 and correct answer of 3. Imagine a model that would have gotten the strawberry question wrong, but now with all of this talk in the corpus, the model can identify the frequent pattern and answer correctly. You can see how this model isn't necessarily "smarter" if it uses the exact same architecture, even though it might seem like some new ability has awakened. I suspect a similar thing might be playing a role here, with people discussing these testing scenarios.