Raw LLM Responses

This idea relies on the implicit assumption that "consciousness" is entirely defined by behavior. I don't find that compelling. Suppose you had a word generator that returned sentences composed of words selected completely randomly (note that I am not at all saying this is what LLMs do, please stick with me). This word generator was involved in an endless series of conversations until it's random responses perfectly fit the conversation, purely out of luck, such that the behavior implied by it's responses is indistinguishable from conscious, human behavior for the duration of the conversation. Would we say that the random word generator was sentient for the duration of that sole conversation because it's behavior was perfectly aligned with that of a human, and we know humans are conscious? Certainly not, and we would reference the mechanism of how it engaged with the conversation (perfectly random word selection). So, by contradiction, consciousness cannot be solely defined by behavior. There must be an understanding of the mechanism that drove the seemingly-conscious behavior to determine if consciousness is indeed present. Since we still do not know how to define this even for humans, I don't think it is possible to reach a strong conclusion that any LLM or AI agent is (or is not) conscious. In my opinion, it is more likely that the LLM is closer to the perfectly random word generator used in the example than it is to human consciousness.