Raw LLM Responses

There's some important gaps in this presentation of the limits of AI. I work in the field, so I know this well. Humans providing WHAT and WHY is very good advice. LLM chatbots providing HOW is excellent advice. Chatbots doing the DO part, is increasingly an option, but hits up against the first big problem, RELIABILITY. 1. Chatbots are not reliable. Worse, they are CONFIDENTLY UNRELIABLE. This ties in with the second big problem. 2. Chatbots have no LOCAL CONTEXT. They have the knowledge of the entire planet, but no real idea of your local context, aside from what you give them in your LLM prompt. This is one reason why humans need to carefully consider carefully all of the WHAT and WHY. Retrieval Augmented Generation (RAG) provides some solutions for this, but each solution is very much a custom solution to the local problem. 3. Worse, chatbots CANNOT LEARN, because they are Pre-Trained and run live with a Frozen Model. You must provide everything it needs to know in its Context Window (aka the chat thread). 4. Chatbots FORGET, and they start forgetting at about 5000 words in a conversation thread. You cannot load up an entire textbook (or even just a textbook chapter), into the conversation and expect the chatbot to capably apply the information therein. The chatbot will default back to its Pre-Trained knowledge, forgetting chunks of the information you just gave it. This can be improved with careful Prompt Engineering, guiding and reminding the chatbot which pieces of the information you just uploaded to it are relevant. This is called the Attention Mechanism, and attention problems leads to "context rot" in long conversations. *Chatbots need humans* to provide local context, to fact check, to sanity check, to understand their limitations, because they certainly don't understand their own limitations. Are they getting better? Absolutely, but none of these limitations are really going away.