Raw LLM Responses

If you gave a list of weights and activations to a reasercher the only way they could understand what the model does is run tests with it and check the output. We can predict what llms do. We can build llms. But we dont understand them. They are not whats called "explainble ai", its not like linear regrssion or arima or something that you could look at the weights and actuly understand the thinking process. There have been some minor advancment in tryibg to explain why nn models do what they do but these advamcments have also been showen to work in unpredictble ways..