Raw LLM Responses

5:13 <Krystal> "And he would keep asking it [for a diagnosis based on the exact same data, and the evaluations would change] You get a B [..] You get a D [..] You get an F" Yes: this is a core "design feature" of LLM / GPT-based chat tools. There two inherent problems: 1) if you are asking for summary statistics of raw data - e.g. trend analysis, first and second derivative, etc - you might achieve good-enough results. However, as soon as you step into unbounded "future probabilities" prediction rather than historic analysis, your risk of a poor response increases substantially. One way to reduce such problems might be to provide a verified set of known data profiles that result in a solid, expert-verified diagnosis that would act as known anchors or markers for your own analysis to be considered against. 2) all that said, you're essentially fighting against foundational design principles. If you attempt to eradicate response variation completely (exact repetition in responses based on a specific prompt and associated inputs), they essentially don't work (they don't produce responses humans find appealing). Although you can tune "Temperature" - which increases or decreases the variability, randomness, or "creativity" of responses, you can only really adjust this so far before the results at either end of the scale are poor. This parameter acts as a "weighting" mechanism on the probability distribution of the next predicted token (word or word part). Again, you can tune this a little bit).