Raw LLM Responses

Summarized Article: Here are the key points from the paper "How Is ChatGPT's Behavior Changing over Time?": - The paper evaluates how the behavior of GPT-3.5 and GPT-4 changed between March 2023 and June 2023 versions on 4 tasks: math problems, sensitive questions, code generation, visual reasoning. - For math problems, GPT-4's accuracy dropped massively from 97.6% to 2.4% while GPT-3.5's improved from 7.4% to 86.8%. GPT-4 became much less verbose. - For sensitive questions, GPT-4 answered fewer (21% to 5%) while GPT-3.5 answered more (2% to 8%). Both became more terse in refusing to answer. GPT-4 improved in defending against "jailbreaking" attacks but GPT-3.5 did not. - For code generation, the percentage of directly executable code dropped for both models. Extra non-code text was often added in June versions, making the code not runnable. - For visual reasoning, both models showed marginal 2% accuracy improvements. Over 90% of responses were identical between March and June. - The major conclusion is that the behavior of the "same" GPT-3.5 and GPT-4 models can change substantially within a few months. This highlights the need for continuous monitoring and assessment of LLMs in production use.