Raw LLM Responses

Yeah the researchers are just being dumb. One of those "First Elon spoke about electric cars, I knew nothing about electric cars, so I assumed he was a genius. Then he spoke about rockets, and I am not a rocket scientist, so I assumed he was a genius. But now he speaks about software development, and I am a software developer, and he's saying idiotic things. Now I question his cars and rockets" vibe. Paper basically says regarding code that GPT-4 is formatting the code, therefor its "non-executable code". But formatted code isn't "not executable", you just need to parse the formatting. It's better for copy-pasting, the standard use case of ChatGPT, but its an extra step if you try to interact with it through code cause now you have to parse it. They didn't update the tests to parse and instead threw their hands in the air and said "it added extra characters and now the code does not execute" Truly the dumbest thing I've heard a researcher say recently. When I prompt ChatGPT, I ALWAYS ask for it to format the code in a code block, cause copy-pasting normally the GPT-3 way was always a pain and I'd have to manually fix the formatting when I copied text. So if the researchers are that out of touch about prompting it with code, I have to question how they're handling the other tests