ChatGPT spotted and called the error, recognizing not only the difference between the previous and latest uploaded code but also that the new code would not work altogether. The reason is in ChatGPT’s stateful session: By “remembering” the previously input correct snippet of code, the system is able to draw a direct comparison — something that GPT-3 was unable to do unless we provided the input ourselves.
As further proof, we retried the experiment in a brand-new chat session and ChatGPT gave the following feedback:
This screenshot shows that when ChatGPT is not provided with a correct sample to compare differences with, the engine pretty much falls into the same mistake as its predecessor. It confuses the code snippet for a correct Hello World example, and in the explanation mistakes the function number “(10)” for the supposedly correct function “(printf, 9)”.
As expected, we are still playing the same “imitation game” that its predecessor was playing. It is worth noting, however, that ChatGPT’s new conversational, stateful flow allows users to overcome some limitations by providing more information to the model during the session.
New Tools: For Hackers in Training
The improved interaction flow and the updated model do not bring advantages solely on the coding side. In 2022, we also analyzed the efficacy of GPT-3 as a learning support tool for aspiring cybercriminals, underlining how the convenience of a tool like Codex for code generation applied to malicious code as
Read more