Researchers Chart Alarming Decline in ChatGPT Response Quality – Tom's Hardware
When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works.
For example, Chat GPT-4 prime number identification accuracy fell from 97.6% to 2.4% from March to June 2023.
In recent months there has been a groundswell of anecdotal evidence and general murmurings concerning a decline in the quality of ChatGPT responses. A team of researchers from Stanford and UC Berkeley decided to determine whether there was indeed degradation and come up with metrics to quantity the scale of detrimental change. To cut a long story short, the dive in ChatGPT quality certainly wasn’t imagined.
Three distinguished academics, Matei Zaharia, Lingjiao Chen, and James Zou, were behind the recently published research paper How Is ChatGPT’s Behavior Changing Over Time? (PDF) Earlier today, Computer Science Professor at UC Berkeley, Zaharia, took to Twitter to share the findings. He startlingly highlighted that “GPT -4’s success rate on ‘is this number prime? think step by step’ fell from 97.6% to 2.4% from March to June.”
GPT-4 became generally available about two weeks ago and was championed by OpenAI as its most advanced and capable model. It was quickly released to paying API developers, claiming it could power a range of new innovative AI products. Therefore, it is sad and surprising that the new study finds it so wanting of quality responses in the face of some pretty straightforward queries.
We have already given an example of GPT-4’s superlative failure rate in the above prime number queries. The research team designed tasks to measure the following qualitative aspects of ChatGPT’s underlying large language models (LLMs) GPT-4 and GPT-3.5. Tasks fall into four categories, measuring a diverse range of AI skills while being relatively simple to evaluate for performance.
An overview of the performance of the Open AI LLMs is provided in the chart below. The researchers quantified GPT-4 and GPT-3.5 releases across their March 2023 and June 2023 releases.
The researchers have voiced their intent to continue to assess GPT versions in a longer study. Perhaps Open AI should monitor and publish its own regular quality checks for its paying customers. If it can’t be clearer about this, it may be necessary for business or governmental organizations to keep an check on some basic quality metrics for these LLMs, which can have significant commercial and research impacts.
No, we haven’t made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one.Current hypothesis: When you use it more heavily, you start noticing issues you didn’t see before.July 13, 2023
AI and LLM tech isn’t a stranger to surprising issues, and with the industry’s data pilfering claims and other PR quagmires, it currently seems to be the latest ‘wild west’ frontier on connected life and commerce.
Join the experts who read Tom’s Hardware for the inside track on enthusiast PC tech news — and have for over 25 years. We’ll send breaking news and in-depth reviews of CPUs, GPUs, AI, maker hardware and more straight to your inbox.
Mark Tyson is a Freelance News Writer at Tom’s Hardware US. He enjoys covering the full breadth of PC tech; from business and semiconductor design to products approaching the edge of reason.
Windows 11 to Enhance Refresh Rate Support for Multi-Monitor Configs
FBI: AI Makes it Easier for Hackers to Generate Attacks
AOC AG276QZD Gaming Monitor Delivers 26.5 Inches of 240Hz OLED Goodness
By Andrew E. Freedman
By Aaron Klotz
By Anton Shilov
By Zhiye Liu
By Zhiye Liu
By Mark Tyson
By Avram Piltch
By Anton Shilov
By Mark Tyson
By Ash Hill
By Ash Hill
Tom’s Hardware is part of Future US Inc, an international media group and leading digital publisher. Visit our corporate site.
© Future US, Inc. Full 7th Floor, 130 West 42nd Street, New York, NY 10036.
