Why GPT-4 Is a Major Flop – Techopedia
ICT (Information and Communications Technology) is the use of computing and telecommunication technologies, systems and tools to facilitate the way information is created, collected, processed, transmitted and stored. It includes computing technologies like servers, computers, software applications and database management systems (DBMSs)… View Full Term
Don’t miss an insight. Subscribe to Techopedia for free.
Table of Contents
Since January 2017 Tim Keary has been a freelance technology writer and reporter covering enterprise technology and cybersecurity.
Stanford University’s study showed that GPT-4 and GPT-3.5’s aren’t necessarily getting better over time, which means LLM’s are nowhere near capabilities of true artificial general intelligence.
GPT-4 made big waves upon its release in March 2023, but finally, the cracks in the surface are beginning to show. Not only did ChatGPT’s traffic drop by 9.7% in June, but a study published by Stanford University in July found that GPT-3.5 and GPT-4’s performance on numerous tasks has gotten “substantially worse over time.”
In one notable example, when asked whether 17,077 was a prime number in March 2023, GPT-4 correctly answered with 97.6% accuracy, but this figure dropped to 2.4% in June. This was just one area of many where the capabilities of GPT-3.5 and GPT-4 declined over time.
James Zou, assistant professor at Stanford University, told Techopedia:
“Our research shows that LLM drift is a major challenge in stable integration and deployment of LLMs in practice. Drift, or changes in LLM’s behaviors, such as changes in its formatting or changes in its reasoning, can break downstream pipelines.”
“This highlights the importance of continuous monitoring of ChatGPT’s behavior, which we are working on,” Zou added.
Table of Contents
Stanford’s study, “How is ChatGPT’s behavior changing over time,” looked to examine the performance of GPT-3.5 and GPT-4 across four key areas in March 2023 and June 2023.
A summary of each of these areas is listed below:
Although many have argued that GPT-4 has got “lazier” and “dumber,” with respect to ChatGPT, Zou believes “it’s hard to say that ChatGPT is uniformly getting worse, but it’s certainly not always improving in all areas.”
The reasons behind this lack of improvement, or decline in performance in some key areas, is hard to explain because its black box development approach means there is no transparency into how the organization is updating or fine-tuning its models behind the scenes.
However, Peter Welinder, OpenAI’s VP of Product, has argued against critics who’ve suggested that GPT-4 is on the decline but suggests that users are just becoming more aware of its limitations.
“No, we haven’t made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one. Current hypothesis: When you use it more heavily, you start noticing issues you didn’t see before,” Welinder said in a Twitter post.
No, we haven't made GPT-4 dumber. Quite the opposite: we make each new version smarter than the previous one.
Current hypothesis: When you use it more heavily, you start noticing issues you didn't see before.
— Peter Welinder (@npew) July 13, 2023
While increasing user awareness doesn’t completely explain the decline in GPT-4’s ability to solve math problems and generate code, Welinder’s comments do highlight that as user adoption increases, users and organizations will gradually develop greater awareness of the limitations posed by the technology.
Although there are many potential LLM use cases that can provide real value to organizations, the limitations of this technology are becoming more clear in a number of key areas.
For instance, another research paper, developed by Tencent AI lab researchers Wenxiang Jiao and Wenxuan Wang, found that the tool might not be as good at translating languages as is often suggested.
The report noted that while ChatGPT was competitive with commercial translation products like Google Translate in translating European languages, it “lags behind significantly” when translating low-resource or distant languages.
At the same time, many security researchers are critical of the capabilities of LLMs within cybersecurity workflows, with 64.2% of whitehat researchers reporting that ChatGPT displayed limited accuracy in identifying security vulnerabilities.
Likewise, open-source governance provider Endor Labs has released research indicating that LLMs can only accurately classify malware risk in just 5% of all cases.
Of course, it’s also impossible to overlook the tendency that LLMs have to hallucinate, invent facts, and state them to users as if they were correct.
Many of these issues stem from the fact that LLMs don’t think but process user queries, leverage training data to infer context, and then predict a text output. This means it can predict both right and wrong answers (not to mention that bias or inaccuracies in the dataset can carry over into responses).
As such, they are a long way away from being able to live up to the hype of acting as a precursor to artificial general intelligence (AGI).
The public reception around ChatGPT is extremely mixed, with consumers sharing optimistic and pessimistic attitudes about the technology’s capabilities.
On one hand, Capgemini Research Institute polled 10,000 respondents across Australia, Canada, France, Germany, Italy, Japan, the Netherlands, Norway, Singapore, Spain, Sweden, the UK, and the U.S. and found that 73% of consumers trust content written by generative AI.
Many of these users trusted generative AI solutions to the extent that they were willing to seek financial, medical, and relationship advice from a virtual assistant.
On the other side, there are many who are more anxious about the technology, with a survey conducted by Malwarebytes finding that not only did 63% of respondents not trust the information that LLMs produce, but 81% were concerned about possible security and safety risks.
It remains to be seen how this will change in the future, but it’s clear that hype around the technology isn’t dead just yet, even if more and more performance issues are becoming apparent.
While generative AI solutions like ChatGPT still offer valuable use cases to enterprises, organizations need to be much more proactive about monitoring the performance of applications of this technology to avoid downstream challenges.
In an environment where the performance of LLMs like GPT-4 and GPT-3.5 is inconsistent at best or on the decline at worse, organizations can’t afford to enable employees to blindly trust the output of these solutions and must continuously assess the output of these solutions to avoid being misinformed or spreading misinformation.
Zou said:
“We recommend following our approach to periodically assess the LLMs’ responses on a set of questions that captures relevant application scenarios. In parallel, it’s also important to engineer the downstream pipeline to be robust to small changes in the LLMs.”
For users that got caught up in the hype surrounding GPT, the reality of its performance limitations means it’s a flop. However, it can still be a valuable tool for organizations and users that remain mindful of its limitations and attempt to work around them.
Taking actions, such as double-checking the output of LLMs to make sure facts and other logical information are correct, can help ensure that users benefit from the technology without being misled.
Share This Article
Since January 2017 Tim Keary has been a freelance technology writer and reporter covering enterprise technology and cybersecurity.
Tech moves fast! Stay ahead of the curve with Techopedia!
Join nearly 200,000 subscribers who receive actionable tech insights from Techopedia.
This site is protected by reCAPTCHA and the GooglePrivacy Policy andTerms of Service apply.
By: Mensholong Lepcha | Crypto & Blockchain Writer
By: Sam Cooling | Contributor
By: Alexandra Pankratyeva | Technology Writer
By: Arthur Cole | Contributor
By: Maria Webb | Technology Journalist 
Techopedia™ is your go-to tech source for professional IT insight and inspiration.
We aim to be a site that isn’t trying to be the first to break news stories,
but instead help you better understand technology and — we hope — make better decisions as a result.
Copyright © 2023 Techopedia Inc. – Terms of Use –Privacy Policy – Editorial Review Policy
By clicking sign up, you agree to receive emails from Techopedia and agree to our Terms of Use & Privacy Policy.
This site is protected by reCAPTCHA and the GooglePrivacy Policy andTerms of Service apply.
