ChatGPT's Accuracy Under Scrutiny in New Study – Fagen wasanni
Fagen Wasanni Technologies
All about XXI Century Technologies
Fagen Wasanni Technologies
ChatGPT, the AI chatbot that has been making waves since its launch in November 2022, is facing questions about the accuracy of its responses. According to a recent study by Purdue University, the chatbot failed to answer 52 percent of software engineering questions correctly.
Researchers analyzed ChatGPT’s responses to 517 software engineering questions from Stack Overflow and discovered that 52 percent of the answers were inaccurate. Additionally, 77 percent of the responses were deemed to be ‘verbose’. The inaccuracies were primarily attributed to the chatbot’s lack of understanding of the underlying concepts in the questions. Even when ChatGPT did comprehend the question, it often struggled to arrive at a solution, resulting in a higher number of conceptual errors. The study also raised concerns about the AI tool’s limited reasoning capabilities.
The researchers suggested that prompt engineering and human-in-the-loop fine-tuning could partially address the problem of understanding questions. However, they emphasized the need to address the conceptual errors and the limitations of reasoning that are inherent to the chatbot.
In related news, a report by Analytics India Magazine highlighted a decline in ChatGPT’s userbase. The number of active users dropped by 12 percent from June to July, with 1.5 billion people actively using the chatbot. The report also noted that OpenAI, the company behind ChatGPT, is not yet profitable and is relying on a $10 billion investment from Microsoft to sustain its operations. Operating ChatGPT costs OpenAI approximately $700,000 per day.
These developments raise concerns about the financial viability of OpenAI if the company does not become profitable in the near future. Despite ChatGPT’s popularity, its inaccuracies and the decline in user numbers may pose challenges for the company’s long-term sustainability.
