I tried replacing Google with ChatGPT for 3 days. It was frustrating. – Insider
Jump to
In December 2022, Paul Buchheit, the entrepreneur who created Gmail, posted on X, the platform formerly known as Twitter, suggesting that “Google may be only a year or two away from total disruption” as a result of AI-powered platforms such as ChatGPT.
Buchheit is one of several people speculating that AI tools like ChatGPT could render search engines like Google obsolete
I decided to try replacing Google with ChatGPT for three days to see how successful it would be.
Like many people, I use Google every day.
Most often, I use it to check the weather, find the opening and closing times of places I want to visit, and find fun local events in my area like an evening candle-making class and a park run.
I also use Google for my job as a journalist to help me find interesting stories, search for experts to interview, and fact-check things that I’ve read online.
Going into the experiment, I was aware of some of the pitfalls of ChatGPT. I knew that one of the biggest downsides of the platform was that its last “knowledge update” was in September 2021, which meant that it wouldn’t be able to provide me with the most up-to-date information. I also knew that ChatGPT was likely to provide me with biased information based on the data it was trained on.
Despite these limitations, I was hoping that ChatGPT would still be able to help me find more accurate information quickly. In particular, I was hoping that it would eliminate the time that I often waste scrolling through ads and analyzing search results to see if they can actually answer my question before I click on them.
The first thing I decided to ask ChatGPT was a question about bed bugs. Since moving to London a couple of months ago, I’ve spent quite a lot of time thinking about the cleanliness of the London subway, also known as the Tube. Earlier that day, I’d seen a post on X (formerly Twitter) about bed bugs being “rife” on there.
So I asked ChatGPT, “Are there bed bugs on the Tube?”
It responded with a very vague answer: “I don’t have real-time information, but bed bugs can be found in various environments, including public transportation like the tube. It’s always a good idea to be cautious and take preventive measures if you’re concerned about pests.”
I found this response was vague, useless, and slightly patronizing. I tried to change my query slightly, hoping that it would give me more information. I wrote, “Where on the tube are there bed bugs?”
The response read: “As of my last knowledge update in September 2021, reports of bed bugs infesting the London Tube or other public transportation systems are rare and not well-documented.” It recommended that I check other sources for more up-to-date guidance.
While this answer was slightly more informative, its vagueness was still frustrating. It was also outdated, and there was no information about where it actually got the “reports” it was referring to.
I tried searching “bed bugs on Tube” on Google. Initially, I thought this was much more successful because a featured snippet at the top of the search engine results page showed an excerpt from an article in The Guardian that revealed where the “bedbug hotspots” are.
But when I clicked on the article, I realized it was published in 2018. I found it strange that Google was showing me a snippet from five years ago as the top search result. I concluded that perhaps it just wasn’t a widely used search term and that bed bugs probably weren’t a huge thing to be concerned about.
After feeling reassured that I probably don’t need to strip off my clothes whenever I’ve been on the subway in fear that bed bugs will take over my house, I decided to ask ChatGPT some more questions about living in London.
When I moved to the city two months ago, I used a Facebook group to find somewhere to live. While I do enjoy the neighborhood I’m living in, I wondered whether there might be better options out there. I still don’t have a good grasp on the area, so I decided to see whether ChatGPT could help me.
I typed in, “Best place to live in London,” and it provided me with a list of locations in London along with a brief description of each. It told me Richmond was famous for its “beautiful parks and a slower pace of life,” and that Wimbledon was “famous for its tennis championships.”
This was nice to know I suppose, but it didn’t answer my question. For instance, it didn’t provide any information about safety or rent prices, which are probably two of the most important things I wanted to know about.
Once again, it also failed to tell me where it got this information from, which made me reluctant to trust it. The vagueness of the information felt like something that had quickly been written up by someone who had never set food in London.
Meanwhile, searching for the same question on Google led me to a Reddit thread in which someone had asked the same question two years ago. The thread was full of responses from people who had (presumably) actually lived in the areas of London they were recommending.
While there was still no way to tell how accurate the information was, it made me feel more confident that the answers were probably written by humans with some actual experience of living in these areas.
There was some cross-over with ChatGPT’s answers, but these were much more in-depth and included information that was actually important, such as rent prices, transport links, how accessible shops were, and how safe the areas felt.
After realizing that ChatGPT probably wouldn’t be able to help me much with my London research, I decided to see how helpful it would be when it came to making my work more efficient.
I was working on an article about Gen Alpha’s defining teen movie. As part of my research, I asked ChatGPT to summarize some of the movies I’d featured in the articles.
ChatGPT was very helpful for answering basic movie trivia questions. One quick query told me the year each movie was released, the plot, and the actors involved. It was much quicker than Google, which showed me a long list of different search results that all seemed to provide the same information.
For another article, I needed some information about population statistics.
I asked ChatGPT, “How many people live in England?” It told me that as of September 2021, there were “over 56 million people” living in England. But it failed to tell me where it got this information from, and suggested that I check more recent sources.
Given that the platform’s last knowledge update was in 2021, I’d been expecting it to give me slightly outdated information. But the vagueness of the answer made me skeptical. I’d been hoping it would give me a more exact number, and that it would have told me where it got this figure from.
I had high hopes that Google would be able to answer this query, but somehow it managed to do an even worse job. To my disappointment, it showed me an extremely outdated graph from 2018, which revealed that 55.98 million people lived in England. This was the second time that Google had recommended information from over five years ago.
It did, however, provide me with its source, which allowed me to access more up-to-date statistics, although they differed from the answer that ChatGPT gave me, which made me wonder where ChatGPT was getting its information from, and whether it was accurate.
A key part of my job as a journalist is finding sources to interview. For another article that I was writing that week about a TikTok challenge involving parents cracking eggs on their toddlers’ heads, I had to find pediatricians and child psychologists to interview.
I typed in “pediatrician contact details” to see if ChatGPT would help. I was hoping it would provide me with a list of reputable pediatricians that would make my job easier and save me from having to sift through loads of search results.
Unfortunately, ChatGPT told me that it “can’t provide any specific contact details.” It didn’t even recommend anywhere online that I could look to find these contact details. Instead, it suggested I reach out to my local clinic, hospital, or healthcare provider for assistance.
Google was much more helpful, and it immediately provided me with a list of contact details of accredited pediatricians.
Admittedly, this is a very specific scenario. But if you’re searching for a doctor, dentist, plumber, or another service in your local area, I suspect ChatGPT won’t be much help.
This experiment did make me think about the main reason I love being online: I enjoy searching for and reading answers from people from across the world who can provide me with their own experiences.
I felt like ChatGPT stripped away the originality of the internet, and simply spat out whatever it predicted would most resemble its training data, making me miss the human element.
Overall, when it comes to searching for information, I found that ChatGPT can be more helpful than Google under a very limited set of circumstances. It was good for answering simple questions that have a definite answer, where the information doesn’t need to be updated over time. Other than that, it was pretty frustrating and useless.
Not having to scroll through loads of search results to try and figure out which answer is the one you’re looking for is nice, and having ChatGPT spit out an answer in a couple of seconds can save a significant amount of time (as long as you’re not expecting it to tell you where it got the information from). For everything else, I still preferred Google.
After the experiment, I figured I’d try and incorporate ChatGPT into my work more to answer basic questions. But I haven’t used ChatGPT at all. This is mostly because it feels like there’s no point in using it if I have to then spend more time fact-checking everything afterward to make sure it’s true, or to see where the information came from.
I’ve seen people say that ChatGPT can be helpful when it comes to summarizing information, such as meetings, books, movies, or articles, and that it can save time when drafting letters or writing emails. But as a search engine, it falls short.
If the information was more up-to-date, or if I could see where it came from, I might trust ChatGPT more. For now, I’m going back to Google.
Read next
