Early reaction to Google's Gemini: It's better than GPT-4 – Business Insider
Jump to
The wait for Gemini felt a bit like waiting for Godot this year.
OpenAI, a relatively small upstart company, seriously upstaged Google when it fatefully released ChatGPT in November 2022.
When would the tech giants, particularly Google, follow?
Now we know.
On Wednesday, Google CEO Sundar Pichai and the Google DeepMind boss Demis Hassabis introduced the world to Gemini, Google’s new generative AI model, which it describes as its “most capable and general model yet.”
Gemini, which comes in three different flavors named Ultra, Pro, and Nano, is a multimodal artificial-intelligence system, which means it processes not only text but code, audio, images, and video to respond to user prompts. It also integrates directly into mobile devices, a first for an AI model and a point of excitement for app makers.
It’s such an important moment for Google that the cofounder Sergey Brin was hands-on “basically every day” with its development, as one Gemini developer put it.
Name order was randomised (except for the first 6 names which spell out Gemini) – Sergey was in with us basically every day, often pairing!
It’s early, but initial reactions suggest Gemini is putting in a good showing against GPT-4, the latest large language model powering ChatGPT.
First, some numbers that illustrate Gemini’s performance chops.
Google says its most powerful Ultra model, rolling out next year, “exceeds current state-of-the-art results on 30 of the 32 widely-used academic benchmarks” to assess large language models.
TL;DR: Gemini probably is better than GPT-4. But a closer look at the details suggests the outperformance is only marginal.
On one benchmark, Gemini Ultra has a 74.4% success rate in Python coding tasks, for example, versus 67% for GPT-4. On another benchmark, Gemini Ultra has a reading comprehension score of 82.4 compared with GPT-4’s 80.9.
As marginal as this is, first impressions of Gemini still appear to be positive among users who are getting a taste of Gemini through Bard, Google’s version of ChatGPT.
Mihir Patel, a research engineer at MosaicML, posted screenshots to X that compared responses from Gemini and GPT-4 to the question, “What is mamba in deep learning?”
Gemini’s response was more detailed in his screenshots and linked to external research papers. ChatGPT was closer to a smart Wikipedia entry.
Patel’s reaction: “Gemini is so good. SO much better and SO much faster than GPT-4.”
Gemini is so good. SO much better and SO much faster than GPT-4 🤯. I’m sold — switching over pic.twitter.com/BQI7ywwxSR
Another demo showed Gemini accurately describing a developing picture of a duck swimming in water. That elicited several semi-jokey “Google is so back” responses on social media.
Google is wild for this. They are so back. pic.twitter.com/cGEBZip6mH
Developers will probably welcome Gemini as an interesting alternative to OpenAI’s offering, too. Google told the Financial Times that because the Nano model was built to “run natively” on its Pixel phones, Android developers would have an easier time building AI apps.
The verdict is still out on just how successful Gemini will be and whether Google can pry users away from ChatGPT with it. Lots of users wanting to test Gemini may have to wait as the company continues to work on non-English-language versions of the models.
Gemini also still appears to be vulnerable to the hallucination issues that have plagued ChatGPT.
Pichai, Hassabis, and other Google execs will be familiar with the innovator’s dilemma — the idea that big companies risk losing their market leadership if they don’t stay nimble in product development. Early reactions to Gemini suggest there’s life in the old search engine yet.
Read next