On December 6th Eastern Time, Google CEO Sandal Pichai announced the official launch of the largest and most powerful Google model, Gemini 1.0. Gemini is a native multimodal big model and the first step in the new era of Google's big models. It includes three levels: the most powerful Gemini Ultra, the Gemini Pro for multitasking, and the Gemini Nano for specific tasks and end sides.
After Pichai's official tweet was released, Musk also commented below, "Gemini is impressive.". On the same day, Google also released a 6-minute delayed demonstration video showcasing Gemini's multimodal features (such as combining spoken dialogue prompts with image recognition). As of publication, the video has received 1.41 million views on YouTube.
However, just one day after Gemini's release, there have been voices outside accusing Google of "falsifying" Gemini's performance.
Among them, a Bloomberg column stated that Google distorted Gemini's AI performance in a demonstration video. Columnist Parmy Olson believes that in this video released by Google, Gemini seems to be very powerful, but a bit too powerful. In response to this question, Google admitted that the video demonstrating Gemini's performance was not real-time, but instead used still image frames from the original lens and written text prompts to prompt Gemini to respond.
6-minute demonstration video raises questions
Olson believes that Gemini's demonstration video is indeed very impressive. Gemini is able to infer that the drawn content is a crab based solely on some random points, demonstrating the large-scale model reasoning ability trained by Google DeepMind's artificial intelligence laboratory over the years. However, Olson pointed out that some of the features displayed by Gemini in Google's video are not unique to it, and ChatGPTPlus also has similar reasoning abilities.
The Daily Economic News reporter noticed that in this 6-minute video, Gemini seems to be able to quickly recognize images and respond within a few seconds. However, if users click on the description of this video posted on YouTube, Google has written an important "disclaimer" stating that "in order to achieve Gemini's demonstration purpose, latency has been artificially reduced, and Gemini's output time has been shortened for simplicity." This means that Gemini actually takes longer to answer each question than in the video demonstration.
Machine learning instructor Santiago Valdarrama hinted in an article on the X platform that Google's "disclaimer" for the aforementioned video seems to "showcase carefully selected results, not recorded in real-time but edited." He bluntly stated, "This is misleading, and anyone involved should feel embarrassed."
In addition, the MMLU multitasking language comprehension dataset test released by Google shows that the Gemini Ultra not only surpasses the GPT-4, but even surpasses human experts. However, many industry experts have found that in MMLU testing, the results of Gemini Ultra are marked with a small gray font below them cot@32 , represents the use of the thought chain suggestion technique and the selection of the best result after 32 attempts. As a comparison, GPT-4 did not have prompt word techniques and only attempted 5 times.
Denying fraud, Gemini's manager stated that they only shortened the reaction time for simplicity
In a report by American technology media The Verge, it is fair to say that this is not the first time that large technology companies have edited their product demonstration videos. Apart from Google, other large technology companies will make slight adjustments to the videos to avoid any technical issues caused by on-site demonstrations, which is also very common.
But Google firmly denies the claim of video fraud. In a blog post, Oriol Vinyals, Vice President of Google DeepMind and Joint Head of Gemini, explained the process of making Gemini demonstration videos: performance demonstration videos are not real-time, but use still image frames from the original lens, then write text prompts, and require it to respond through prediction.
"All user prompts and outputs in the video are authentic, but shortened for simplicity (Gemini's reaction time). This video showcases a multi-modal user experience built using Gemini, and we created it to motivate developers," emphasized Viales.
Olson did not buy it. She wrote in her column, "This is completely different from what Google describes - Google claims that anyone can have smooth voice conversations with Gemini because Gemini can observe the world around it in real-time and respond."
She also pointed out that Google's official Gemini modal performance shows that Gemini Ultra (highlighted in blue in the figure below) outperforms GPT-4 in 7 out of 9 standard benchmark tests. These benchmark tests are often used to test the ability of artificial intelligence models in high school physics, professional legal, and ethical scenarios.
However, in most benchmark tests, Gemini Ultra is only a few percentage points higher than OpenAI's GPT-4, and some even less than 1 percentage point. Olson believes that, in other words, Google, the so-called top-level artificial intelligence model, has only made limited improvements to the work completed by OpenAI a year ago.
It should be pointed out that Google's 6-minute Gemini demonstration video does not indicate that the model being demonstrated is Gemini Ultra.
Olson believes that a year ago, Google, a clumsy search giant, was caught off guard by ChatGPT of OpenAI and has since been hoping to catch up with the wave of generative artificial intelligence. Google hopes to make people remember through its powerful marketing that it has one of the world's most powerful artificial intelligence research teams and can access more data than anyone else. However, from a technical perspective, Google still lags behind OpenAI in terms of generative artificial intelligence.
However, in the technology industry, no one can guarantee that everything will go smoothly and stand firm. The early mobile phone giants Nokia and BlackBerry are examples. After Apple launched the more powerful and popular product iPhone, Nokia and BlackBerry quickly lost their market share. In the software field, the success of the market comes from systems with the most powerful performance.