Gemini Ultra Released - A First Look
Bard becomes Gemini. Ultra powers Gemini Advanced, the latest and greatest GPT-4 contender.
Google Releases Ultra … aka Gemini Advanced
Yesterday, Google rebranded its Bard AI services as Gemini, released Google’s top-of-the-line Ultra AI model as “Gemini Advanced,” and launched new AI app and subscription services based on Gemini.
We’ve been expecting Gemini Ultra since its pre-release announcement in early December, and already there has been a flurry of unboxing and break-downs of this new AI model, which we’ll discuss below. However, as Google CEO Sundar Pichai announced, this is more than just a model release.
Gemini is evolving to be more than just the models. It supports an entire ecosystem — from the products that billions of people use every day, to the APIs and platforms helping developers and businesses innovate. -
Google's AI services are moving beyond existing tools like Assistant and Bard, replacing them both and merging them to offer a more comprehensive and integrated AI experience under the Gemini brand:
Their top-tier Gemini Advanced is powered by the Gemini Ultra 1.0 AI model. Access to Gemini Advanced requires a subscription to Google One AI Premium plan; it is priced at $20 per month, but it comes with a two months free trial. With the plan, users get other features (like 2TB storage) and will soon be able to use Gemini in Gmail, Docs, Sheets, Slides and Meet.
They are launching Google Gemini as a mobile app on Android (now) and iOS (in coming weeks).
The free tier Gemini uses the Gemini Pro AI model, suitable for answering questions and summarizing information, at a capability level comparable to original ChatGPT.
Gemini Advanced Features and Impressions
To even access Gemini Advanced and Ultra 1.0 therein requires a subscription sign-up, even though its zero cost for now. Once signed up, Google takes you to a distinctly ChatGPT-like interface.
It’s a testament to GPT-4 and ChatGPT’s dominance that the first question people have is how it compares to GPT-4, but it’s baffling that Google fell into a trap of building something so close to ChatGPT, even to the point of having the same price point for their subscription.
Some of Gemini Advanced best features are distinctly different from GPT-4:
Gemini Ultra is a lot faster than GPT-4. Asking it to write 500 words on any topic, or to create React code for a website, took about 20 seconds. It outputs faster than you can read. This is great in a user interface, but
Gemini’s “double-check” is a good fact-checking and anti-hallucination feature, that can provide sources for information cited in an answer. This is available not just on Advanced but on standard Gemini interface.
Gemini plug-ins: Gemini can connect with Google Maps, Flights, Hotels, and YouTube. Overall, this is quite limited, but I found the YouTube plug-in to be a very useful add-on to help with answers on topics referencing YouTube as a source, and the Google Maps makes for a good AI search on places “What’s the best or searching over YouTube.
You can share chats or an individual prompt and response to others, through public links that you can create and manage.
You can ask for responses to be shorter, longer, simpler, more casual, or more professional. This is similar to Copilot’s Creative / Balanced / Precise choice, but more general and useful. You can also show (or hide) multiple drafts of a response.
Several Gemini interface elements and features are different and better than what’s in ChatGPT. Hopefully, ChatGPT takes the hint and improves their own interface.
Head-to-Head - Ultra vs GPT-4
For evaluation purposes, we will call it “Ultra” as the Ultra 1.0 AI model is what is backing Gemini Advanced. AI models are constantly improving, so any assessments will get superseded by AI model updates, i.e. a future Ultra 2.0 AI model under the “Gemini Advanced” hood.
There have already been many “first look” reviews already for Gemini Advanced / Ultra, and I acknowledge several sources in the Acknowledgements section below. The biggest question most reviewers are asking as they give Ultra a first look is: How does Ultra compare to GPT-4? Is it as good as GPT-4 or even better?
Summarizing head-to-head comparisons across a number of user reviews and my own testing, here are some first impressions:
Gemini Ultra seems to be a bit more creative than GPT-4. Like ChatGPT, it’s able to write out stories, poems, and essays without difficulty, and as noted above, can do reply to prompts very quickly.
Gemini Ultra is good at reasoning, but not as good as GPT-4. it misses some logical questions that GPT-4 can get. It’s a mixed bag and not a clear-cut win for GPT-4, but overall GPT-4 does better.
For fact-based questions and how-to queries, Ultra gives typically correct and well-organized responses, on par but not necessarily better than GPT-4. The “double-check” sourcing feature mentioned prior helps confirm correctness.
Coding: Ultra will quickly generate code for a range of requests, but both myself and other reviewers found Ultra unable to complete some coding tasks that GPT-4 could do. I had trouble getting it to code up a Fibonacci spiral, its Python curses-based game came close for me and others but didn’t work, etc. How useful it is here remains to be seen, but this is another area where GPT-4 has an edge.
Gemini context limitations can cause issues. Google says, “Gemini’s ability to hold context is purposefully limited for now. As Gemini continues to learn, its ability to hold context during longer conversations will improve.” You might need Claude or GPT-4 if you need a long context window.
Multi-modal image understanding: Reviewers noted cases where it didn’t do well on image understanding. This area seems underwhelming when stacked up against their demos on this, but more study needed to evaluate.
Image generation: The same Imagen 2 backing Gemini models is doing the image generation here. It’s fine for many tasks such as the cover art for this article, good at photorealism but not quite Midjourney v6. Prompt fidelity isn’t great.
The Ultra vs GPT-4 verdict? Ultra is quite good, on-par or close with GPT-4 on most things, but not distinctly better. Overall I’d give the edge to GPT-4, and I’d stick with GPT-4 if I had to choose only one model.
Google oversold this as a GPT-4 killer, with bogus MMLU claims and a demo video that went way too far. It’s not. GPT-4 remains the world’s best overall frontier AI model. What makes GPT-4 ever more impressive it was trained in 2022 and is still holding its own as the top-tier AI model, not yet dethroned.
So, Sundar and Demis: Keep trying, you’ll get there.
Gemini’s “About Me”
If you ask Gemini Advance about itself, it gives you a well-organized response about its “characteristics, parameters, and the guardrails in place to guide responses.”
It’s in the “Gemini family of AI models” so expect Gemini to remain as a long-standing brand for Google’s generative AI models. We still don’t know details about Ultra 1.0, such as how big it is, the training dataset (except it is “massive” and consists of text and code).
One interesting self-reporting snippet: “Learning: I continue to learn and update my knowledge by accessing up-to-date information. I don't have a fixed knowledge cutoff date.” This makes it possibly different from other pre-trained models. Is it using some internal form of RAG, or is it doing online model updates?
The speed of inference that is superior to ChatGPT suggests either a different (possibly smaller) AI model architecture, or alternatively, the Google TPU v5 is just blazingly fast at inference. A downside of proprietary models is we are left guessing on all these internal matters.
Finally, here is Gemini on its own Guardrails:
These are some of the key guardrails designed to keep me safe, helpful, and aligned with specific principles:
Fact-Checking: I cross-reference information against verified sources to reduce the risk of spreading misinformation.
Anti-Harm: I am programmed to avoid generating responses that could be harmful, discriminatory, unsafe, or biased.
Self-Assessment: I analyze my outputs to identify areas where I might have provided incorrect information or harmful content. This helps me refine my responses.
User Context: I try to tailor my responses to the user's situation. For example, avoiding graphic content for sensitive topics or situations.
This is fine but mostly boilerplate; Google has given few details about specific guardrails in their technical documents. Overall, reviewers have noticed a bit more censorship around topics that it might not need to censor, while AI Explained was able to jailbreak by asking a question in Arabic.
Summary
Gemini Advanced powered by Ultra 1.0 is a rock-solid AI model release that brings a lot to the table. It has many useful new features and is the best AI model you can get from a company not named OpenAI. However, when compared with GPT-4, it comes with both strengths and weaknesses that nets out to nearly comparable to GPT-4. It’s not a GPT-4 killer, a GPT-4 competitor.
Power users and those tied into the Google ecosystem will likely want to use this, but those satisfied and tied in with GPT-4 won’t need to jump. If the price is right for API tokens, the Gemini Ultra 1.0 API could be useful to developers, as its speed could be an advantage for AI agent and other applications.
It’s baffling that Google don’t link to overall search better yet, like Perplexity AI has managed to do. It’s not excusable for the leading search engine provider to be bested by a startup and not have the best AI-driven search solution.
Acknowledgements
YouTube influencers and AI thought leaders on X in the first 24 hours have already been picking apart Gemini Advanced, and I should acknowledge and recognize the Gemini reviews from MattVidPro, AI Explained, Matthew Berman, All about AI, Arun Prakash, Matt Wolfe, and Wes Roth as informing my own use, testing, and commentary on the Gemini Advanced / Ultra release.
If you want to watch just one (super snarky) review, go watch Fireship on Gemini Ultra. Come for the tech release FOMO, stay for the snarky memes: