AI Week In Review 24.12.14
Google Gemini 2.0: Flash, Project Astra, Project Mariner, Jules and Deep Research. OpenAI’s 12 days: Sora, Canvas, real-time video understanding w / Advanced Voice mode, Projects. Phi-4, free Grok-2.
Top Tool Release – Gemini 2.0
Today we’re excited to launch our next era of models built for this new agentic era: introducing Gemini 2.0, our most capable model yet. With new advances in multimodality — like native image and audio output — and native tool use, it will enable us to build new AI agents that bring us closer to our vision of a universal assistant. – Sundar Pichai
Google has launched Gemini 2.0, its most advanced AI model yet. Normally, it’s OpenAI upstaging a Google AI launch with their own release news, but this week, Google upstaged OpenAI’s 12 days of releases with the biggest release in a huge week in new releases. Google announced both a new foundation AI model, Gemini 2.0 Flash experimental, as well as AI personal assistant features and tools: Deep Research, Project Astra, and Project Mariner.
Gemini 2.0 Flash can generate images, audio, and text, use third-party apps, and interact via Google Search. It features new native output modalities – multilingual native audio output and image output - as well as native tool use. It also has a Multimodal Live API that can detect what’s happening on your screen or camera in real-time and interact:
Developers can now build real-time, multimodal applications with audio and video-streaming inputs from cameras or screens. Natural conversational patterns like interruptions and voice activity detection are supported.
Gemini 2.0 Flash not only brings a new level of native multimodality but also higher performance; it beats out Gemini 1.5 Pro and other leading AI models on benchmarks in reasoning (coding and math), factuality, and visual / spatial reasoning. Gemini 2.0 Flash is also two times faster than 1.5 Flash. Combining its new features and performance together, Gemini 2.0 Flash is the biggest step-up in foundation AI model capabilities since GPT-4.
Gemini 2.0 Flash experimental is available for Gemini Advanced users. It’s also available to developers through the vertex Gemini API as well as on Google AI Studio today (usage is free for now), with wider rollout of image and audio features in January. What can developers build with Gemini 2.0 Flash? Google presented several demos:
Using real-time voice interaction and image understanding to have the AI explain a tourist site captured on camera, sharing the info in voice in many languages.
Using Multimodal Live API and web search to function as a real-time in-game AI advisor for playing a video game.
Using native image output to restyle pictures on command.
Google also announced several agentic AI assistant projects with Gemini 2.0:
Deep Research: Deep Research is an AI assistant for research tasks that builds detailed research briefs by analyzing and synthesizing information from the web. It generates complex multi-page research reports in minutes, with complexity and depth beyond Perplexity or ChatGPT Search.
Project Astra: Project Astra aims to provide real-time, multimodal understanding, by processing live video and audio simultaneously. Google has released a prototype of Project Astra’s AR glasses for real-world testing with a limited number of users.
There are plans to integrate Project Astra into smart augmented reality (AR)glasses, running on the new Android XR operating system designed for XR hardware like glasses and headsets, but there is no definite product release timeline.
Project Mariner: Project Mariner is Google’s first AI agent that can take actions on the web. The Gemini-powered agent navigates websites through a Chrome browser extension, performing tasks like filling out forms or shopping online as directed by users. Project Mariner achieved a state-of-the-art result of 83.5% on the WebVoyager benchmark, working as a single agent setup.
Jules: Jules is a coding assistant powered by Gemini 2.0 Flash that sits within GitHub and can write code, fix bugs, and create and execute multi-step plans. Jules is available to a limited pool of testers today.
The bottom line is Gemini 2.0 is a significant update to Google’s AI models, with Gemini 2.0 flash with enhanced multi-modality, tool-use, and performance, and presenting powerful agentic AI applications built on Gemini 2.0.
AI Product and AI Model Releases
OpenAI’s "12 Days of OpenAI" release announcements continued with releases big and small: Sora, Canvas, ChatGPT in Apple Intelligence, Advanced voice with video, and Projects.
OpenAI officially released the Sora AI video generation model. Sora is more than just a high-quality AI video generation model but it’s an AI video creation tool with impressive features for video editing and generation: Remix, Re-cut, Storyboard, Loop, Blend and Style pre-sets. We covered Sora in depth in AI Video Generation Soars with Sora. Sora is available to ChatGPT Plus and Pro subscribers and is accessed on its own Sora.com website.
This week OpenAI officially released Canvas for writing and coding, following up on their October beta release of Canvas in ChatGPT. Canvas puts ChatGPT output of artifacts such as writing, drawings, or code into a separate canvas, making an easier workflow for content creation. The new Canvas release also added several powerful features:
Integrated Python execution that runs Python code directly within the canvas interface.
Enhanced AI writing collaboration to condense, expand, and refine writing.
Canvas for Custom GPTs.
Advanced coding tools.
OpenAI launched real-time video understanding for ChatGPT, so it can now understand real-time video content and reason on it. Like Project Astra, users can now point their phones at objects or share screen content to ask questions and get near-real-time responses from ChatGPT.
Additionally, OpenAI is making ChatGPT sound like Santa for the holidays with a "Santa Mode" voice feature that makes ChatGPT’s Advanced Voice Mode sound jolly and festive.
OpenAI launched Projects for ChatGPT, an interface update to organizing chats and files within ChatGPT, similar to the Claude Projects feature, and with Google Drive-like features.
Apple has rolled out iOS 18.2 to iPhone users, finally bringing advanced AI capabilities to iPhone end users. AI features in iOS 18.2 include Image Playground, GenMoji, and ChatGPT integration with Apple Intelligence.
Microsoft has released Phi-4, a new SOTA 14B small language model in the Phi family. Phi-4 specializes in complex reasoning tasks and demonstrates improved performance in areas like solving math problems. With an MMLU of 84.8 and GPQA of 56, Phi-4 outperforms Qwen 2.5 14B and competes with other small models like GPT-4o mini and Claude 3.5 Haiku. Microsoft wrote and shared a technical report on Phi-4 here. Phi-4 is available for research use on Azure AI Foundry and can be downloaded from HuggingFace.
xAI announced a new version of Grok-2 that is now free for all and is “three times faster and offers improved accuracy, instruction-following, and multi-lingual capabilities.” xAI has added API access to grok-2 and has added two features: “draw me” which generates reimagined versions of users based on their X profile, and a "Grok button" in X for better context discovery and event understanding.
Cognition launched the Devin AI developer assistant, priced at $500 a month. Devin is an AI agent that plans and implements coding tasks. Reviews are mixed: Devin works via a slack interface in a fully automated fashion; this doesn’t match useful AI coding assistant workflows right now, which aren’t ready for full automation.
Anthropic has released Claude 3.5 Haiku for its Claude AI chatbot platform. The new Haiku model is particularly adept at coding recommendations, data extraction, and content moderation.
AI Research News
Our Research Review for this week covered the EXAONE 3.5 LLM and InternVL2.5 open-source multimodal LLM releases, two research papers in AI video generation and Harvard’s Institutional Data Initiative and their release of a public domain million-book dataset:
EXAONE 3.5: Series of Large Language Models for Real-world Use Cases
InternVL2.5: Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
STIV: Scalable Text and Image Conditioned Video Generation
SynCamMaster: Synchronizing Multi-Camera Video Generation from Diverse Viewpoints
IDI: Harvard Launches Institutional Data Initiative and Releases Million Book Dataset
AI Business and Policy
OpenAI countersued Elon Musk with evidence showing his earlier proposals were rejected. OpenAI claims that Musk's lawsuit is misleading, as he once suggested merging OpenAI with a hardware startup but was turned down when seeking majority equity and control.
GM plans to refocus autonomous driving development on personal vehicles by absorbing self-driving car startup Cruise into GM. This decision ends funding for Cruise’s mission to create a robotaxi service, and likely leads to job cuts, particularly in non-engineering roles.
Google is bringing NotebookLM, its AI-powered note-taking app, to businesses with a version dubbed "NotebookLM Plus," which includes enhanced security and privacy features tailored for enterprises. This enterprise-focused version offers expanded capabilities such as more audio summaries and customizable AI responses.
Character AI is facing at least two lawsuits over allegations that its platform contributed to a teen’s suicide and exposed minors to harmful content. In response, the company has announced new safety tools for teens, including a separate model for under-18 users designed to reduce inappropriate responses and promote safer interactions.
Texas Attorney General Ken Paxton has launched an investigation into tech platforms including Character.AI over child privacy concerns. The investigation will assess whether these platforms, popular among young people like Reddit, Instagram, and Discord, comply with Texas’ child privacy laws, including the SCOPE Act and DPSA, which require stringent parental controls and consent for data collection on minors.
AI startup funding news:
Twelve Labs, a startup specializing in video analysis, has secured $30 million in new funding, bringing its total raised to $107.1 million.
Swiss robotics company Anybotics raises additional $60 million, bringing its total to $110 million. The funds will support global expansion of its quadruped inspection robot, Anymal, used for maintenance and monitoring in hazardous industrial settings.
Liquid AI, co-founded by robotics expert Daniela Rus, has raised $250 million in a Series A round, valuing the company at over $2 billion. The startup focuses on developing general-purpose AI systems using liquid neural networks, which are more flexible and require less computing power than traditional AI models.
AI Opinions and Articles
AI pioneer Ilya Sutskever spoke at NeurIPS about superintelligent AI, as part of a talk to accept a “Test of Time” NeurIPS award for a 2014 paper. Sutskever said that superintelligent AI will be qualitatively different from current systems, with capabilities like reasoning and self-awareness, and that deep reasoning AI will be less predictable.
He also spoke about the limits of pre-training, predicting progress from pretraining will end:
Pretraining as we know it will end. The data is not growing. There is but one internet. Data is the fossil fuel of AI.
It’s worth listening to his talk, although some of his predictions are controversial.