The Era of AI Acceleration
So this is what The Singularity feels like
The AI Innovation Rocket
So much is happening at once in the world of AI. Today (November 6th, 2023) is OpenAI’s DevDay. We will have more to say about it soon, but preliminary rumors and leaks suggest some massive reveals. This is two days after X launched Grok and follows a week of huge announcements and upgrades plus big policy news. There is also a constant barrage of research and improvement in AI on a weekly basis.
All this is to say that the pace of AI development is accelerating like a rocket heading to orbit, and trying to follow it all is like drinking from a fire hydrant. To illustrate this further, I’d like to share what Paul Harvey would have called “the rest of the story” of Zephyr.
The Zephyr Story
Hugging Face co-founder Thomas Wolf shared this story of rapid open source innovation on X as a ‘feel-good’ story. I’m sharing it in full as an ‘AI acceleration’ story:
It’s a story of people on three continents building and sharing in the open a new small efficient and state-of-the-art AI model. It started a couple of months ago when a new team in the AI scene released their first model from their headquarters in Paris (France): Mistral 7B. Impressive model, small and very strong performances in the benchmarks, better than all previous models of this size.
And open source! So you could build on top of it.
Lewis in Bern (Switzerland) and Ed (in Lyon, in the South of France) both from the H4 team, a team of researchers in model fine-tuning and alignment were talking about it over a coffee, in one of these gatherings that often happen at Hugging Face to break the distance between people (literal distance as HF is a remote company). What about fine-tuning it using this new DPO method that a research team from Stanford in California just posted on Arxiv, says one? Hey, that’s a great idea, replies the other. We've just build a great code base (with Nathan, Nazneen, Costa, Younes and all the H4 team and TRL community) let's use it!
The next day they start diving in the datasets openly shared on the HF hub and stumble upon two interesting large and good quality fine-tuning datasets recently open-sourced by OpenBMB, a Chinese team from Tsinghua: UltraFeedback and UltraChat.
A few rounds of training experiments confirm the intuition, the resulting model is super strong, by far the strongest they have ever seen in their benchmarks from Berkeley and Stanford (LMSYS and Alpaca). Join Clementine, the big boss of the open evaluation leaderboard. Her deep dive into the model capabilities confirms the results: impressive performance. But the H4 team also hosts a famous faculty member, Pr. Sasha Rush, Associate Professor at Cornell University in his daytime, hacker at HF in his nighttime. Joining the conversation, he proposes to quickly draft a research paper to organize and share all the details with the community.
A few days later, the model, called Zephyr (a wind like Mistral), paper, and all details are shared with the world. Quickly other companies, everywhere in the world starts to use it. LlamaIndex, a famous data framework and community, shares how the model blew their expectations on real-life use-case benchmarks, while researchers and practitioners discuss the paper and work on the Hugging Face hub.
All this happened in just a few weeks catalyzed by open access to knowledge, models, research, and datasets released all over the world (Europe, California, China) and by the idea that people can build upon one another work in AI to bring real-world value with efficient and open models.
Stories like this are numerous everywhere around us and make me really proud of the AI community and see how we can build amazingly useful things together.
The AI 2023 Story - Speed
Let’s dissect the data points:
Open source AI models are getting fine-tuned, improved, and optimized in matters of weeks and even days. Another example is the Phind fine-tuned from code lllama coding model recently claiming to beat GPT-4 on HumanEval. they have iterated on 7 versions of the fine tune in a bit over two months.
Shorter knowledge diffusion cycles: Papers are published on Archiv as pre-prints, foregoing lengthy peer review, so there are sometimes errors, but speeding the diffusion of knowledge.
Collaboration and progress is global. Mistral is based in France, with a team from Meta AI Research (US-based). The dataset was developed by Chinese researchers. Global reach means more participants means faster progress.
AI itself is a part of acceleration. Using automation, AI and ML can speed coding, development and data processing work.
The moral of the Story is Speed. Open Source plus global collaboration plus AI equals rapid advances. The OODA loop or feedback loop of AI R &D is measured in days.
Back in March, a Google employee sounded the alarm that “Google has no moat” based on the concern that open-source would be lapping the proprietary models in innovation speed. The open source development cycles for smaller models have been remarkably rapid, and they are keeping pace as “fast followers,” but Big Tech incumbents have used access to large super-computing clusters to build a moat around the largest foundation AI models.
However, the only way they can outpace the open source innovation cycle is to innovate rapidly themselves. This brings us full circle to OpenAI’s DevDay. If the rumors pan out, this will confirm AI is on an exponential innovation trajectory.
AI keeps getting better.