Fundamental Thoughts on The State of AI
Scale begat Foundational Models begat today's AI that's changing the world ...
Scale is effective in AI - unreasonably and amazingly effective
Many of the problems in deep learning and AI in the past decade have been solved by scaling up. More data applied to train larger and deeper neural networks using more compute has yielded more effective deep learning models in image recognition, speech recognition, language, robotics, everywhere. The reason the Transformers paper (“Attention is all you need”) in 2017 has been perhaps the most important deep learning research paper in NLP that decade is that it opened the door to greatly more efficient scaling in training, leading to Large Language Models. Scale in AI rides on Moore’s Law scaling in semiconductors (cheaper and more powerful GPUs enable more training effort).
The term ‘unreasonable’ relates to how we luckily see positive benefits of large models and scaling that wasn’t necessarily anticipated or expected, such as the fact that LLMs are ‘few-shot learners’ and benefit from transfer learning. A lot of recent progress in AI has consisted of scaling up training, data and models, hoping for the best, and being pleasantly surprised at how good the results turned out.
We are in the era of Foundational Models
The deep learning scaling that was enabled by Transformers has led to very effective Large Language Models, what have been called Foundational Models. Because the pre-training on LLMs is so broad and on so much data, and the models have billions of parameters, these models capture a broad sense of language and knowledge. As such, they are useful across a range of tasks, topics and use cases. The broad interest and wide range of uses of recently released chatGPT LLM bears this out. An LLM can have even more utility in a specific area by taking that base LLM and fine-tuning it on specific domain data; examples include Github Copilot as well as effort to fine-tune models for medical use cases, scientific research, math, etc. Fine-tuned LLMs are a more effective architecture than a model build just on specialty data alone, as it benefits from the general purpose language understanding to grasp subtleties of the specific domain.
The prompt is the interface
More specifically, the prompt is the interface for the current generation of AI based on Foundational Models, specifically LLMs. The way to get LLMs to do things is through a natural language interface, and the way to get them to do the best tricks will be via the best instructions. That’s the prompt. Expect prompt engineering to rise in importance and the concepts of UI/UX to change, perhaps dramatically, as natural language interfaces take over.
Current state-of-the-art AI will change the world
AI is getting very effective, fast, and cheap: Effective - models can be trained in weeks on trillions of words, more words than can be read by an army of a million human scribes in their lifetimes; fast – humans have a neural loop that runs about 27 Hz, while computer GPUs run in the GHz range, allowing AIs generate AI outputs of texts, code and images in seconds what it would take a human to do in hours, days or weeks; cheap - the cost of creating all manner of creative output is cut down by many orders of magnitude.
Even if AI never gets far beyond human ‘intelligence’, achieves consciousness or agency or things that might challenge the uniqueness of the human mind, the entire world of creative, intelligent ‘white collar’ work is going to be upended by AI more thoroughly than the lives of weavers in the industrial revolution. So AI doesn’t have to be super-human to change the world.
2023 is the inflection point for the AI adoption
Just as the 1981 release of IBM PC was for personal computers, and like 1995 was the inflection point for the ‘world wide web’, and the 2007 iPhone 1 release was for smartphones, the 2023 release of high-quality AI models will be viewed in hindsight as the AI inflection point for AI adoption. Yes, machine learning and AI has been embedded in products over the years (speech recognition, recommendation engines, earlier chatbots), but these new models are at a much high level and change how we interact with software far more significantly. How we live and work in 2030 will be very different from before today because of the increasing importance of AI. Most notably, we don’t even need vastly improved AI from the GPT-4, PaLM, Claude, Copilot generations. This level of quality of the current best-in-class is enough to change business workflows and make people much more productive.
Anyone who says “AI cannot do X” will eventually be wrong
Practically all the stated limits on AI are practical limits, not fundamental limits. When faced with the assertion “AI cannot do X” the best response might be to say “AI cannot do X yet”. Instead, ask the question: “If AI had 100x the data, 100x the model size and 10,000x the compute, would it be able to do that task X?” If the answer isn’t yes, it’s likely to be “we don’t know,” because as models scale up they have been able to acquire surprising abilities. AI surely has limits, but as of right now, we don’t know the real limits of AI.
We will have super-human AI by 2029
The future is closer than you think - In a prior post I used this slogan and mentioned Ray Kurzweil’s model of technology progress as an exponential curve. AI as a tool is an accelerator of intellectual work, including writing, research and data analysis. This will cause the pace of technology progress generally and the pace of AI progress specifically to accelerate, as innovation feeds on itself. Kurzweil himself has described that acceleration effect as an exponential on an exponential. As fast as the pace of technology has been in our lives, it’s only getting faster, and the pace of AI technology specifically is not slowing down.
Given the rate of progress from GPT-2 to GPT-3 to GPT-4, it is likely that within about three or four generations of Foundational Models from current best-in-class models (Chinchilla, GPT-4, PaLM etc.), we will have an AI that out-performs a human on any type of intellectual or creative task you give it. Three or four generations of Foundational Models is likely to happen in about six years, that is by 2029. Even we don’t need super-human AI for AI to change the world, we will have super-human AI by 2029. That will accelerate technology to a more rapid pace than ever and change forever how we work and live. The singularity will be upon us sooner than we think.
By super-human, I don’t mean super-human consciousness or that we will have robot overlords. Rather, we will have a useful tool that can do just about any job better than a human. Just as any car can go faster than any human runs, a calculator can add numbers faster than any human can, and a database can store data larger than any human memory, any task we can think of requiring some intelligence will be done better by an AI. It may have emergent behaviors, and time will tell what those will be. The future of work is going to be very different.