Software 3.0 and The AI Engineer
AI-based Applications are the future and AI Engineers are its builders
The AI evolution and the rise of LLMs and LMMs is disrupting everything, but more than anything it is disrupting the software stack, changing what software means and how we deliver it. We try to fit this new paradigm into pre-existing concepts, and so our prior article looked at AI as the new Operating System.
One thought leader sorting through this is Swyx, co-host of the Latent Space podcast. He has introduced both a way to label the AI paradigm shift - Another way to look at this shift is Software 3.0. He has labelled the job title for those building the AI-based applications in this new paradigm: the AI Engineer.
Software 3.0
Software 3.0 describes a software paradigm where the LLM itself encoding the decisions that in prior iterations of software would be hard-coded into the software code itself.
The term comes from this era of being the next iteration beyond Software 2.0, a label given by Andrei Karpathy in a 2017 blog post to describe the machine learning model paradigm for software. Software 2.0 referred to a shift in way software was being developed, where machine learning models, in particular deep learning neural networks, played a significant role in automating the creation of software.
In Software 2.0, deep learning networks learn and generate code based on data and examples, essentially replacing some aspects of traditional human-driven software development. By 2017, we had already solved difficult challenges such as speech synthesis and language translation this way.
In the ML Software 2.0 paradigm, you don’t have to invent the algorithm or write it, but if you bring enough data, you can solve the problem. What you also need though is not just the data, but the data science and the ML pipeline to bring it to fruition.
Now, with LLMs, the idea is taken to the next level. The software is not in a machine learning model but is now embodied in the LLM, zero-shot AI available on demand, through an API. Now, in the Software 3.0 paradigm, application developers don’t need to bring data or develop ML pipelines to get LLM functionality. This high-level functionality is accessible behind the LLM API.
Software 3.0 is a recognition that Foundation AI Models are the bedrock of these new possible applications, and thus represents a new paradigm for how most new software will be created. The extreme capability and generality of LLMs and their new multi-modal kin the LMM (Large Multi-Modal Models) make them the basis of a number of features and stand-alone applications, including vast varieties of tools and applications that couldn’t otherwise exist.
In this Software 3.0 paradigm, the Foundation AI Model is the base layer of the AI-based software stack, accessible via its API. The main challenge in creating an AI-based application is to channel the AI model’s capability to address the desired application use case accurately and safely. We need an AI-based ecosystem to support that, to:
Connect AI models to data - with vector databases such as PineCone and connectors like LlamaIndex
Prompt, guide and check AI model inputs and outputs - with tools such as GuardRails
Chain AI model responses iteratively to solve complex problems - with frameworks such as LangChain
Who will do this work of building these AI-centric applications? The AI Engineer.
What the AI Engineer Does
In simple terms, an AI Engineer is a software engineer who develops AI-centric software applications.
The AI revolution would be very stunted if it was only accessible to the AI researchers and ML engineers building the LLMs and other large AI models. There are thousands of AI researchers and ML engineers in organizations like OpenAI (about 600 employees) or other Big Tech AI labs building frontier AI models. But there are millions of software engineers who build enterprise applications, internet SaaS applications, websites, and mobile apps.
Because GPT-4 and similar AI models are now general, powerful and accessible via an API, you can build powerful AI applications without knowing how to make an AI model. The AI Engineer can leave the training of models to the ML Engineers and data scientists, and can abstract away model internals. That makes the Software 3.0 paradigm accessible and viable for tens of millions of software engineers.
On the continuum, the AI Engineer looks like this:
So, again, the AI Engineer’s role and expertise is in building the tools and the applications based on the new AI-based Software 3.0 stack. This could mean:
Building the tools and infrastructure to extract value out of AI models.
Turning raw AI model capabilities into proven and reliable systems to automate workflows or serve as useful intelligent assistants.
Building agents or agent-like applications based on AI models
The AI Engineer Summit and Ecosystem
Swyx has not only given us a label for the new software stack - Software 3.0 - and those builders of it, AI Engineers, but he has gone all in on helping building the communities needed to power this AI ecosystem forward.
One of these efforts is the first AI Engineer Summit in San Francisco, going on this week (October 9-11, 2023). At this summit, they have mentioned and announced efforts in support of the AI Engineer community, the AI Engineer Foundation.
Swyx Keynote at AI Engineer Summit
Swyx also gave his own thoughts on the state of AI and the importance of the AI Engineer in his own keynote at the AI Engineer Summit.
We are just in time. - Swyx
One useful takeaway was a view of where we are in the AI Revolution. We all can see we are in a large technology revolution, but where are we?
When did the AI Revolution start? He pins it at 2012, with AlexNet. He points to work by Carlota Perez on Technology revolutions to pin the current stage of AI as early, and that this AI Summit is like the 1927 summit of physicists.
Other technology revolutions took 70 years to go through the innovation, development, deployment, and adoption cycles, and AI may be quicker, but is no less profound. We are in the early stages of AI, an exciting time and place to be; as he put it “We are just in time.”
He considers how far AI can go by asking: What happens if you take scaling seriously? We can put the amount of compute used to train models as a proxy for their power. We increased the compute for training AI models by 6 orders of magnitude since the development of transformers, and another 6 orders of magnitude more compute by 2030 is possible.
He keys off a George Hotz analysis of compute effort to train a model, equating a person to 20 PetaFlops. Then GPT-3 was 1.5 person-years, GPT-4 was 96 person-years. Then scaling 100x each generation, we’d have GPT-5 be 100x GPT-4, so 100 lifetimes. A GPT-9 by 2030 might be 10 billion lifetimes. This scaling is beyond comprehension.
He points to the power of AI in answering What is an AI Engineer? He talks about the types of AI Engineer:
Software engineer building AI products - AI product Engineer
Software engineer enhanced by AI tools - AI enhanced Engineer
An AI Agent itself doing software development - AI Engineer Agent.
Inspired by Replit CEO Amjad Masad’s description of the 1000x developer, he takes this concept to the next level to describe the 1000x AI Engineer, who leverages AI to 1000x their value, along the lines of three types of AI Engineer:
10x AI Enhanced Engineer: Using tools such as Copilot, Ghostwriter, Codium, and build you own dev tools and AI tools to increase productivity.
10x AI Product Engineer: Using RAG/LLM frameworks; models, infrastructure, and finetuning; doing AI UX and growth hacking to create value quickly.
10x AI Engineer Agent: Writes faster than you, while you sleep, in parallel on demand.
Figure 3. the 1000x AI Engineer
When you combine the 1000x AI Engineer and 6 orders of magnitude improvement in AI models possible by 2030, the possibilities and headroom in what Software 3.0 is vast. The AI era is just starting, and the AI Engineer Summit is the start of something much bigger. We will give a more complete review on the summit in a followup.