Runway Gen-4 Is Next-Gen AI Video
AI video generation has crossed the uncanny valley to new heights, generating high-quality, realistic, consistent, and controlled video.
Runway Gen-4 is Released
We have been waiting for this moment for some time. Just like waiting for “sparks of AGI,” we’ve been waiting for AI video to get good enough to fool us completely, giving us realistic physics, higher resolution, hyper-realism, and consistent characters in AI video generation.
We got glimpses of it with Sora, demonstrated in February 2024 then released in December 2024, with realistic physics and features like remixing and recutting to go beyond just text-to-video. Google’s Veo 2 gave us a taste of it with 4K resolution, an improved understanding of physics, and excellent photo-realism and realistic movements. Luma AI’s Ray 2 from December 2024 likewise showed more natural motion and photo-realism.
With the newly released Runway Gen-4, fully real AI video generation feels closer than ever.
The Gen-4 model excels in generating consistent characters, locations, and objects across scenes. It allows users to set a specific look and feel, maintaining coherent environments while preserving distinctive styles and cinematographic elements.
When it comes to AI video generation, seeing is believing, and writing about AI video alone doesn’t do justice to it. So, I’ll share the relevant YouTube videos links directly to help you experience it yourself, below. Here’s the Runway Gen 4 demo video:
They also shared a video short called The Lonely Little Flame, starring a skunk, a flame and a rock, in a Claymation style that reminds me of a kid’s cartoon (any Pingu fans out there?). The AI animation comes across as indistinguishable from traditional animation. Characters stay consistent and move while background doesn’t:
The Herd is a short set on a farm with two characters and many cows, showing consistent characters across multiple shots, realistic physics of fire, and natural motion. Gen-4 can regenerate elements from multiple perspectives within scenes and utilizes visual references combined with instructions to create new images and videos with consistent styles and subjects.
Runway's Gen-4 improves the physical realism of motion with more realistic rendering of moving cars. Combined with controls on camera movement, lighting, and motion, this makes the AI video generation have a much closer match toa cinematic-directed scene or realistic video.
AI Video is the New CGI
The creatives generating AI videos are learning the best practices to get the most out of AI video generation. They need high-quality output, but just as important as realism is the level of control, editing, and consistency. Once the primary features are achieved, secondary features come to the fore.
The best way to obtain high-quality AI video generation with control over characters is to start with high-quality images to define the scene and characters. Runway Gen-4’s demo emphasized how an object could be directly placed in scenes and maintain its consistency, with editing controls to remix objects in different scenes. One result was the videos of animals in New York City scenes.
This is shown in an example shared on X. Midjourney and MagnificAI were used to generate high-resolution images of whimsical forest creatures, then Runway Gen-4 video brought them to life. The details are exquisite and are as photo realistic as magical creatures can be.
AI video generation has become a kind of animation that can be in any style, including a hype-realistic style no different from authentic video. Once AI image generation is scaled to Hollywood-level quality, all CGI special effects and animations will use AI. Animation and live action will merge (and it won’t be nightmare fuel like the seven dwarves).
The Best AI Video Models
In our review of multimodal AI models at end of 2024, we summarized the progress in AI video generation, saying:
AI video generation made huge leaps in 2024, and progress will accelerate this year as multiple competitors race to be the best in the category.
The field of contenders is crowded. Besides Runway Gen-4, we have the aforementioned Sora, Luma, and Veo2. We also have Meta's Movie Gen, Kling, Hailou Minimax, and several open-source AI video generation models, such as Genmo Mochi-1, Tencent’s Hunyuan Video, and Alibaba’s Wan 2.1.
There is a video arena leaderboard ranking different AI video generation models. It shows Kling as the leader with Minimax, Sora, Luma, and Runway (Gen-3) rounding out the top in that order.
Kling is strong at turning images into natural motion videos, due to its advanced motion dynamics and features like lip-syncing dialogue. Kling 1.6, their latest version released in December 2024, has improved sharpness, clarity, and dynamic lighting adjustments, resulting in more lifelike visuals.
There’s a direct comparison of Kling versus Runway Gen-4 on video generation starting from an image plus a text-prompt. Both seem quite capable, I’d give the edge to Runway Gen-4 on a few of these, but Kling seems to follow prompts better.
Each AI video generation model has its unique strengths and weaknesses. Those creating videos benefit from trying several models and seeing what works best for them. Alex Patrascu has his favorites, but YMMV:
My go-to platform has been Kling lately for random stuff, and Luma for bigger projects. … I still think Veo 2 excels at text-to-video. It's by far the best out there, but with images, it's not that powerful imo.
Conclusion - More to Come
The competition in AI video generation is intense and improvements are rapid, just as with other types of AI models. Runway Gen-4 is next-level and keeps Runway in a leading position in AI video generation. AI video generation has crossed the uncanny valley. We now have high-quality, realistic, consistent, and controlled video generation from AI.
However, AI video generation is far from its full potential. More improvements are coming:
More controls and detailed directions on motion, camera angles, lighting, and scenery.
AI models for various tasks beyond generation – such as editing, re-mixing, up-scaling, colorizing.
More physical realism and 3D world consistency. We might even see world models merging with video models to make this work, merging simulation models with video models.
More agency and autonomy. AI agents may come to AI video generation in the form of an agentic layer to help convert storyboards into detailed prompting and image-to-video generation flows.
Extending AI video generation into audio, which was introduced in Meta’s Movie Gen.
Multi-modality: We might get the equivalent of GPT-4o native image generation but with video.
As with all things AI, what we have already is enough to change everything, but there’s more to come.